Proceedings of the 24th Conference on Computational Natural Language Learning, pages 365–375Online, November 19-20, 2020. c©2020 Association for Computational Linguistics

https://doi.org/10.18653/v1/P17

365

Analogies minus analogy test: measuring regularities in word embeddings

Louis Fournier 1

louis.fournier@polytechnique.edu

Emmanuel Dupoux 1 2

1Cognitive Machine Learning (ENS-CNRS-EHESS-INRIA-PSL Research University), France2Facebook A.I. Research, Paris, France

3Laboratoire de Linguistique Formelle (CNRS-Paris Diderot-Sorbonne Paris Cite), France4University of Toronto, Toronto, Canada

emmanuel.dupoux@gmail.com

Ewan Dunbar 1 3 4

ewan.dunbar@utoronto.ca

Abstract

Vector space models of words have longbeen claimed to capture linguistic regulari-ties as simple vector translations, but prob-lems have been raised with this claim. Wedecompose and empirically analyze the clas-sic arithmetic word analogy test, to motivatetwo new metrics that address the issues withthe standard test, and which distinguish be-tween class-wise offset concentration (simi-lar directions between pairs of words drawnfrom different broad classes, such as France–London, China–Ottawa, . . . ) and pairing con-sistency (the existence of a regular transforma-tion between correctly-matched pairs such asFrance:Paris::China:Beijing). We show that,while the standard analogy test is flawed, sev-eral popular word embeddings do neverthelessencode linguistic regularities.

1 Introduction

Vector semantic models saw a surge in interest afterembeddings trained under the word2vec architec-ture (Mikolov et al., 2013a) were shown to encodelinguistic regularities (Mikolov et al., 2013b). Thedemonstration relied on the arithmetic analogy test:relations such as king + woman−man ≈ queenwere shown to hold for a variety of semantic andgrammatical relations. Evaluation of word embed-dings on analogy tests and training on related lossfunctions remains current. There is also continuedinterest in theoretically grounding the success ofdistributional embeddings on these tests (Allen andHospedales, 2019; Ethayarajh et al., 2019).

There is, however, a substantial literature point-ing to problems with word analogies (see Section2), leading to the conclusion that word analogiesare fragile in practice, and sometimes going so faras to imply that the positive results were erroneous.These critiques have been ambiguous as to whetherthe problem is that the embeddings in question do

not really encode the relevant linguistic structure,or whether the issue is merely that the arithmeticanalogy test as it is standardly defined, is flawed.

The current paper confirms that there are seriousproblems with the standard analogy test, as it con-founds three different properties of word vectorswhile purporting to measure only one: class-wiseoffset concentration (similar directions betweenpairs of words drawn from different broad classes,such as France–London, China–Ottawa), within-pair similarity between test words (for example,the similarity between Paris and France), as wellas the pairing consistency the test sets out to mea-sure (the presence of a regular direction encodingrelations such as capital-of : France–Paris, China–Beijing). We give an algebraic decomposition ofthe standard analogy test that explains previous neg-ative results in terms of within-pair similarity. Us-ing new measures, we show that, in practice, offsetconcentration, rather than true pairing consistency,may account for part of word embeddings’ success.Nevertheless, we show that several standard wordembeddings do show pairing consistency for therelations tested by the BATS analogy benchmark(Gladkova et al., 2016).1

2 Related work

The validity of the arithmetic analogy test has beenquestioned in several papers, starting with Levy andGoldberg (2014). We detail in Section 3 severalmajor issues with the test as raised by Levy andGoldberg (2014), Linzen (2016), and Rogers et al.(2017). Finley et al. (2017), Newman-Griffis et al.(2017), Chen et al. (2017) and Schluter (2018) alsoraised concerns about the test and its assumptions.More recently, Nissim et al. (2020) argued againstthis test as an inadequate tool for studying bias in

1Code is available at www.github.com/bootphon/measuring-regularities-in-word-embeddings.

366

word embeddings. Rogers (2019) observes thatmany of the issues have been ignored.

Some works have proposed other measures oflinguistic relations in word embeddings. Levyand Goldberg (2014), Vylomova et al. (2016), andRogers et al. (2017) all examined the similarity ofvector offsets in a more direct way than the stan-dard analogy test (see Section 3 below). Drozdet al. (2016) proposed a method based on predict-ing the class of the unknown word, and Bouraouiet al. (2018) relaxed the assumptions by allowingprobabilistic models to predict the relations.

We claim that these works still do not providea satisfactory measure of how well word vectoroffsets encode linguistic relations. Without sucha measure, it is impossible to assess whether theoriginal conclusions are correct. We develop thisargument, and then develop a new measure, below.

3 The arithmetic analogy test

Mikolov et al. (2013b) proposed to measure thepresence of linguistic relations in the structure ofword embeddings using vector arithmetic. For twopairs of words representing the same linguistic re-lation, (a, a∗) and (b, b∗), the test assesses whether:

b + (a∗ − a) ≈ b∗ (1)

In practice, the test assesses whether b∗ is thenearest neighbour to b + a∗ − a, using the cosinesimilarity (cosine of the angle between x and y):

sim(x, y) =x · y‖x‖‖y‖

(2)

The assumption is that, if the offset oa=a∗−a isparallel to the offset ob=b∗−b, then this commonoffset direction encodes the same linguistic relation.For example, if Paris minus France goes in thesame direction as Beijing minus China, and as othercapital–country pairs, the common offset directioncan be seen as encoding the relation capital-of.

Thus, the idea behind the arithmetic analogytest is that offsets corresponding to the same lin-guistic relation, if they are the same or roughlythe same, should be interchangeable. Therefore,France + (Beijing − China) should be the sameas France + (Paris − France) (modulo the vectormagnitude, if we are hypothesizing that it is thedirection, and not the precise normed vector, thatencodes the relation capital-of: hence the use ofthe cosine similarity).

The criticisms of Mikolov et al. (2013b) and theresearch it inspired have often been characterizedas problems with “word analogies.” This is am-biguous: it does not distinguish between problemswith the method of using the arithmetic test justdescribed for testing the presence of linguistic reg-ularities, and the veracity of the conclusion thatthere are linguistic regularities coded in word em-beddings as vector offsets. In order to resolve thisambiguity, one would need to know for sure, usingsome better measure, whether linguistic regularitiesare coded. We propose such a measure beginningin Section 4. We first analyze the key problems thathave been previously raised.

3.1 Within-pair similarity

The analogy will be scored correct if b∗ is thearg maxx sim(b + oa, x). Levy and Goldberg(2014) call this objective 3COSADD. Let us callsim(b + oa, b

∗) the “analogy score.” Putting asidenorms, the analogy score can be decomposed as:

sim(b + oa, b∗) ∝ b · b∗ + oa · b∗

∝ b · b∗ + oa · ob + oa · b(3)

The first term, the dot product of b and b∗, is pro-portional to the within-pair similarity, sim(b, b∗).The second term is proportional to the similaritybetween the offsets, which would appear at firstglance, to be the term of principal interest in as-sessing whether a linguistic relation is consistentlycoded by the offsets. The third term is proportionalto the similarity between the start word b and theoffset oa, and does not depend on b∗.

We develop an analysis of the offset similarityin sections 4–5, deriving from it the offset con-centration and pairing consistency properties wepropose to measure. Previous work suggested thatoffset similarity is not very high in practice, com-pared to within-pair similarity. Levy and Gold-berg (2014) showed that evaluating on the basisof similarity between offsets, using PAIRDIREC-TION to determine the nearest neighbours, leadsto failure on analogy tests that would otherwisesucceed using the 3COSADD objective. Rogerset al. (2017) showed that within-pair similarity iscorrelated with performance on the standard arith-metic analogy test. Clearly, measuring whetherParis is similar to France is not the same as mea-suring whether the relation capital-of is present inthe word embedding. As Levy and Goldberg pointout, within-pair similarity is not irrelevant to the

367

question—if the idea is that there is a consistentdimension capital-of which should constitute theprincipal difference between Paris and France, thenthe two words should otherwise be similar. But atest dominated by within-pair similarity can leadto spurious results, since pairs of words may besimilar without capturing the relation in question.

Using the Bigger Analogy Test Set (BATS)(Gladkova et al., 2016) and the pre-trained GoogleNews skip-gram embeddings of Mikolov et al.(2013a), we examine the role of the three termsof (3). BATS consists of forty relations, each com-posed of fifty word pairs, grouped into four broadtypes: inflectional morphology, derivational mor-phology, encyclopedic semantics (like capitals andanimal sounds), and lexicographic semantics (likesynonyms and antonyms). When multiple pairsexist for the same input word, we only keep thefirst. For each of the forty relations, we take themean for each term over all a :a∗ ::b :b∗ tuples.

Complementing previous analyses, we displaymeans for these three terms in Figure 1, confirmingthat within-pair similarity is indeed positive andlarge compared to the other terms. Notably, offsetsimilarities vary between relations, but are almostalways smaller. However, they are always positive.

3.2 Honest analogies: predicting input words

Linzen (2016) observes that the arithmetic anal-ogy test, as practised, excludes a, a∗, and b fromthe arg max—otherwise, one of these input wordstends (incorrectly) to be the response—most of thetime, b. Rogers et al. (2017) and Schluter (2018)imply that part of the issue may be that ‖oa‖ issmall. This cannot be why: the offsets have simi-lar magnitudes, so if ‖ob‖ is long enough to bringb to b∗, then ‖oa‖ should be too. Importantly, ifwe compute the decomposition in (3), but supposethat b∗ is equal to b + oa (oa = ob: the analogyscore is thus always 1), we observe empiricallythat the within-pair similarity- and offset similarity-driven terms have similar values (Figure 8 in theappendix). This means that ‖oa‖ is similar to ‖b‖.The weakness of the offset-similarity term in thescore is not driven by the offsets being small.

Define ∆sim as the analogy score minus the sim-ilarity of b + oa to the start word b, which must benegative in order for the “honest” analogy test toreturn b. We observe that ∆sim is equal to:

∆sim =b+oa‖b+oa‖

·( b∗

‖b∗‖− b

‖b‖) (4)

By replacing b∗ by b + oa, we can expand thescore, which is then proportional to:

‖b‖−‖b∗‖‖b‖

·(b+oa)·b + oa ·ob + b·ob (5)

This decomposes ∆sim into three terms. Thefirst term is proportional to the difference betweenthe norms of b and b∗, which we have empiricallyobserved to be small, and which will be null fornormalized embeddings. The second term is pro-portional to the offset similarity, and the third termis proportional to the similarity between b and ob.For normalized word vectors, the third term is equalto sim(b, b∗)− 1, which is the negative cosine dis-tance between b and b∗. Since ‖b‖ and ‖b∗‖ tendto be similar, we find that this is a good approxi-mation even for non-normalized vectors. ∆sim isthus negative, roughly, whenever the offset similar-ity is less than the cosine distance between b andb∗. Intuitively, the similarity of the offsets mustcompensate for the difference between b∗ and b.

In Figure 2, we plot means for the three termsfrom (5). We observe that the first term is indeedsmall and can be ignored. Importantly, we observethat the offset similarity term is smaller in magni-tude than the negative b–b∗ distance term).

Linzen’s observation that an input word is typ-ically predicted under the 3COSADD objective—and not an unrelated vocabulary word—impliesthat not only is ∆sim negative, but that the equiva-lent value is negative for all words in the vocabulary.This in turn would imply (for normalized embed-dings) that the similarity between oa and the offsetof b with respect to any other word is smaller thanthe distance between b and that word. Far fromstaying in the neighbourhood of b, oa moves b in adirection far from every word in the vocabulary.

Our analysis of these known problems with wordanalogies details the central problem: the pairs ofoffsets tested are not similar enough to support thekind of arithmetic demanded by the 3COSADD ob-jective. An illustration of the dominance of within-pair similarity is given in Figure 3. Note that wepresent this figure only to illustrate the dominanceof within-pair similarity. Two-dimensional projec-tions are misleading: the vectors in Figure 3 donot have the property, discussed above, that theoffsets are of length similar to the word vectorsthemselves—this is difficult to reconcile with highwithin-pair similarity in two dimensions.

None of this, however, implies that word embed-

368

Figure 1: Decomposition of the analogy score in three terms, with only two depending on the word to predict. Thelargest component is b · b∗, proportional to within-pair similarity. b ·oa is proportional to the similarity of the offsetto the start word b. ob · oa is proportional to the similarity between offsets. All terms are divided by the sameoverall normalization term (not indicated in the legend).

Figure 2: Decomposition of ∆sim, the difference between the similarity of the analogy to b∗ and to b. The termproportional to the dissimilarity of b to b∗ is greater than the one proportional to the offset similarity, which meansthe common direction of the offset is not enough to bring the analogy closer to b∗ than to b.

dings do not show linguistic regularities. It simplyimplies that the standard arithmetic analogy testdoes not measure these regularities correctly, andshould not be used. While it may be true that thequantity of interest, the similarity between offsets,is generally insufficient to dominate the standardanalogy test, that does not mean it is too small tobe considered an encoding of a linguistic regularity.How similar is similar enough? In the followingsection, we propose an answer to this question.

4 Measuring Regularities

The similarity between offsets, sim(oa, ob), is thequantity of primary interest to assess whether lin-guistic relations are represented as consistent di-rections. We seek a measure of whether they are.As noted, Levy and Goldberg (2014) showed that

arg maxx sim(a∗ − a, x− b) does not tend to findthe right answer (b∗). However, the fact that b∗ doesnot maximize offset similarity does not answer thequestion of what a meaningful level of offset sim-ilarity is. Rogers et al. (2017) analyzed examplesof correct and false analogies and showed them tohave comparable offset similarities. This makessense: a baseline level of offset similarity can befound by looking at unrelated pairs. However, thatpaper examined only one isolated example.

We propose an offset concentration score(OCS) measuring the offset similarity among pairsbelonging to the same relation and a pairing con-sistency score (PCS) measuring the distinctness ofthe similarities from a baseline. We propose PCSas the correct measure of linguistic regularity, withOCS a source of supplementary information.

369

Figure 3: Schematic illustration of an analogy underlin-ing the role of within-pair similarity in the arithmeticanalogy test. At left, China + (Paris − France) pre-dicts Beijing in part for the “right” reason (offsets areparallel). At right, the offsets are not parallel, but Bei-jing is likely to be predicted, since its cosine similar-ity to China is high. The problem is worse than the2D schematic would suggests, since it can occur evenwhen the word vectors are normalized, and when theoffsets are quite long. China will be predicted if it isnot excluded, since it is also in the neighbourhood.

Perfectly parallel offsets would imply thatFrance and Paris differ on the same dimension asChina and Beijing—and on no other dimension—and similarly for the other offsets of the same kind(Canada−Ottawa, . . . ). Perfect parallelism is notnecessary for sim(oa, ob) to contribute to successon the analogy test. By offset concentration, wemean the degree to which, for an entire set of wordpairs, the offsets are parallel to one another (thus,concentrated in a single direction).

Once we take into account that offsets need notbe perfectly parallel, we must bear in mind thatpositive cosine similarity between offsets does notimply that the embedding space captures linguisticrelations. For example, recalling Schluter (2018),training embeddings to capture distributional reg-ularities may group words into linguistic classes—with, for example, country names such as France,China. . . , occupying a region of the space distinctfrom that of Paris, Beijijng. . . , due to distributionalcommonalities. This by itself could yield offsetconcentration: offsets would all come from a simi-lar origin, and go to a similar destination. However,sim(France-Paris,China-Beijing) would not nec-essarily be larger than sim(France-Beijing,China-Paris). We illustrate this issue in Figure 4.

In such an example, we might be able to assertthat the class of capital city names is distinct fromthe class of country names. But asserting that therelation capital-of is captured is a stronger prop-erty. It would require that moving from the France

Figure 4: Schematic illustration of an analogy un-derlining the pitfalls of offset concentration. At left,Paris − France appears nearly orthogonal to Bei-jing − China. At right, however, the offsets havemuch greater cosine similarity, simply because Parisand Beijing are far away. The similarity betweenParis − China and Beijing − France (dotted lines) isalmost as high. Unlike what might be suggested by the2D schema, this can arise easily even for normalizedword vectors.

vector in the direction of the common capital-ofvector lead to Paris and not to Beijing, nor to anyother capital city—or, at least, that it passes closerto Paris. We call this property pairing consistency.

4.1 Offset concentration

Figure 5 shows a histogram of the pairwise similar-ities between all pairs of BATS analogy offsets forthe pre-trained word2vec embeddings used above,grouped by broad BATS relation type. As above,these similarities are small but generally positive.We propose an offset concentration score:

OCS =1

N(N − 1)

∑i

∑j 6=i

oi · oj (6)

which is the mean similarity between normalizedoffsets within a single relation, where the nor-malized offset vector oi of a given word pair isoi =

a∗i−ai‖a∗i−ai‖

. We discuss in the appendix the linkbetween the OCS and the mean direction of theoffsets.

4.2 Pairing consistency

To measure the pairing consistency, we comparethe direction of the offsets for a given set of wordpairs against comparable shuffled offsets from thesame BATS relation. Shuffled offsets are generatedby holding the start words constant but permutingthe end words (avoiding true pairs). We expect trueoffsets to be more parallel than shuffled offsets.

We sample sets of shuffled offsets for each re-lation. For each set, as for the true offsets, we

370

Figure 5: Distribution of the cosine similarities between vector offsets for word2vec embeddings, grouped by broadrelation type. The Offset Concentration Score is the mean offset similarity for all pairs within a single relation (e.g.,country–capital).

compute all pairwise similarities between offsets.Rather than averaging similarities as in the OCS,we calculate the separation between distributions—the true offset similarities versus each set of shuf-fled offset similarities. We empirically constructthe receiver operating characteristic curve for lin-early separating true versus shuffled similarities(i.e., setting a criterion on the similarity dimen-sion above which offsets are true offsets and belowwhich offsets are shuffled offsets). We calculatethe area under the curve (AUC). The AUC will be1 for perfectly separable distributions, 0.5 for in-distinguishable distributions, and below 0.5 whentrue offsets are systematically less parallel than thebaseline. We average AUC over all shuffled sets.

For any set of offsets O , we refer to the set of itsoffset similarities as sim(O) = {oi ·oj , ∀(oi, oj) ∈O2, i < j}. For Ss a list of Ns sets of shuffledoffsets O′ (here Ns = 50), we define the followingpairing consistency score, where O are the trueoffsets:

PCS =1

Ns

∑O′∈Ss

AUC(sim(O), sim(O′)

)(7)

5 Exploration of word2vec

We perform a series of experiments to assess thepresence of linguistic regularities in the pre-trainedword2vec embeddings used above, and to show thebehavior of our new measures.

5.1 Random baselinesIn addition to the real BATS test, we apply ourmeasures to artificial analogy sets, which we con-struct to show no pairing consistency. We seek tojustify the use of pairing consistency, rather than di-rectly using offset concentration, by demonstrating

that offset concentrations can remain non-zero evenfor arbitrarily associated pairs. For each BATS re-lation, we construct ten of each of the followingtypes of permuted relations. We calculate the meanOCS and the mean PCS over the ten instances. Weexpect the mean PCS to be 0.5 in each case.

First, we construct sets of word pairs derivedfrom the BATS by permutation within-category:for each category, the end words are randomly re-assigned to start words. In other words, we con-struct categories for which shuffling should haveno effect (they are already shuffled). Comparingthe OCS to that of the real BATS also allows us tomeasure the effect of shuffling on the OCS.

If permuted word pairs can show positive off-set concentration simply because the start wordsand/or the end words are found in a coherent regionof the embedding space, then positive OCS shouldbe found in word pairs drawn from mismatchedcategories. For each pair of start/end word cate-gories among the BATS relations (for example, thenoun–plural reg relation maps the start categorysingular nouns to the end category regular pluralnouns), we randomly re-assign the end category(yielding, for example, singular nouns–past tenseverbs). We then randomly construct fifty start/endpairs. We compare mismatched categories fromwithin versus across broad BATS category types(inflectional morphology, derivational morphology,encyclopedic semantics, lexicographic semantics).

Finally, to assess the impact of the geometriccoherence of words within categories on the OCS,we construct pairs without respect for category:we compare word pairs with random start words,random end words, and random start and endwords, with ten different random word sets.

371

5.2 Offset concentration

Table 1 shows the OCS and PCS scores on real andrandomized BATS relations using word2vec. TheOCS scores for the real BATS sets are not close toone (as expected, given the results above).

The fully random set has an OCS of zero, con-firming that offset concentration is unlikely to oc-cur by chance, at least for these word vectors. Theother random analogy sets all show non-zero OCS.Some random sets have even higher OCS scoresthan the real BATS sets. Thus, as predicted, offsetconcentration can exist even when linguistic rela-tions are, by construction, not coded in any system-atic way. Even the random-start and random-endpairs show non-zero OCS,2 indicating that geo-metric coherence within one category is sufficientto give some degree of parallelism to the offsets.Surprisingly, the OCS is systematically lower forthe permuted within-category baseline than for themismatched-category baselines. Mismatched cat-egories may be further apart than matched cate-gories, reducing the angles between offsets overall.

These results have important consequences forhow we measure meaningful regularities in vectorembeddings. Not only is the arithmetic analogytest not fit to detect the presence of similar direc-tions among related pairs, even if it were, similardirections alone are not sufficient to deduce thatlinguistic regularities are coded. The test can bedominated by spurious within-pair similarity andby spurious offset concentration—which can ariseeven when matching against random words.

5.3 Pairing consistency

The real BATS sets show PCS above chance, con-trary to all random analogy sets, which show PCSof 0.5, as expected.3 The PCS scores are low for thesemantic (encyclopedic, lexicographic) relations.

In Section 3 above, we remarked that within-pairsimilarity is not completely irrelevant to analogies,as a higher within-pair similarity amplifies the com-ponent along the common direction. Here, the re-lations which show greater within-pair similarity(Figure 1) also show higher PCS.

2We observe a systematic difference between the randomstart and random end results, which seems to indicate a biasin the BATS analogy sets, with the end words being moresusceptible to forming clusters.

3As noted, these scores are the mean over ten replicationsof the random sets. Even where the mean is not precisely 0.5,it is always within one half-interquartile-range of 0.5.

6 Comparing word embeddings

We evaluate the pairing consistency on the BATStest for popular word embeddings. We includeGlove (Pennington et al., 2014), a purely distri-butional model similar to word2vec; two embed-dings making use of external semantic knowledge:dict2vec (Tissier et al., 2017) and ConceptNet Num-berbatch (Speer et al., 2017); and the static tokenembeddings of BERT (Devlin et al., 2019) andGPT-2 (Radford et al., 2019).

The left subfigure in Figure 6 presents the stan-dard 3COSADD analogy test accuracy (input wordsexcluded). Within broad relation types, we seesimilar results across embeddings, with a few ex-ceptions (the static GPT-2 embeddings are poorthroughout, but much better than the others onderivational morphology). Consistent with previ-ous results using the arithmetic analogy test, deriva-tional morphology and encyclopedic and lexicalsemantic relations obtain lower scores than inflec-tional morphology (Gladkova et al., 2016; Rogerset al., 2017).

The PCS scores are presented at right in Figure 6.The general picture is similar: there are systematicdifferences between relation types, and inflectionalmorphology tends to be well coded. However, con-trary to the conclusions of the standard analogy test,the PCS reveals that Numberbatch codes linguisticrelations better than the other embeddings, whichis more consistent with the evaluations presentedin Speer et al. (2017). The PCS also shows thatderivational morphology occupies an intermediateposition between inflectional morphology and se-mantics: word2vec, Glove, and Numberbatch infact code derivational morphology better, in thePCS sense, than encyclopedic semantic relations.The elevated analogy test performance of someembeddings on encyclopedic relations is likely anartefact of the kind of category-level geometric co-herence discussed in Section 5. As we show in theappendix in Table 2 , encyclopedic relations showhigher OCS overall than derivational morphologyrelations, in spite of lower PCS.

7 Contributions

We have made new arguments against the use ofthe standard arithmetic analogy test in the evalua-tion of word embeddings. We show in detail howregularities in vector offsets, which are the objectof primary interest for the analogy test, are, on theone hand, washed out by within-pair similarity, and,

372

Inflectional Derivational Encyclopedic LexicographicAnalogy set OCS PCS OCS PCS OCS PCS OCS PCSReal BATS .295 .851 .156 .679 .198 .559 .031 .539Permuted within-category .111 .500 .088 .500 .170 .500 .015 .500Mismatched category (within type) .147 .501 .120 .500 .260 .501 .093 .500Mismatched category (across type) .175 .500 .173 .500 .223 .500 .134 .500Random start .090 .500 .075 .500 .204 .499 .069 .500Random end .063 .500 .060 .500 .137 .501 .059 .499Random start and end .000 .500

Table 1: Offset Concentration Scores and Pairing Consistency Scores for the real and random baseline analogysets (higher is better; 0.5 is chance level for PCS). Random baseline scores are averaged across ten permutations.All scores are then averaged across BATS relations. Mismatched categories (across broad BATS relation type) aregrouped along the start type. Random start and end is not assigned to a BATS type, as all the words are drawnrandomly. Bold indicates the highest OCS score in a given column: we see that OCS is not always highest for realanalogies. Random baseline PCS scores are always within a half-IQR of 0.5.

Figure 6: Left: accuracy on the standard arithmetic analogy test, across embeddings, averaged across BATS re-lations (within the four broad types). Right: the same figure for the Pairing Consistency Score, starting at .5(chance-level).

on the other, insufficient to establish the existenceof linguistic regularities. We explain the previouslyobserved phenomenon that arithmetic analogy testsconsistently predict the input words.

We propose a new measure of the presence oflinguistic relations in word embeddings, the pair-ing consistency score, which measures the degreeto which offsets between related pairs are parallelabove chance, and the offset concentration score, acomplementary measure of the absolute degree ofparallelism. We show that a variety of word vectorsreally do capture linguistic regularities, in spite ofdistortions introduced by the arithmetic analogytest. We show that these distortions lead to spu-rious conclusions when comparing performance.Arithmetic analogy tests and loss functions basedon them are deprecated and should be replaced bydirect evaluation of pairing consistency.

Acknowledgments

This work was funded in part by the Euro-pean Research Council (ERC-2011-AdG-295810BOOTPHON), the Agence Nationale pour la

Recherche (ANR-17-EURE-0017 Frontcog, ANR-17-CE28-0009 GEOMPHON, ANR-10-IDEX-0001-02 PSL*, ANR-19-P3IA-0001 PRAIRIE 3IAInstitute, ANR-18-IDEX-0001 U de Paris, ANR-10-LABX-0083 EFL) and grants from CIFAR(Learning in Machines and Brains), Facebook AIResearch (Research Grant), Google (Faculty Re-search Award), Microsoft Research (Azure Creditsand Grant), and Amazon Web Service (AWS Re-search Credits).

ReferencesCarl Allen and Timothy Hospedales. 2019. Analo-

gies Explained: Towards Understanding Word Em-beddings. In Proceedings of the 36th InternationalConference on Machine Learning, volume 97 of Pro-ceedings of Machine Learning Research, pages 223–231, Long Beach, California, USA. PMLR.

Zied Bouraoui, Shoaib Jameel, and Steven Schock-aert. 2018. Relation Induction in Word Embed-dings Revisited. In Proceedings of the 27th Inter-national Conference on Computational Linguistics,pages 1627–1637, Santa Fe, New Mexico, USA. As-sociation for Computational Linguistics.

373

Dawn Chen, Joshua C. Peterson, and Thomas L. Grif-fiths. 2017. Evaluating vector-space models of anal-ogy. CoRR, abs/1705.04416.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, andKristina Toutanova. 2019. BERT: Pre-training ofDeep Bidirectional Transformers for Language Un-derstanding. In Proceedings of the 2019 Conferenceof the North American Chapter of the Associationfor Computational Linguistics: Human LanguageTechnologies, Volume 1 (Long and Short Papers),pages 4171–4186, Minneapolis, Minnesota. Associ-ation for Computational Linguistics.

Aleksandr Drozd, Anna Gladkova, and Satoshi Mat-suoka. 2016. Word Embeddings, Analogies, andMachine Learning: Beyond King - Man + Woman =Queen. In Proceedings of COLING 2016, the 26thInternational Conference on Computational Linguis-tics: Technical Papers, pages 3519–3530, Osaka,Japan, December 11-17.

Kawin Ethayarajh, David Duvenaud, and Graeme Hirst.2019. Towards Understanding Linear Word Analo-gies. In Proceedings of the 57th Annual Meetingof the Association for Computational Linguistics,pages 3253–3262, Florence, Italy. Association forComputational Linguistics.

Gregory Finley, Stephanie Farmer, and Serguei Pakho-mov. 2017. What Analogies Reveal about Word Vec-tors and their Compositionality. In Proceedings ofthe 6th Joint Conference on Lexical and Computa-tional Semantics (*SEM 2017), pages 1–11, Vancou-ver, Canada. Association for Computational Linguis-tics.

Anna Gladkova, Aleksandr Drozd, and Satoshi Mat-suoka. 2016. Analogy-Based Detection of Morpho-logical and Semantic Relations with Word Embed-dings: What Works and What Doesn’t. In Proceed-ings of the NAACL-HLT SRW, pages 47–54, SanDiego, California, June 12-17, 2016. ACL.

Omer Levy and Yoav Goldberg. 2014. Linguistic Reg-ularities in Sparse and Explicit Word Representa-tions. In Proceedings of the Eighteenth Confer-ence on Computational Natural Language Learning,pages 171–180, Ann Arbor, Michigan. Associationfor Computational Linguistics.

Tal Linzen. 2016. Issues in Evaluating SemanticSpaces Using Word Analogies. In Proceedingsof the First Workshop on Evaluating Vector SpaceRepresentations for NLP. Association for Computa-tional Linguistics.

Tomas Mikolov, Kai Chen, Greg Corrado, and JeffreyDean. 2013a. Efficient Estimation of Word Repre-sentations in Vector Space. In Proceedings of Inter-national Conference on Learning Representations(ICLR).

Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig.2013b. Linguistic Regularities in Continuous Space

Word Representations. In Proceedings of NAACL-HLT 2013, pages 746–751, Atlanta, Georgia, 9–14June 2013.

Denis Newman-Griffis, Albert Lai, and Eric Fosler-Lussier. 2017. Insights into Analogy Completionfrom the Biomedical Domain. In BioNLP 2017,pages 19–28, Vancouver, Canada,. Association forComputational Linguistics.

Malvina Nissim, Rik van Noord, and Rob van der Goot.2020. Fair Is Better than Sensational: Man Is to Doc-tor as Woman Is to Doctor. Computational Linguis-tics, 46(2):487–497.

Jeffrey Pennington, Richard Socher, and ChristopherManning. 2014. GloVe: Global Vectors for WordRepresentation. In Proceedings of the 2014 Con-ference on Empirical Methods in Natural LanguageProcessing (EMNLP), pages 1532–1543, Doha,Qatar. Association for Computational Linguistics.

Alec Radford, Jeffrey Wu, Rewon Child, David Luan,Dario Amodei, and Ilya Sutskever. 2019. LanguageModels are Unsupervised Multitask Learners. Ope-nAI Blog, 1(8):9.

Anna Rogers. 2019. On word analogies and negativeresults in NLP. Hacking Semantics.

Anna Rogers, Aleksandr Drozd, and Bofang Li. 2017.The (Too Many) Problems of Analogical Reasoningwith Word Vectors. In Proceedings of the 6th JointConference on Lexical and Computational Seman-tics (* SEM 2017), pages 135–148.

Natalie Schluter. 2018. The Word Analogy TestingCaveat. In Proceedings of the 2018 Conference ofthe North American Chapter of the Association forComputational Linguistics: Human Language Tech-nologies, Volume 2 (Short Papers), pages 242–246,New Orleans, Louisiana. Association for Computa-tional Linguistics.

Robyn Speer, Joshua Chin, and Catherine Havasi. 2017.ConceptNet 5.5: An Open Multilingual Graph ofGeneral Knowledge. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence,AAAI’17, page 4444–4451. AAAI Press.

Julien Tissier, Christophe Gravier, and AmauryHabrard. 2017. Dict2vec : Learning Word Em-beddings using Lexical Dictionaries. In Proceed-ings of the 2017 Conference on Empirical Methodsin Natural Language Processing, pages 254–263,Copenhagen, Denmark. Association for Computa-tional Linguistics.

Ekaterina Vylomova, Laura Rimell, Trevor Cohn, andTimothy Baldwin. 2016. Take and Took, Gaggleand Goose, Book and Read: Evaluating the Utilityof Vector Differences for Lexical Relation Learning.In Proceedings of the 54th Annual Meeting of the As-sociation for Computational Linguistics (Volume 1:Long Papers), pages 1671–1682, Berlin, Germany.Association for Computational Linguistics.

374

Figure 7: Distribution of the cosine similarity of vector offsets to the mean direction for the default word2vecembeddings for each category, then grouped by broad type. The mean similarity improves the similarity even foranisotropic offsets, as discussed. Variance changes seem category dependent.

Appendix A: The offsets’ mean direction

For a set of offsets, the OCS measures how concen-trated the hypercone of the offsets is. The center ofthis hypercone is the average normalized offset d,the unit vector defined by

d ∝∑i

a∗i − ai‖a∗i − ai‖

(8)

This vector has different interesting properties.First, we can note that it is the unit vector maximiz-ing the sum of the similarities of the offsets with it.With the normalized offsets oi=

a∗i−ai‖a∗i−ai‖

,

d = arg maxu,‖u‖=1

∑i

oi · u (9)

Furthermore, let’s define MSM, the mean simi-larity of d to the offsets. We can first prove that theMSM is equal to the norm of d before normaliza-tion:

MSM =1

N

∑j

sim(1N

∑i oi

‖ 1N

∑i oi‖

, oj)

=1

N·∑

i oi∑

j oj

‖∑

i oi‖=∥∥∥ 1

N·∑i

oi

∥∥∥(10)

Indeed, the average offset will have a norm closeto 1 if the offsets are higly similar, but close to 0if they are anisotropic. We can now go further andlink the MSM to the OCS.

1

N2

∥∥∥∑i

oi

∥∥∥2 =1

N2(∑i

o2i +∑i

∑j 6=i

oi · oj)

=1

N+

N(N − 1)

N2· OCS

MSM =

√1

N+

(N − 1)

N· OCS

(11)

This result shows that the MSM is a strict(parabolic) improvement to the OCS, with a com-putable baseline decreasing with N . For BATS,N = 50 (if all words are in the vocabulary), andthus the minimal MSM is

√1/50≈0.14 even for

anisotropic offsets (with an OCS = 0). This scoreshows that offset concentration can be quickly am-plified in unrealistic ways. Still, we can considerd as a close representation of the offsets’ relation.We display in Figure 7 the distribution of the simi-larities of the offsets to d.

Appendix B: Additional figures

375

Inflectional Derivational Encyclopedic LexicographicEmbedding Analogy Metrics Analogy Metrics Analogy Metrics Analogy Metrics

N H OCS PCS N H OCS PCS N H OCS PCS N H OCS PCSword2vec .686 .099 .295 .851 .158 .005 .156 .679 .198 .203 .198 .559 .063 .006 .031 .539Glove .645 .224 .345 .860 .167 .022 .237 .710 .281 .122 .255 .623 .080 .008 .004 .550dict2vec .503 .001 .099 .700 .127 .000 .079 .630 .162 .050 .213 .628 .030 .006 .024 .534Numberbatch .360 .226 .357 .924 .060 .029 .224 .805 .056 .066 .251 .674 .004 .006 .034 .552BERT tokens .568 .018 .217 .821 .288 .229 .178 .632 .041 .073 .151 .569 .002 .015 .016 .517GPT-2 tokens .119 .019 .097 .651 .359 .248 .270 .732 .014 .050 .071 .513 .003 .011 .011 .508

Table 2: Comparison of analogy test scores and our OCS and PCS metrics for different word embeddings, groupedby broad type. N designates the Normal analogy test scores, and H the Honest analogy test scores where theinput words are allowed to be predicted. OCS and PCS designate the Offsets Concentration Scores and PairwiseConsistency Scores. Bold means highest score for this type.

Figure 8: Decomposition of the analogy score if oa =ob (the score being therefore equal to 1). The similar valuesof b · (b + oa) and oa · oa indicates that the length of oa and b is similar, and thus that the offset length does notexplain the low value of oa ·ob in the real decomposition.