Proceedings of the Joint Workshop on Multiword Expressions and WordNet (MWE-WN 2019), pages 30–39Florence, Italy, August 2, 2019. c©2019 Association for Computational Linguistics

30

Learning to Predict Novel Noun-Noun Compounds

Prajit DharLeiden University

dharp@liacs.leidenuniv.nl

Lonneke van der PlasUniversity of Malta

lonneke.vanderplas@um.edu.mt

Abstract

We introduce temporally and contextually-aware models for the novel task of predictingunseen but plausible concepts, as conveyed bynoun-noun compounds in a time-stamped cor-pus. We train compositional models on ob-served compounds, more specifically the com-posed distributed representations of their con-stituents across a time-stamped corpus, whilegiving it corrupted instances (where head ormodifier are replaced by a random constituent)as negative evidence. The model capturesgeneralisations over this data and learns whatcombinations give rise to plausible compoundsand which ones do not. After training, wequery the model for the plausibility of auto-matically generated novel combinations andverify whether the classifications are accurate.For our best model, we find that in around 85%of the cases, the novel compounds generatedare attested in previously unseen data. An ad-ditional estimated 5% are plausible despite notbeing attested in the recent corpus, based onjudgments from independent human raters.

1 Introduction

Compounding is defined as the process of com-bining two or more lexemes to form a new con-cept (Bauer, 2017). For most compounds in En-glish, the first constituent is the modifier, whereasthe second is the head. The head usually deter-mines the class to which the compound belongs,whereas the modifier adds specialisation, e.g applecake is a type of cake. Compounding is thoughtof as one of the simplest forms of concept forma-tion1 as it involves use of elements that are alreadypart of the language and requires little or no mor-phological changes, particularly in English. Fromthe perspective of language acquisition, Berman

1We avoid the usage of “word formation” due to there be-ing no consensus on the definitions for both words and com-pounds (Bauer, 2017, chapter 2).

(2009) found that children acquired compoundingconstruction skills before the other forms of wordformation.

Comparatively little effort has been put into in-vestigating the productive word formation processof compounding computationally. Although com-pounding is a rather challenging process to modelas it involves concepts of compositionality andplausibility along with an intricate blend of seman-tic and syntactic processes, it is, in our view, one ofthe best starting points for modeling linguistic cre-ativity. In contrast to relatively more studied topicsin linguistics creativity, such as automatic poetrygeneration (Ghazvininejad et al., 2017), aesthet-ics are not involved. Moreover, compounding islimited to phrase level processes, as it involves acombination of known lexemes.

In general, the creative power of language hasbeen understudied in the field of natural languageprocessing (NLP). The main focus is indeed onprocessing, as the name suggests. Creative think-ing is a cognitive ability that fuels innovation.Therefore, the modelling and understanding of theunderlying processes of novel concept creationis relevant. Ultimately, we aim to create toolsthat enhance peoples ability to interface more cre-atively with large data sets, to build tools that findinspiration in data.

Our main contributions are the introduction ofa new task in NLP that sheds light on the ba-sic mechanisms underlying conceptual creativity;an automatic way of evaluating newly generatedlanguage; a temporally-aware neural model thatlearns what are plausible new conceptual combi-nations by generalising over attested combinationsand corrupted instances thereof.

31

2 Related Work

The related task of automatic novel compounddetection was introduced by Lapata and Las-carides (2003). Their aim is to distinguish rarenoun compounds from rare but nonce noun se-quences. The biggest difference between theirwork and ours is that while they identify exist-ing, albeit rare, and therefore possibly relativelynovel compounds in corpora, we predict unseen,and therefore novel compounds, in an absolutesense. Still, the overlap between the tasks makesthe work relevant. In their experiments, sur-face features, such as the frequency of the com-pound head/modifier, the likelihood of a word asa head/modifier, or the surface-grammatical con-text surrounding a candidate compound performalmost as well as features that are estimated onthe basis of existing taxonomies such as Word-Net. Although the semantic features they gath-ered from WordNet did not do very well, we be-lieve our distributional semantic features are morefine-grained. The simple statistical features thatdid well in distinguishing rare compounds fromnonce terms, would not be suitable in our scenario,where we try to generate novel, plausible com-pounds. We did however, follow their method-ology for the automatic extraction of noun-nouncompounds from corpora based on their PoS.

Keller and Lapata (2003) obtain frequencies forunseen bigrams (including noun-noun bigrams) incorpora using Web search engines and show evi-dence of the reliability of the web counts for natu-ral language processing, also by means of studyingthe correlation between the web counts and humanplausibility judgments. The unseen bigrams aregenerated in a random fashion from corpus data,as the aim is not to generate plausible combina-tions, but to overcome data sparseness by provid-ing counts from Web searches.

O Seaghdha (2010) uses topic models for se-lectional preference induction, and evaluates hismodels on the same data as Keller and Lapata(2003) outperforming previous work. As this worktries to predict the plausibility of unseen combi-nations, it is more closely related to our work.We are, however, first and foremost interestedin the temporal aspect of novel compound cre-ation, and therefore use a time-stamped corpus andtemporally-aware models. We also use this time-stamped corpus for evaluation, in addition to hu-man plausibility judgments.

We find a number of works in the related field ofcognitive science that focus on predicting humanacceptability or plausibility ratings with composi-tional distributional models. Vecchi et al. (2017)focus on novel adjective noun-phrases. They showthat the extent to which an adjective alters the dis-tributional representation of a noun it modifies isthe most significant factor in determining the ac-ceptability of the phrase. Gunther and Marelli(2016) are also concerned with predicting plausi-bility ratings, but focus on noun-noun compoundsinstead. The main difference between their workand ours is the fact that our systems are partly neu-ral and use slightly different features, and aim togenerate novel, plausible compounds that are eval-uated by checking for their existence in a futurecorpus, whereas they check for correlation withhuman plausibility ratings on a set of attested andcorrupted compounds. However, their careful in-vestigation of the different distributional seman-tic factors in their model have been very insight-ful for us and they inspired one of our systems.For example, they found that a higher relatednessbetween head and compound is associated withhigher plausibility. And the similarity betweenthe constituents is associated with a slightly higherplausibility as well. We used these features in oneof our models as well.

More recently, Marelli et al. (2017) presenteda data-driven computational system for com-pound semantic processing based on composi-tional distributional semantics and evaluate themodel against behavioral results concerning theprocessing of novel compounds. They find thatthe phenomena of relational priming and relationaldominance are captured by the compositional dis-tributional semantic models (cDSMs), whose pre-dictions pattern nicely with the results from the be-havioral experiments. Although this work provesthat the cDSM is psychologically real when itcomes to processing novel compounds, and wefind inspiration in the architecture of their model,their work is mainly aimed at modelling com-pound processing, whereas we are focusing oncompound prediction.

For the architecture of our model, we weremainly inspired by Van de Cruys (2014). Theproblem we are focusing on has many similar-ities with the task that their paper focuses on:predicting selectional preferences for unseen data.We adopted the architecture of the neural network

32

from this paper as well as the method to gener-ate negative data for training purposes, originallyproposed by Collobert and Weston (2008). Apartfrom the difference in the task we are trying to ad-dress, the main differences between their work andours is the fact that we are adding a temporal as-pect to the neural networks.

3 Novel Compound Prediction

In this paper, we address the task of novel com-pound prediction. Three models are created thatuse count-based distributed word representationsof known compound constituents to predict un-seen, but plausible compounds.

3.1 Intuitions and evaluation

In particular, we address the task of predictingnovel noun-noun (N-N) compounds: compoundsconsisting of two constituents that are both nouns.Our method is based on the generalisation powerof machine learning. We reason that by compress-ing the space in ways that are in line with distribu-tional patterns found in observed data, estimatesfor unobserved yet plausible combinations shouldbe close to the estimates gathered from attestedexamples. For example, if we have seen glass-bottom boats in corpora, but we have never seenthe combination glass canoe (a recent invention),we can infer from the similarity between the com-ponents of the compounds that a glass canoe couldbe a plausible compound even though it has neverbeen seen.

Evaluating plausibility prediction models fornovel combinations is non-trivial and previouswork has relied mainly on human judgments(Gunther and Marelli, 2016). We aim to find an au-tomatic evaluation method to ease parameter opti-misation and model comparison. To this end, weuse a time-stamped corpus divided into decades.The last decade is used for testing, with the previ-ous decades used for training the models. This al-lows us to check whether the novel generated com-pounds are good predictions of compounds thatmight emerge in the future. Because the future ex-tends beyond the last decade of our corpus, andthe results from our automatic evaluation are pes-simistic, we ask human judges to rate the plausi-bility of a sample of automatically generated novelcompounds that are not attested in the last decade(see Sections 7 and 9).

3.2 Two Aspects for CompoundingWe hypothesise that in order to computationallymodel the phenomenon of compounding, we needour models to be both contextually-aware andtemporally-aware, which we explain in detail inthe subsequent section.

3.2.1 The Contextual AspectPsycholinguistic research on N-N compounds inDutch that seemed to suggest that constituentssuch as -molen ‘-mill’ in pepermolen ‘peppermill’are separately stored as abstract combinatorialstructures rather than processed on-line andunderstood on the basis of their independentconstituents: molen ‘mill’ (De Jong et al., 2002).For English open compounds, a similar effectwas found for the right constituent. To test ifthis phenomenon has an effect in the processof generating novel compounds, we decided toexperiment with two types of contextual contexts:CompoundCentric and CompoundAgnostic.

CompoundAgnostic: These are the standardwindow-based contexts used in vector-based rep-resentations of words. We capture the distribu-tional vectors of the words, irrespective of whetherlexemes are found as constituents of a compoundor as simple standalone words (see Figure 1b).

CompoundCentric: To the best of our knowl-edge, distributional models that are sensitive tothe role a lexeme plays in a compound have notbeen tested before. Here we capture the distri-butional vectors of words based on their usageas constituents in a compound. So the word millwould have different representations, dependingon its role in a compound. In Figure 1a, we showan example context that mill gets as a head, and anexample context it gets as a modifier.

3.2.2 The Temporal AspectPrevious works such as Hamilton et al. (2016)have shown that meanings of certain words changeover time. The same can be observed for com-pounds such as melting pot. The meaning of melt-ing pot deviated from its original meaning (”Ablast furnace”) to its current meaning, that of a so-ciety where people from many different culturesare living together, and assimilating into a cohe-sive whole. To test if time does impact our task,we envision two settings for our models:

DecadeCentric: In this setting, we empha-sise the temporal aspect by collecting counts of

33

Head context

context

The red pepper︸ ︷︷ ︸Modifier

mill︸︷︷︸Head

context

is on the shelf .

Modifier context

context

The new mill︸︷︷︸Modifier

context

stone︸ ︷︷ ︸Head

has started .

(a)

context

The hugemill︸︷︷︸Word

context

is at the lake .

context

The red peppermill︸︷︷︸Word

context

is on the shelf .

context

The newmill︸︷︷︸Word

context

stone has started .

(b)

Figure 1: Contexts for (a) CompoundCentric and (b) CompoundAgnostic aspects

individual compounds and their constituents perdecade. We reason that knowing about the us-age trend of the constituents of a compound mighthelp to predict which constituents will be com-bined next. For example, if a certain word is trend-ing, you would expect it to crop up in novel com-binations.

DecadeAgnostic: To test if our intuition aboutthe temporal aspect indeed holds true, we also col-lect the counts of the individual compounds andtheir constituents without any temporal informa-tion.

4 System Overview

Figure 2 shows the system overview for our twomain models. The two aspects explained in theprevious section are clearly visible as distinctroutes in the system overview. From the GoogleNgram Corpus (Michel et al., 2010), distribu-tional vectors that are either CompoundAgnosticor CompoundCentric are collected2. The countsare either collected per decade (DecadeCentric)or without any temporal information (DecadeAg-nostic). The vectors then undergo dimensional-ity reduction (to 300 dimensions) using singular-value decomposition (SVD), after which they areeither directly input to our two semantic mod-els (DecadeAgnostic), or passed through a Longshort-term memory (LSTM) model (DecadeCen-tric) before they are input to the two semanticmodels. The reason for doing so is that it makesit easier to compare the models for each aspect.The LSTM we use takes a sequence of constituentsrepresentations for each decade as input and re-

2Even though the CompoundAgnostic aspect returnsphrases and words instead of actual compounds and con-stituents, respectively, we refer to them as same for simplic-ity.

Google Ngram Corpus

Phrases

Words

Compounds Heads

Modifiers

CompoundAgnostic CompoundCentric

SVD

LSTM

DecadeCentric

DenseConstituentEmbeddings

DecadeAgnostic

DSM NNM

Figure 2: System Overview for the DSM and NNMmodels

34

turns a single representation for the constituentthat ideally encompasses its entire history. In thefollowing subsections, we will provide more de-tails on the data processing we performed.

5 Data

We will describe how we collected our data, pre-processed it and how we generated negative datafor the classifier.

5.1 Data Collection

We constructed four different datasets based on theaforementioned aspects from the Google BooksNgram corpus (Michel et al., 2010). This corpusdescribes the usage of words and phrases over aperiod of five centuries. Information of around6% of all books in eight languages was gatheredusing Optical Character Recognition (OCR). Forour study, we focused on the extraction of Englishunigrams and bigrams from the corpus and on 20decades (1800s to 1990s), with the final decade(2000s) only used to collect the newly generatedcompounds. The corpus was tagged using a Con-ditional Random Field (CRF) based tagger (Laf-ferty et al., 2001), which has an error rate of 1-5%.The universal PoS tagset (Petrov et al., 2011) wasused to perform the PoS tagging.

5.2 Data Pre-processing

Similar to Lapata and Lascarides (2003), we madesure that our compounds are not surrounded bynouns, in order to avoid extracting parts of com-pounds that have more than two constituents.Also, constituents and compounds containing non-alphabetic characters are excluded from the exper-iments. For each compound and its constituents, acontext window of 3 words is used (i.e. 3 words oneither side of the target word), in order to retrievetheir distributional representations. We were ini-tially limited to a 2-word context window size dueto the nature of the Google Ngram 5-gram cor-pus. However with the use of a sliding windowapproach, this was increased to a window size of3. A bigram (say the compound water cycle) couldoccur in four different positions in the 5-grams (1-2, 2-3, 3-4 and finally 4-5). The contexts for eachof these positions is then captured.

We only consider the top 50k most commonlyoccurring nouns, adjectives, verbs and adverbs tobe candidate contexts. Finally, the heads are lem-matised and converted to lowercase. A compound

is considered to be novel if it only exists in the fi-nal decade (the 2000s) and has a frequency countof atleast 3. We chose the cut off count of 3, so asto capture most rare and plausible compounds butat the same time eliminate hapax legomena (termsthat occur only once).

5.3 Negative Data GenerationIn order to train the classifiers for the task of pre-dicting plausible novel compounds, we need bothpositive and negative data. The positive classfor our models is comprised of the modifier andhead of compounds that were newly created in thedecade of the 2000s. Note that the decade 2000s isnot used for our training purposes, but only usedto extract the positive examples. In the absenceof an attested negative class, i.e. compounds thatare implausible, we used the strategy from Col-lobert and Weston (2008) to generate our own neg-ative class. This class is made up of corrupt tuples,that are constructed by randomly replacing one ofthe constituents in the tuple (m,h) with a corre-sponding constituent (heads are replaced by headsand modifiers by modifiers) from another attestedcompound.

We then have two scenarios: the CorruptHeadand the CorruptModifier scenarios. For a Cor-ruptHead scenario, the head h is replaced by ran-domly selected head h′ with the modifier remain-ing the same. Similarly, for a CorruptModifierscenario, we replace the modifier m with m′ (seeTable 1).

Disambiguation Task The models have the taskof disambiguating the attested tuples from theircorrupted counterparts. The purpose of allowingthese two scenarios is to test whether corruptingthe head leads to better negative data points, thancorrupting the modifier. We also make sure thatnone of the corrupted tuples from the aforemen-tioned procedures results in the generation of anovel or a previously existing compound.

6 Semantic Models

In Figure 2, we show that the distributional vec-tors, be they CompoundAgnostic or not and bethey DecadeAgnostic or not are fed into two dis-tinct semantic models. Note that at this point thedistributional vectors are turned into dense repre-sentations. We experimented with one additionalmodel, the Distributional Feature Model (DFM),that uses sparse embeddings for constructing its

35

CorruptHead Novel Compound CorruptModifierm h′ m h m′ h

water fox water absorption blanket absorptionpitch minister ← pitch accent → cement accentgene psychiatry gene sequence dolphin sequence

Table 1: Generation of negative examples using the CorruptHead and CorruptModifier scenarios

features. The other two models: the Distribu-tional Semantic Models (DSM) and Neural Net-work Model (NNM) use dense embeddings. 3

6.1 Distributional Feature Models

In the DFM, for each compound, distributionalsemantic features are constructed. The raw fre-quency counts were used to construct the featuresas weighting measures such as PPMI worsened thefinal results. The first three are adopted from In-formation Theory and commonly used to find col-locations between words (for more detail see Man-ning and Schutze (1999)) -

1. Positive Pointwise Mutual InformationPPMI: A variation of the Pointwise MutualInformation (PMI) where the negative PMIvalues are replaced by 0’s. PPMI is preferredover PMI as it has been shown to outperformPMI on semantic similarity tasks (Bullinariaand Levy, 2007). The PPMI for a compoundcomp and its two constituents m and h isdefined as

PPMI(comp) = max(log2P (comp)

P (m)P (h), 0),

(1)where P (comp) is the probability of both mand h occurring together (i.e. the compounditself).

2. Log likelihood-ratio LLR: PPMI scores arebiased towards rare collocations as they as-sign rare words with rather high PMI val-ues. To overcome this bias and to incorporatethe frequency counts of the constituents, thelog likelihood ratio is used as another feature,similar to Lyse and Andersen (2012).

3. Local Mutual Information LMI: It is anothermetric that tries to overcome the bias of

3All the implementation details are provided inthe GitHub repository https://github.com/prajitdhar/Compounding.

PPMI and does so by comparing the proba-bility of observing m and h together with theprobability of observing the two by chance:

LMI(comp) = P (m) · P (h) · P (comp)P (m)P (h)

(2)

The three features below relate to the calcula-tion of the similarity between the three com-ponents of the compound. The similarity be-tween any two target words is defined by thecosine similarity between their vectors.

4. Similarity between the Compound and itsConstituents sim-with-head and sim-with-mod: The similarity between a compoundcomp and a constituent c is defined as:

cos(−−−→comp,−→c ) =−−−→comp · −→c‖−−−→comp‖‖−→c ‖

, (3)

where −−−→comp and −→c are the vector represen-tation of comp and c, respectively.

5. Similarity between the Constituents: Thesimilarity between the constituents of a com-pound is computed as well. Gunther andMarelli (2016) and Lynott and Ramscar(2001) have found this score to be useful indiscerning plausible compounds. Formally,the similarity between a modifier mod and ahead head is defined as:

cos(−−→mod,

−−→head) =

−−→mod ·

−−→head

‖−−→mod‖‖

−−→head‖

, (4)

where−−→mod and

−−→head are the vector represen-

tation of mod and head, respectively.

Finally to retrieve the features for the constituents,the distributional features of the compounds are

36

averaged. We also collect the standard devia-tion of each feature, so as to get the best ap-proximation of the distribution of the original val-ues. The 6 constituent features (and 120 for theDecadeCentric aspect as there are 20 decades) arethen concatenated to represent the compound. Soin total, we have 12 features (and 240 for theDecadeCentric aspect) as input to the DFM, whichuses the stochastic gradient boosting model (XG-Boost, Chen and Guestrin, 2016) to perform thesupervised learning. The tree-based models weretrained using logistic regression as the loss func-tion. The following parameters were then tunedand set as follows : learning rate = 0.1, maximumtree depth = 3, number of estimators = 100, mini-mum child weight = 6, sub-sample ratio = 0.5, γ =0, α = 0.05 and β = 1.

6.2 Neural Network Model

As shown by Van de Cruys (2014) and Tsubakiet al. (2016), neural networks have been shown tobe successful in composing a representation of aphrase or sentence. Since we expect our novelcompounds to be compositional in nature, i.e.its representation could be derived from its con-stituents’ representations, a compositional neuralnetwork should be able to discriminate betweenplausible candidate compounds and their nonsen-sical counterparts.

A candidate compound is represented by itsmodifier and head:

x = [im, jh], (5)

where i and j are the vectors representations ofthe modifier m and head h, respectively, and theresultant composed vector x serves as the input tothe neural network.The vector x is then the inputto the NNM.

The architecture of the NNM is similar to thetwo-way selectional preferences model from Vander Cruys (2014) and it comprises of a feed-forward neural network with one hidden layer asshown in Figure 3.

An element-wise activation function rectifiedlinear unit (ReLU ) is used in the hidden layer:

a = f(W1x+ b1), (6)

where a is the activation function of the hiddenlayer with H nodes, f(.) performs an element-wise ReLU function and W1 and b1 are the

im

jh

x

W1

W2

a

y

Plausible

Not Plausible

Figure 3: Architecture of NNM

weights and bias of the first layer, respectively. Inthe end, NNM generates a plausibility score y:

y =W2 a, (7)

where W2 is the weight matrix for the final layer.The neural network was trained for 50 epochs withstochastic gradient descent (SGD) used for opti-mization. A batch size of 100 was chosen and thehidden layer was set to H = 300 nodes.

6.3 Distributional Semantic ModelsIn order to discern if a compositional neural net-work is indeed required for our disambiguationtask, we also implement Distributional SemanticModels (DSMs) as a baseline. Since both models(DSMs and NNM) would be using the same con-stituent embeddings for all four aspects, the finalresults would help us answer this question. Sim-ilar to DFM, we concatenate the constituent fea-tures (here embeddings) which are then taken asinput to a gradient boosting model, to predict theplausibility.

7 System Evaluation

We evaluate our overall system in two ways: onthe basis of corpus data, and by means of humanjudges, which we cover in Section 9.

For the automatic evaluation of our system, wemeasure how many of the compounds it predictsare attested among the previously unseen com-pounds in the last decade, the 2000s. The 20decades prior to the 2000s (1800s to 1990s) areused to train the models. The constituents that ex-ist only in the decade of the 2000s are thereforeexcluded from the training phase. This way wemake sure that the prediction of novel entities is

37

dependent only on information derived from theprior decades.

8 Results on Corpus Data

Since all the datasets were equally balanced, weonly report the accuracy scores for each of themodels. We control the randomness of the nega-tive data generation and run all our semantic mod-els using 10 different datasets. Each of thesedatasets consists of the same attested novel com-pounds, along with a different negative class. Theresults of all the models in our experiments areshown in Table 2.

The NNM produces by far the best results. Itattains an accuracy of 85% when the Compound-Centric and DecadeCentric aspects are observedand the heads are corrupted (CorruptHead) in or-der to generate negative data. This means that for85% of the compounds generated, the model isable to correctly classify the compound as plau-sible or implausible. The accuracy in this case issolely determined by the fact that the novel com-pound is found in the last decade (that is excludedfrom training). We will show in a separate evalua-tion (see Section 9) with human judges that thisscore, although already quite impressive is pes-simistic, because some of the novel compoundspredicted by the model are not found in the lastdecade, but still plausible, according to humanjudges.

Furthermore, the following observations can bemade :

Overall, the models that are sensitive to the tem-poral aspect (DecadeCentric), outperform modelsthat do not take the temporal aspect into account(DecadeAgnostic). This underlines our hypothesisthat the temporal aspect is crucial when modellingthe process of compounding.

Also, our hypothesis that distributional mod-els should be sensitive to the fact that lexemes aspart of a compound differ in meaning from lex-emes that appear as standalone words seems tohold true. In general, models that observe theCompoundCentric aspect, perform better. Lastly,corrupting the head to generate negative evidenceseems a better alternative than corrupting the mod-ifier. This is to be expected as the head determinesmost of the compound’s meaning. Replacing thehead with another head has a higher probabilityof generating an implausible compound than re-placing the modifier. Implausible compounds are

needed to generate negative evidence.

9 Human Evaluation

Evaluation on corpus data does not guarantee fullcoverage. In other words, if a novel compoundgenerated by the system is not found in the con-temporary corpus this does not mean per se thatthe compound is not plausible. The compoundmight be plausible but not yet ’invented’. Wetherefore also ran a small-scale manual annotation.

Taking our best model, which was NNM un-der CompoundCentric and DecadeCentric aspects,a subset of plausible compounds that were pre-dicted by our system (but not found in test corpusand hence counted as incorrect) were annotated byhuman judges. Following the annotation guide-lines of Graves et al. (2013), each annotator wasasked to rate each candidate compound between 0(makes no sense) and 4 (makes complete sense).250 plausible compounds were annotated in total.

Each candidate compound was evaluated by atleast three annotaters. Table 3 shows some of theannotation results. We see that compounds such asart direction and service ramp, that are predictedby the system, but not found in the decades 2000s,is deemed plausible by the annotators. In fact, wefound that around 5% of the test data set was rated3 or higher, on average, by the annotators, indicat-ing that we cannot just rely on a corpora for theevaluation of the novel compound predictor, andthat the accuracies given in Table 2 are pessimistic.

10 Conclusions and Future Work

We propose a method for the task of novel com-pound prediction. We show that this task canbe modeled computationally, and that our mod-els need to be both temporally and contextuallyaware, in order to properly model compounding.The evaluation method we proposed that uses acontemporary corpus to evaluate the novel com-pound predicted, provides an objective and cheapalternative to evaluation with human judges. In aseparate evaluation, we show that the latter pro-vide more optimistic results.

Although previous work has shown correlationsbetween human plausibility judgments on unseenbigrams and frequencies in larger corpora (Kellerand Lapata, 2003), we would like to study thecorrelation between human plausibility judgmentsand occurrence in the last decade, to rigorouslytest the viability of the evaluation method. We

38

DecadeCentricCompoundCentric CompoundAgnostic

Model CorruptHead CorruptMod CorruptHead CorruptMod

DFM 71.57± 0.31 68.76± 0.2 70.95± 0.35 69.29± 0.36DSM 68.18± 0.33 64.77± 0.26 67.05± 0.71 64.03± 0.35NNM 84.69± 0.33 84.55± 0.46 74.32± 0.63 76.48± 0.56

DecadeAgnosticCompoundCentric CompoundAgnostic

Model CorruptHead CorruptMod CorruptHead CorruptMod

DFM 69.17± 0.24 66.69± 0.25 69.67± 0.39 67.33± 0.27DSM 68.26± 0.43 65.04± 0.34 67.52± 0.58 65.04± 0.34NNM 82.92± 0.2 82.54± 0.4 72.38± 0.92 75.02± 0.57

Table 2: Results of the Semantic Models, represented with accuracy and the standard deviation

Compound Plausibility rating

Service ramp 4Art direction 3.34Resource companion 2Funeral fish 0

Table 3: Human evaluations (average plausibility rat-ings) for compounds that are non-attested in corpus

would also like to take the graded nature of theplausibility judgments into account when evaluat-ing our models.

In addition, we would like to experiment withother models such as cDSMs and other neural net-work architectures. Our current system uses arather simple LSTM architecture to encode tem-poral information into one representation, andprior tests have shown that enhanced neural net-work architectures such as Schuster and Paliwal(1997) and Raffel and Ellis (2015) that use bidirec-tional LSTMS and attention-based networks, re-spectively, are better at encoding representation.

Lastly, would like to cover a more diverse set ofcompounds in future work. Our experiments arecurrently limited to collecting N-N compounds.In subsequent experiments, we aim to add closedcompounds and compounds separated by a hy-phen, as well as compounds that are composed ofother parts of speech, such as adjective-noun com-pounds.

Acknowledgements

We would like to thank the anonymous review-ers for their valuable comments and also thank Ja-

nis Pagel for providing feedback towards the laterstages of this research.

ReferencesLaurie Bauer. 2017. Compounds and Compounding.

Cambridge Studies in Linguistics. Cambridge Uni-versity Press.

Ruth Berman. 2009. Children’s acquisition of com-pound constructions. In R. Lieber and P. Stekauer,editors, Handbook of Compounding, pages 298–322. Oxford University Press.

John A. Bullinaria and Joseph P. Levy. 2007. Ex-tracting semantic representations from word co-occurrence statistics: A computational study. Be-havior Research Methods, 39(3):510–526.

Tianqi Chen and Carlos Guestrin. 2016. XGBoost: Ascalable tree boosting system. In Proceedings of the22nd ACM SIGKDD International Conference onKnowledge Discovery and Data Mining, KDD ’16,pages 785–794, New York, NY, USA. ACM.

Ronan Collobert and Jason Weston. 2008. A unifiedarchitecture for natural language processing: Deepneural networks with multitask learning. In Pro-ceedings of the 25th International Conference onMachine Learning, ICML ’08, pages 160–167, NewYork, NY, USA. ACM.

Tim Van de Cruys. 2014. A neural network approachto selectional preference acquisition. In Proceed-ings of the 2014 Conference on Empirical Methodsin Natural Language Processing, EMNLP 2014, Oc-tober 25-29, 2014, Doha, Qatar, A meeting of SIG-DAT, a Special Interest Group of the ACL, pages 26–35.

Nivja H. De Jong, Laurie B. Feldman, RobertSchreuder, Matthew Pastizzo, and R. HaraldBaayen. 2002. The processing and representation of

39

dutch and english compounds: peripheral morpho-logical and central orthographic effects. Brain andLanguage, 81:555–67.

Marjan Ghazvininejad, Xing Shi, Jay Priyadarshi, andKevin Knight. 2017. Hafez: an interactive poetrygeneration system. In Proceedings of ACL 2017,System Demonstrations, pages 43–48, Vancouver,Canada. Association for Computational Linguistics.

William W. Graves, Jeffrey R. Binder, and Mark S. Sei-denberg. 2013. Noun–noun combination: Meaning-fulness ratings and lexical statistics for 2,160 wordpairs. Behavior Research Methods, 45(2):463–469.

Fritz Gunther and Marco Marelli. 2016. UnderstandingKarma Police: The Perceived Plausibility of NounCompounds as Predicted by Distributional Modelsof Semantic Representation. PLoS ONE, 11(10):1–36.

William L. Hamilton, Jure Leskovec, and Dan Jurafsky.2016. Diachronic word embeddings reveal statisti-cal laws of semantic change. In Proceedings of the54th Annual Meeting of the Association for Compu-tational Linguistics (Volume 1: Long Papers), pages1489–1501. Association for Computational Linguis-tics.

Frank Keller and Mirella Lapata. 2003. Using the webto obtain frequencies for unseen bigrams. Coputa-tional Linguistics, 29(3):459–484.

John D. Lafferty, Andrew McCallum, and FernandoC. N. Pereira. 2001. Conditional random fields:Probabilistic models for segmenting and labeling se-quence data. In Proceedings of the Eighteenth Inter-national Conference on Machine Learning, ICML’01, pages 282–289, San Francisco, CA, USA. Mor-gan Kaufmann Publishers Inc.

Mirella Lapata and Alex Lascarides. 2003. Detectingnovel compounds: The role of distributional evi-dence. In Proceedings of the Tenth Conference onEuropean Chapter of the Association for Computa-tional Linguistics - Volume 1, EACL ’03, pages 235–242, Stroudsburg, PA, USA. Association for Com-putational Linguistics.

Dermot Lynott and Michael Ramscar. 2001. Can wemodel conceptual combination using distributionalinformation? In Proceedings of the 12th Irish Con-ference on Artificial Intelligence and Cognitive Sci-ence, pages 1–10.

Gunn Inger Lyse and Gisle Andersen. 2012. Colloca-tions and statistical analysis of n-grams: Multiwordexpressions in newspaper text. Exploring newspa-per language: Using the web to create and inves-tigate a large corpus of modern Norwegian,, pages79–110.

Christopher D. Manning and Hinrich Schutze. 1999.Foundations of Statistical Natural Language Pro-cessing. MIT Press, Cambridge, MA, USA.

Marco Marelli, Christina L. Gagn, and Thomas L.Spalding. 2017. Compounding as Abstract Oper-ation in Semantic Space: Investigating relationaleffects through a large-scale, data-driven computa-tional model. Cognition, 166:207–224.

Jean-Baptiste Michel, Yuan Kui Shen, Aviva PresserAiden, Adrian Veres, Matthew K. Gray, TheGoogle Books Team, Joseph P. Pickett, Dale Hol-berg, Dan Clancy, Peter Norvig, Jon Orwant, StevenPinker, Martin A. Nowak, and Erez LiebermanAiden. 2010. Quantitative analysis of culture usingmillions of digitized books. Science.

Diarmuid O Seaghdha. 2010. Latent variable mod-els of selectional preference. In Proceedings of the48th Annual Meeting of the Association for Compu-tational Linguistics, pages 435–444, Uppsala, Swe-den. Association for Computational Linguistics.

Slav Petrov, Dipanjan Das, and Ryan T. McDonald.2011. A universal part-of-speech tagset. CoRR,abs/1104.2086.

Colin Raffel and Daniel P. W. Ellis. 2015. Feed-forward networks with attention can solvesome long-term memory problems. CoRR,abs/1512.08756.

Mike Schuster and Kuldip K. Paliwal. 1997. Bidirec-tional recurrent neural networks. IEEE Trans. Sig-nal Processing, 45:2673–2681.

Masashi Tsubaki, Kevin Duh, Masashi Shimbo, andYuji Matsumoto. 2016. Non-linear similarity learn-ing for compositionality. In Proceedings of theThirtieth AAAI Conference on Artificial Intelligence,AAAI’16, pages 2828–2834. AAAI Press.

Eva Maria Vecchi, Marco Marelli, Roberto Zampar-elli, and Marco Baroni. 2017. Spicy adjectives andnominal donkeys: Capturing semantic deviance us-ing compositionality in distributional spaces. Cog-nitive Science, 41(1):102–136.