Proceedings of the Second BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 213–223Florence, Italy, August 1, 2019. c©2019 Association for Computational Linguistics

213

An Analysis of Source-Side Grammatical Errors in NMT

Antonios AnastasopoulosLanguage Technologies Institute

Carnegie Mellon Universityaanastas@cs.cmu.edu

Abstract

The quality of Neural Machine Translation(NMT) has been shown to significantly de-grade when confronted with source-side noise.We present the first large-scale study of state-of-the-art English-to-German NMT on realgrammatical noise, by evaluating on sev-eral Grammar Correction corpora. We presentmethods for evaluating NMT robustness with-out true references, and we use them for ex-tensive analysis of the effects that differentgrammatical errors have on the NMT output.We also introduce a technique for visualizingthe divergence distribution caused by a source-side error, which allows for additional insights.

1 Introduction

Neural Machine Translation (NMT) has becomethe de facto option for industrial systems inhigh-resource settings (Wu et al., 2016; Has-san Awadalla et al., 2018; Crego et al., 2016)while dominating public benchmarks (Bojar et al.,2018). However, as several works have shown,it has a notable shortcoming (among others, seeKoehn and Knowles (2017) for relevant discus-sion) in dealing with source-side noise, duringboth training and inference.

Heigold et al. (2018) as well as Belinkov andBisk (2018) pointed out the degraded performanceof character- and subword-level NMT modelswhen confronted with synthetic character-levelnoise –like swaps and scrambling– on French,German, and Czech to English MT. Belinkov andBisk (2018) and Cheng et al. (2018) also studiedsynthetic errors from word swaps extracted fromWikipedia edits. Anastasopoulos et al. (2019) fo-cused on a small subset of grammatical errors (ar-ticle, preposition, noun number, and subject-verbagreement) and evaluated on English-to-Spanishsynthetic and natural data.

However, no previous work has extensivelystudied the behavior of a state-of-the-art (SOTA)model on natural occurring data. Belinkov andBisk (2018) only trained their systems onabout 200K parallel instances, while Heigoldet al. (2018) and Anastasopoulos et al. (2019)trained on about 2M parallel sentences from theWMT’16 data. Importantly, though, none of themutilized vast monolingual resources through back-translation, a technique that has been consistentlyshown to lead to impressively better results (Sen-nrich et al., 2016a).

In this work, we perform an extensive analysisof the performance of a state-of-the-art English-German NMT system, with regards to its robust-ness against real grammatical noise. We proposea method for robustness evaluation without gold-standard translation references, and perform ex-periments and extensive analysis on all availableEnglish Grammar Error Correction (GEC) cor-pora. Finally, we introduce a visualization tech-nique for performing further analysis.

2 Data and Experimental Settings

To our knowledge, there are six publicly availablecorpora of non-native or erroneous English that areannotated with corrections and which have beenwidely used for research in GEC.

The NUS Corpus of Learner English (NUCLE)contains essays written by students at the NationalUniversity of Singapore (Dahlmeier et al., 2013).It has become the main benchmark for GEC, as itwas used in the CoNLL GEC Shared Tasks (Nget al., 2013, 2014). The Cambridge Learner Cor-pus First Certificate in English FCE corpus1 (Yan-nakoudakis et al., 2011) consists of essays col-lected from learners taking the Cambridge Assess-ment’s English as a Second or Other Language

1We use the publicly available portion.

214

(ESOL) exams.2 The Lang-8 corpus (Tajiri et al.,2012) was harvested from user-provided correc-tions in an online learner forum. Both have alsobeen widely used for the GEC Shared Tasks. An-other small corpus developed for evaluation pur-poses is the JHU FLuency-Extended GUG corpus(JFLEG) (Napoles et al., 2017) with correction an-notations that include extended fluency edits ratherthan just minimal grammatical ones. The Cam-bridge English Write & Improve (W&I) corpus(Andersen et al., 2013) is collected from an onlineplatform where English learners submit text andprofessional annotators correct them, also assign-ing a CEFR level of proficiency (of Europe. Coun-cil for Cultural Co-operation. Education Commit-tee. Modern Languages Division, 2001). Lastly,we use a portion of the LOCNESS corpus,3 a col-lection of essays written by native English speak-ers. 50 essays from LOCNESS were annotated byW&I annotators for grammatical errors, so we willjointly refer to these two corpora as WI+loc.

All datasets were consistently annotated for er-rors with ERRANT (Bryant et al., 2017), an auto-matic tool that categorizes correction edits.3 Thisallows us to consistently aggregate results andanalysis across all datasets.

2.1 Notation and Experimental SettingsThroughout this work, we use the following nota-tions:

• x: the original, noisy, potentially ungrammat-ical English sentence. Its tokens will be de-noted as xi.• x: the English sentence with the correction

annotations applied to the original sentencex, which is deemed fluent and grammatical.Again, its tokens will be denoted as xi.• y: the output of the NMT system when x is

provided as input (tokens: y j).• y: the output of the NMT system when x is

provided as input (tokens: y j).

For the sake of readability, we use the termsgrammatical errors, noise, or edits interchange-ably. In the context of this work, they will alldenote the annotated grammatical errors in thesource sentences (x). We also define the numberof errors, or the amount of noise in the source, to

2https://www.cambridgeenglish.org/3nucle, Lang8, fce, and WI+loc are pre-annotated with

ERRANT for the BEA 2019 GEC Shared Task. We also an-notated jfleg.

be equivalent to the number of annotated neces-sary edits that the source x requires to be deemedgrammatical (x), as per standard GEC literature.

The main focus of our work is the performanceanalysis of the NMT system, so our experimentaldesign is fairly simple. We use the SOTA NMTsystem of Edunov et al. (2018) for translating boththe original and the corrected English sentencesfor all our GEC corpora.4 The system achievedthe best performance in the WMT 2018 evalua-tion campaign, using an ensemble of 6 deep trans-former models trained with slightly different back-translated data.5

3 Evaluating NMT Robustness withoutReferences

When not using human judgments on output flu-ency and adequacy, Machine Translation is typ-ically evaluated against gold-standard referencetranslations with automated metrics like BLEU(Papineni et al., 2002) or METEOR (Banerjee andLavie, 2005). However, in the case of GEC cor-pora, we do not have access to translations – onlymonolingual data (potentially with ungrammati-calities) and correction annotations.6 Quality Es-timation for MT also operates in a reference-lesssetting (see Specia et al. (2018) for definitions andan overview of the field) and is hence very relatedto our work, but is more aimed towards towardspredicting the quality of the translation. Our goalinstead, is to analyze the behavior of the MT sys-tem when confronted with ungrammatical input.Reference-less evaluation has also been proposedfor text simplification (Martin et al., 2018) andGEC (Napoles et al., 2016), while the grammat-icality of MT systems’ outputs has been evaluatedwith target-side contrastive pairs (Sennrich, 2017).

In this work, the core of our evaluation of a sys-tem’s robustness lies in the following observation:a perfectly robust-to-noise MT system wouldproduce the exact same output for the clean anderroneous versions of the same input sentence.

4We use all data, concatenating train, dev, and test splits.We sample 150K sentences from Lang8.

5Refer to (Edunov et al., 2018) for further system details.6The ideal way to potentially obtain such references of

noisy text is debatable, and the extent to which humans areable to translate ungrammatical text is unknown. A well-crafted investigation could ideally elicit translations of bothoriginal and (the multiple versions of) corrected texts frommultiple translators in order to study this issue. Although wehighly encourage such a study, we could not conduct one dueto budgetary constraints.

215

Denoting a perfect MT system as a func-tion MTper f ect(·) over input sentences to the cor-rect output sentences y, then both input sen-tences x and x would yield the same output:

y = MTper f ect(x) = MTper f ect(x).

In our case, y is unknown and we only haveaccess to a very good (but still imperfect) sys-tem MTactual(·). We propose, therefore, to treat thesystem’s output of the cleaned input (y) as refer-ence. Our assumption is that y is a good approxi-mation of the correct translation y:

y ≈ MTactual(x) = y.

Under this assumption, we can now evaluate oursystem’s robustness by comparing y and y usingautomated metrics at the corpus or sentence level.Here we list the metrics that we use and brieflydiscuss their potential shortcomings.

Robustness Percentage (RB): Given a GECcorpus {X, X}, this corpus-level metric evaluatesthe percentage at which the system outputs agreeat the sentence level:

RB =

∑x,x∈{X,X} cagree(MT(x),MT(x))

|X|,

cagree(y, y) =

1 if y = y,0 otherwise.

f-BLEU: BLEU is the most standard MT evalu-ation metric, combining n-gram overlap accuracywith a brevity penalty. We calculate sentence- andcorpus-level BLEU-4 scores for every y with y asthe reference. Note that the BLEU scores that weobtain in our experiments are not comparable withany previous work (as we do not use real refer-ences) so we denote our metric as faux BLEU (f-BLEU) to avoid confusion.7

f-METEOR: Same as above, we define faux-METEOR using the METEOR MT metric(Denkowski and Lavie, 2014) which is more se-mantically nuanced than BLEU.

7In absolute numbers, we obtain higher scores than thescores of a MT system compared against actual references:the best English-German system from the WMT 2018 evalu-ation (Edunov et al., 2018) obtained a BLEU score of 46.5;our f-BLEU scores are in the [37-65] range, but we considerthem informative only when viewed relative to other f-BLEUscores.

Target-Source Noise Ratio (NR): A notabledrawback of all the previously discussed metrics isthat they do not take into account the source sen-tences x and x or their distance. However, it is ex-pected that minimal perturbations of the input (e.g.some missing punctuation) will also be minimallyreflected in the difference of the outputs, whilemore distant inputs (which means higher levelsof noise in the uncorrected source) would lead tomore divergent outputs. To account for this ob-servation, we propose Target-Source Noise Ratio(NR) which factors the distance of the two sourcesentences into a metric. The distance of two sen-tences can be measured by any metric like BLEU,METEOR, etc. We simply use BLEU:

NR(x, x, y, y) =d(y, y)d(x, x)

=100 − BLEU(y, y)100 − BLEU(x, x)

.

If the average (corpus-level) Noise Ratio score issmaller than 1 (NR(X, X,Y, Y) < 1) then we can in-fer that the MT system reduces the relative amountof noise, as there is higher relative n-gram overlapbetween the outputs than the inputs. On the otherhand, if it is larger than 1, then the MT systemmust have introduced even more noise.8

Recently, Michel et al. (2019) proposed a crite-rion for evaluating adversarial attacks, which re-quires also having access to the correct translationy. Using a similarity function s(·), they declare anadversarial attack to be successful when:

s(x, x) +s(y, y) − s(y, y)

s(y, y)> 1

In our reference-less setting, assuming y ≈ y leadsto s(y, y) = 1. Finally, representing the similarityfunction with a distance function s(·) = 1 − d(·)and simple equation manipulation, the criterionbecomes exactly our Target-Source Noise Ratio.We have, hence, arrived at a reference-less cri-terion for evaluating any kind of adversarial at-tacks.9

4 Analysis

We first review the aggregate results across alldatasets (§4.1) and with all metrics. We also

8As presented, the NR metric assumes that the length ofthe input and target sentences are comparable. In the English-German case, this is more or less correct. A more generalimplementation could include a discount term based on theaverage sentence length ratio of the two languages.

9Indeed, grammatical noise is nothing more than naturaloccurring adversarial noise.

216

datasetnumber of average

RBover non-robust sent

NRsentences #corr/sent. f-BLEU f-METEOR

WI+loc

A 9K 3.4 17.77 46.75 65.29 2.12B 10K 2.6 21.17 54.72 70.80 2.39C 5.9K 1.8 29.07 63.46 76.63 2.73N 500 1.8 28.80 64.79 77.35 3.23

nucle 21.3K 2.0 20.69 59.97 74.6 2.92fce 20.7K 2.4 20.48 50.45 67.49 2.43jfleg 1.3K 3.8 12.42 42.05 61.99 2.18

Lang8 149.5K 2.4 16.06 37.15 58.89 2.20

ALL\Lang8 69K 2.4 20.94 54.65 70.64 2.55ALL 218.5K 2.4 17.60 42.65 62.59 2.55

Table 1: Aggregate results across all datasets. As expected, the NMT system’s performance deteriorates as inputnoise increases. For all metrics except NR, higher scores are better.

present findings based on sentence-level analysis(§4.2). We investigate the specific types of errorsthat contribute to robustness as well as those thatincrease undesired behavior in §4.3. Finally, inSection §4.4 we introduce the more fine-grainednotion of divergence that allows us to performinteresting analysis and visualizations over thedatasets.

4.1 Aggregate Results

Table 1 presents the general picture of our ex-periments, summarizing the translation robustnessacross all datasets with all the metrics that we ex-amined, and also providing basic dataset statistics.Note that the aggregate f-BLEU and f-METEORscores in Table 1 are calculated excluding sen-tences where the system exhibits robustness. Wemade this choice in order to tease apart the dif-ferences across the datasets by focusing on theproblematic instances; having between 17% and29% of the scores be perfect 100 f-BLEU pointswould obscure the analysis. We also report aver-age scores across all datasets (last row) as well asscores without including Lang8, since the Lang8dataset is significantly larger than the others.

Takeaway 1: Increased amounts of noise inthe source degrade translation performance.The first takeaway confirms the previous resultsin the literature (Belinkov and Bisk, 2018; Anas-tasopoulos et al., 2019). The average number ofcorrections per sentence and the robustness per-centage (RB) column have a Pearson’s correlationcoefficient ρ = −0.82, while both f-BLEU and f-

1 2 3 4 5 6 7 8 9 100

20

40

60

Number of Errors in Source

Perc

enta

geRB f-BLEU

Figure 1: Effect of the number of errors on robustness.Robustness Percentage more than halves for each addi-tional input sentence error, while f-BLEU on the non-robust sentences reduces linearly.

METEOR have lower ρ = −0.71.This is further outlined by the results on the

WI+loc datasets. The English proficiency of thestudents increases from the A to B to C sub-sets, and the N subset is written by native Englishspeakers. An increase in English proficiency man-ifests as a lower number of errors, higher robust-ness percentage, and larger f-BLEU scores.

Takeaway 2: The MT system generally mag-nifies the input noise. This is denoted by the NRcolumn which is larger than 1 across the board.This means that the MT system exacerbated theinput noise by a factor of about 2.5. This effect ismore visible when the source noise levels are low,as in the WI+loc C and N or the nucle datasets.

4.2 Sentence-level FindingsWe continue our analysis focusing on instance orsentence-level factors, presenting results combin-ing all datasets.

217

<5 <10 <15 <20 <25 <30 <35

1

2

3

4

5#e

rror

sin

sour

ce

source sentence length

Figure 2: Robustness Percentage broken down by sen-tence length and number of source errors. The radiusof each circle is proportional to the number of sen-tences and the opacity corresponds to RB score (dark-est: RB=33%, lowest: RB≈2%). The model is more ro-bust with few errors regardless of sentence length.

Effect of the input number of errors: Figure 1clearly shows the compounding effect of sourceside errors. Each additional error reduces overallrobustness by more than 50%: from robust be-havior in about 32% of the 1-error instances, to13% for the 2-errors instances, to 6% on instanceswith 3 errors; and so forth, to the point that themodel is robust in less than 1% of instances withmore than 5 source-side errors. The robustnessdrop when computed with f-BLEU is practicallylinear, starting from about 59 f-BLEU points whena single error is present, falling to about 28 whenthe source has more than 9 errors.

Effect of input length: One factor related tothe number of input errors is the effect of thesource sentence length. We find that there is a neg-ative correlation between the input length and themodel’s robustness. This is to be expected, as in-put length and the number of errors are also cor-related: longer sentences are more likely to moreerrors, and inversely, short sentences cannot havea large number of errors.

Figure 2 presents the RB score across these twofactors. We bin the input sentences based on theirsentence lengths and based on the number of er-rors in the source. We only plot bins that have aRB score of more than 1% (reflected in the opacityof the plot). It is clear that more errors in a sourcesentence lead to reduced robustness, while the sen-tence length is not as significant a factor.

A closer look at sentences with a single error

Recoverable Non-recoverableError RB Error EB

VERB-INFL 22% CONJ 3%VERB-SVA 22% OTHER 5%

ORTH 19% NOUN 6%VERB-FORM 17% ADV 7%

WO 17% VERB 7%

Table 2: Some errors are easier to translate correctlythan others. The average error has an RB score of 11%.We present the errors that fall out of the [µ±2σ] range.

reveals that the system is robust about 30% of thetime regardless of their length, with a slight in-crease in accuracy as the length increases. Longersentences provide more context, which presum-ably aids in dealing with the source noise. Thispattern is similar across all rows in Figure 2.

4.3 Error-level AnalysisIn this section we aim to study and identify theerror types from which the NMT system is ableto recover, or not. To avoid the compounding ef-fects of multiple source-side errors, we restrict thisanalysis to sentences that have a single error.

We have already discussed in Section 4.1 howthe NMT system is robust on about 20% of theinstances across all corpora. By selecting thoseinstances and computing basic error statistics onthem, we find that the average error is recoverableabout 11% of the time (µ = 0.11). Table 2 presentsthe errors that are harder or easier to translate cor-rectly. We choose to present the errors that are atthe bottom and top, respectively, of the ranking ofthe errors, based on the average RB score that theircorresponding test instances receive.

The non-recoverable errors on the right side ofTable 2 are mostly semantic in nature: all fiveof them correspond to instances where a seman-tically wrong word was used.10 Correcting andeven identifying these types of errors is difficulteven in a monolingual setting as world knowl-edge and/or larger (document/discourse) context isneeded. One could argue, in fact, that such errorsare not grammatical, i.e. the source sentence is flu-ent. Furthermore, one could form a solid argumentfor not wanting/expecting an MT system to alterthe semantics of the source. The MT system’s jobis exactly to accurately convey the semantics of the

10We refer the reader to Bryant et al. (2017) for a completelist of the error type abbreviations.

218

I want to play with children and see their simle all day.

Ich will mit den Kindern spielen und sie den ganzen Tag sehen.

Ich mochte mit Kindern spielen und ihr Lacheln den ganzen Tag sehen.

I want to play with children and see their smiles all day.

counts:

relative pos:

x

MT(x)

x

MT(x)

+0 +1 +0 +0 +0 +0 +1 +1 +0 +0 +0 +0

−7 −6 −5 −4 −3 −2 −1 0 1 2 3 4

Figure 3: The procedure of computing divergence over a quadruple (x, y, x, y). Each token in output y not in thedesired output y is considered a divergent token (underlined=matching). The x-axis is centered around the token yk

that aligns to the edit xi*→ x j. The counts describe the caused divergence relative to the expected error’s position.

source sentence in the target language.However, there are errors where the intended

meaning is clear but ungrammatically executed, asin Table 2’s left-side errors. There are three plausi-ble (likely orthogonal, but we leave such analysisfor future work) reasons why these errors are eas-ier than average to correctly translate:

1. Self-attention. The encoder’s final repre-sentations are computed through multiple self-attention layers, resulting in a representation heav-ily informed by the whole source context. TheVERB-INFL, VERB-SVA, and VERB-FORM er-ror categories (all related to morphology and syn-tactic constraints) apply to edits that subwordmodeling combined with self-attention would al-leviate. Consider the example of the verb inflec-tion (VERB-INFL) error danceing*/dancing. Thesegmentation in the erroneous and the correctedversion is dance|ing and danc|ing respectively. Inboth cases, the morpheme that denotes the inflec-tion is the same. Verb form (VERB-FORM) errors,on the other hand, typically involve infinitive, to-infinitive, gerund, or participle forms. It seems thatin those cases the self-attention component is ableto use the context to recover, especially because, asin the VERB-INFL example, the stem of the verbwill most likely be the same.

Also, apart from the positional embeddings, noother explicit word order information is encodedby the architecture (unlike recurrent architecturesfocused on by all previous work, which by con-struction keep track of word order). We suggestthat the self-attention architecture makes word or-der errors (WO errors are strictly defined as ex-act match tokens wrongly ordered, e.g. know al-ready*/already know) easier to recover from.

2. The extensive use of back-translation. TheSOTA model that we use has been trained on mas-

sive amounts of back-translated data, where Ger-man monolingual data have been translated intoEnglish. The integral part is that English sourceswere sampled from the De-En model, instead ofusing beam-search to generate the most likelyoutput. This means that the model was alreadytrained on a fair amount of source-side noise,which would make it more robust to such pertur-bations (Belinkov and Bisk, 2018; Anastasopouloset al., 2019; Singh et al., 2019).

Although we do not have access to the back-translated parallel data that Edunov et al. (2018)used, we suspect that translation errors are fairlycommon and therefore more prevalent in the fi-nal training bitext, making the model more robustto such noise. Current English-to-German SOTAsystems might not have issues with translatingnoun phrases, coordinated verbs, or pronoun num-ber, but they still struggle with compound genera-tion, coreference, and verb-future generation (Bo-jar et al., 2018).

3. Data preprocessing and subword-level mod-eling. It is worth noting that ERRANT limitsthe orthography (ORTH) error category to re-fer to edits involving casing (lower↔upper) andwhitespace changes. Our model, as most of theSOTA NMT models, is trained and operates atthe subword level, using heuristic segmentationalgorithms like Byte Pair Encoding (BPE) (Sen-nrich et al., 2016b), that are learned on cleantruecased data. Truecasing is also a standard pre-processing step at inference time, hence dealingwith casing errors. The BPE segmentation alsohas the capacity to deal with whitespace errors.For example, the incorrect token “weatherrelated”gets segmented to we|a|ther|related. Although im-perfect (the segment’s segmentation with properwhitespacing is we|a|ther related), the two seg-

219

mentations agree for 3/4 tokens. Most previouswork e.g. (Belinkov and Bisk, 2018) has focusedon character-level modeling using compositionalfunctions across characters to represent tokens,which are by construction more vulnerable to sucherrors.

4.4 Divergence

We introduce a method for computing a di-vergence distribution. Computing divergence re-quires a quadruple of (x, x, y, y). We will focus oninstances where x and x differ only with a singleedit, as a simple working example.

Process: Given a source side sentence pair x andx with a single grammatical error, it is trivial toidentify the position i∗ of the correction in x, sincewe work on corpora pre-annotated with grammat-ical edits at the token level. Also, using traditionalmethods like the IBM Models (Brown et al., 1993)and the GIZA++ tool, makes it easy to obtain analignment between x and y, as well as between xand y. We use the alignment variable α j = i todenote that the target word y j is aligned to sourceposition i, and equivalently the variables α for thecorrected source pair. We denote as k∗ the targetposition that aligns to the source-side correction,such that αk∗ = i∗.

We define the set of divergent tokens Y∗ as theset of tokens of y that do not appear in y:

Y∗ = {y j | y j < y}.

Now, we use all the previous definitions to definethe setP of target divergent positions for a quadru-ple (x, x, y, y) as the set of target-side positions ofthe tokens that are different between y and y, butrelative to the position of the target-side token thataligns to the source-side correction:

P(x, x, y, y) = { j − k∗,∀y j ∈ Y∗}.

We provide an illustration of this process for asingle-error example in Figure 3. The correctionsimle*/smiles is aligned to the word y7 (Lacheln)in the reference target, so the center of the dis-tribution is moved to k∗ = 7. For the rest of thepositions in the target reference y, we simply up-date the counts based on whether the word y j ispresent in y. The final step is collecting countsacross all instances for all the relative divergentpositions and analyzing the effect of a source-sideerror on the target sentence.

Essentially, we expect some source-side errorsto have a very local effect on the translation output,which would translate in divergence distributionswith low variance (since we center the distributionaround k∗). Other source-side errors might causelarger divergence as they might affect the wholestructure of the target sentence.

In the Figure 3 example, the only difference be-tween x and x is a single word towards the end ofthe sentence, but the outputs y and y diverge onthree words. One of them is 6 words away (be-fore) from where we would have expected the di-vergence to happen (in relative position 0).

After collecting divergence counts for each in-stance, we can visualize their distribution andcompute their descriptive statistics. We focus onthe mean µ, standard deviation σ, and the skew-ness of the distribution as measured by Pearson’sdefinition, using the third standardised moment,defined as:

γ1 = E

[(X − µσ

)3].

Across all datasets and errors, the distribu-tion of the divergence caused by single errors inthe source has a mean µall=0.7, standard devia-tion σall=5.1, and a slight positive skewness withγ1all=0.8. This means that the average error affectsits general right context, in a ±5 word neighbor-hood.

In Figure 4 we present several of the errors withthe most interesting divergence statistics. Someerrors heavily affect their left (e.g. R:ORTH) orright context (U:CONJ). Also, some errors affect asmall translation neighborhood as denoted by thelow variance of their divergence distribution (e.g.U:CONTR). On the other hand, verb form errors(M:VERB:FORM) have the potential to affect alarger neighborhood: this is expected because En-glish auxiliary verb constructions (e.g. ”have eatenX”) often get translated to German V2 construc-tions with an auxiliary verb separated from a final,non-finite main verb (e.g. “habe X gegessen”).

In Figure 5 we present the divergence distribu-tions across the sentence quartiles where the errorappears. We find that errors in the sentence begin-ning (1st quartile) severely affect their right con-text. Errors towards the end of the sentence (4th

quartile) affect their left context. Interestingly, weobserve that mid-sentence errors (2nd, 3rd quar-tiles) exhibit much lower divergence variance thanerrors towards the sentence’s edges.

220

R:ORTH M:VERB:FORM U:ADJµ: -1.9 σ: 5.9 γ1: -2.1 µ: -1.4 σ: 7.6 γ1: -1.2 µ: -1.1 σ: 6.1 γ1: -1.1

lowest mean, large variance, low mean, largest variance, low mean, large variance,most negative skewness negative skewness negative skewness

U:CONTR R:WO U:CONJµ: 1.6 σ: 3.8 γ1: 0.1 µ: 1.8 σ: 4.9 γ1: 1.0 µ: 2.3 σ: 5.6 γ1: -0.2

large mean, smallest variance, large mean, small variance, largest mean, large variance,no skewness positive skewness no skewness

U:PART R:SPELL R:CONTRµ: 0.6 σ: 5.7 γ1: -1.9 µ: 0.5 σ: 5.6 γ1: 5.9 µ: 1.1 σ: 5.6 γ1: 2.1

close-to-zero mean, large variance, almost-zero mean, large variance, large variance,large negative skewness largest positive skewness large positive skewness

Figure 4: Some interesting errors with statistics on their divergence distribution. Some errors (negative mean andskewness: R:ORTH, M:VERB:FORM, U:ADJ) affect the left context of their translation more, while others affect their righttranslation context (positive mean and skewness R:WO, U:CONJ). Errors might affect a small neighborhood (low variance:U:CONTR, R:WO) or a larger part of the translation (high variance: M:VERB:FORM, U:ADJ, R:CONTR).

first quartile(i∗ < 0.25|x|)

second quartile(0.25|x| ≤ i∗ < 0.50|x|)

third quartile(0.50|x| ≤ i∗ < 0.70|x|)

fourth quartile(0.75|x| ≤ i∗ < |x|)

µ: 3.8 σ: 5.6 γ1: 3.3 µ: 1.6 σ: 4.1 γ1: 1.2 µ: -0.2 σ: 3.6 γ1: -1.1 µ: -2.9 σ: 4.8 γ1: -2.5

Figure 5: Divergence Distributions for single source error instances per the error’s location quartile.

221

5 Limitations and Extensions

A major limitation of our analysis is the narrowscope of our experiments. We solely focused ona single language pair using a single MT system.Whether different neural architectures over otherlanguages would lead to different conclusions re-mains an open question. The necessary resourcesfor answering these questions at scale, however,are not yet available. We were limited to Englishas our source side language, as the majority of thedatasets and research works in GEC are entirelyEnglish-centric. Perhaps small-scale GEC datasetsin Estonian (Rummo and Praakli, 2017) and Lat-vian (Deksne and Skadina, 2014) could providea non-English testbed. One would then need er-ror labels for the grammatical edits, so if such an-notations are not available, an extension of a toollike ERRANT to these languages would also berequired. One should also be careful in the de-cision of what (N)MT system to test, as using alow-quality translation system would not producemeaningful insights.

Another limitation is that our metrics do notcapture whether the changes in the output are ac-tually grammatical errors or not. In the example inFigure 3: the German words “mochte” and “will”that we identified as divergent are practically in-terchangeable. Therefore, the NMT model is tech-nically not wrong outputting either of them andit is indeed generally possible that differences be-tween y and y are just surface-level ones. Theinclusion of f-METEOR as a robustness metriccould partially deal with this issue, as it would notpenalize such differences. We do believe it is stillinteresting, though, that a single source error cancause large perturbations in the output, as in thecase of errors with large variance in their diver-gence distribution. Nevertheless, an extension ofour study focusing on the grammatical qualities ofthe MT output would be exciting and automatedtools for such analysis would be invaluable (i.e.MT error labeling and analysis tools extending theworks of Zeman et al. (2011), Logacheva et al.(2016), Popovic (2018), or Neubig et al. (2019)).

A natural next research direction is investigat-ing how to use our reference-less evaluation met-rics in order to create a more robust MT system.For instance, one could optimize for f-BLEU orany of the other reference-less measures that weproposed, in the same way that an MT system isoptimized for BLEU (either by explicitly using

their scores through reinforcement learning or bysimply using the metric as an early stopping crite-rion over a development set). Cheng et al. (2018)recently proposed an approach for training morerobust MT systems, albeit in a supervised settingwhere noise is injected on parallel data, and theproposed solutions of Belinkov and Bisk (2018)and Anastasopoulos et al. (2019) fall within thesame category. However, no approach has, to ourknowledge, used GEC corpora for training MTsystems robust to grammatical errors. In any case,special care should be taken so that any improve-ments on translating ungrammatical data do notworsen performance on clean ones.

6 Conclusion

In this work, we studied the effects of grammaticalerrors in NMT. We expanded on findings from pre-vious work, performing analysis on real data withgrammatical errors using a SOTA system. Withour analysis we were able to identify classes ofgrammatical errors that are recoverable or irrecov-erable. Additionally, we presented ways to evalu-ate a MT system’s robustness to noise without ac-cess to gold references, as well as a method forvisualizing the effect of source-side errors to theoutput translation. Finally, we discussed the limi-tations of our study and outlined avenues for fur-ther investigations towards building more robustNMT systems.

Acknowledgements

The author is grateful to the anonymous review-ers, Kenton Murray, and Graham Neubig for theirconstructive and insightful comments, as well as toGabriela Weigel for her invaluable help with edit-ing and proofreading the final version of this pa-per. This material is based upon work generouslysupported by the National Science Foundation un-der grant 1761548.

ReferencesAntonios Anastasopoulos, Alison Lui, Toan Q.

Nguyen, and David Chiang. 2019. Neural machinetranslation of text from non-native speakers. Proc.NAACL-HLT.

Øistein E Andersen, Helen Yannakoudakis, FionaBarker, and Tim Parish. 2013. Developing and test-ing a self-assessment and tutoring system. In Proc.BEA-NLP.

222

Satanjeev Banerjee and Alon Lavie. 2005. Meteor: Anautomatic metric for mt evaluation with improvedcorrelation with human judgments. In Proceedingsof the acl workshop on intrinsic and extrinsic evalu-ation measures for machine translation and/or sum-marization.

Yonatan Belinkov and Yonatan Bisk. 2018. Syntheticand natural noise both break neural machine transla-tion. In Proc. ICLR.

Ondrej Bojar, Rajen Chatterjee, Christian Feder-mann, Mark Fishel, Yvette Graham, Barry Had-dow, Matthias Huck, Antonio Jimeno Yepes, PhilippKoehn, Christof Monz, et al. 2018. Proceedingsof the third conference on machine translation. InProc. WMT.

Peter F. Brown, Vincent J. Della Pietra, StephenA. Della Pietra, and Robert L. Mercer. 1993. Themathematics of statistical machine translation: Pa-rameter estimation. Computational Linguistics,19(2):263–311.

Christopher Bryant, Mariano Felice, and Ted Briscoe.2017. Automatic annotation and evaluation of er-ror types for grammatical error correction. In Proc.ACL.

Yong Cheng, Zhaopeng Tu, Fandong Meng, JunjieZhai, and Yang Liu. 2018. Towards robust neuralmachine translation. In Proc. ACL.

Josep Crego, Jungi Kim, Guillaume Klein, Anabel Re-bollo, Kathy Yang, Jean Senellart, Egor Akhanov,Patrice Brunelle, Aurelien Coquard, YongchaoDeng, et al. 2016. Systran’s pure neural machinetranslation systems. arXiv:1610.05540.

Daniel Dahlmeier, Hwee Tou Ng, and Siew Mei Wu.2013. Building a large annotated corpus of learnerenglish: The nus corpus of learner english. In Proc.BEA-NLP.

Daiga Deksne and Inguna Skadina. 2014. Error-annotated corpus of latvian. In Proc. Baltic HLT.

Michael Denkowski and Alon Lavie. 2014. Meteoruniversal: Language specific translation evaluationfor any target language. In Proc. WMT.

Council of Europe. Council for Cultural Co-operation.Education Committee. Modern Languages Division.2001. Common European Framework of Refer-ence for Languages: learning, teaching, assessment.Cambridge University Press.

Sergey Edunov, Myle Ott, Michael Auli, and DavidGrangier. 2018. Understanding back-translation atscale. In Proc. EMNLP.

Hany Hassan Awadalla, Anthony Aue, Chang Chen,Vishal Chowdhary, Jonathan Clark, Christian Feder-mann, Xuedong Huang, Marcin Junczys-Dowmunt,Will Lewis, Mu Li, Shujie Liu, Tie-Yan Liu, Ren-qian Luo, Arul Menezes, Tao Qin, Frank Seide,

Xu Tan, Fei Tian, Lijun Wu, Shuangzhi Wu, YingceXia, Dongdong Zhang, Zhirui Zhang, and MingZhou. 2018. Achieving human parity on automaticchinese to english news translation.

Georg Heigold, Stalin Varanasi, Gunter Neumann, andJosef Genabith. 2018. How robust are character-based word embeddings in tagging and mt againstwrod scramlbing or randdm nouse? In Proc. AMTA.

Philipp Koehn and Rebecca Knowles. 2017. Six chal-lenges for neural machine translation. In Proc.WNMT.

Varvara Logacheva, Chris Hokamp, and Lucia Specia.2016. Marmot: A toolkit for translation quality esti-mation at the word level. In Proc. LREC.

Louis Martin, Samuel Humeau, Pierre-EmmanuelMazare, Eric De La Clergerie, Antoine Bordes, andBenoıt Sagot. 2018. Reference-less quality estima-tion of text simplification systems. In Proc. ATA.

Paul Michel, Xian Li, Graham Neubig, andJuan Miguel Pino. 2019. On evaluation of ad-versarial perturbations for sequence-to-sequencemodels. In Proc. NAACL-HLT.

Courtney Napoles, Keisuke Sakaguchi, and JoelTetreault. 2016. There’s no comparison: Reference-less evaluation metrics in grammatical error correc-tion. In Proc. EMNLP.

Courtney Napoles, Keisuke Sakaguchi, and JoelTetreault. 2017. Jfleg: A fluency corpus and bench-mark for grammatical error correction. In Proc.EACL, Valencia, Spain.

Graham Neubig, Zi-Yi Dou, Junjie Hu, Paul Michel,Danish Pruthi, and Xinyi Wang. 2019. compare-mt:A tool for holistic comparison of language genera-tion systems. In Proc. NAACL-HLT.

Hwee Tou Ng, Siew Mei Wu, Ted Briscoe, ChristianHadiwinoto, Raymond Hendy Susanto, and Christo-pher Bryant. 2014. The conll-2014 shared task ongrammatical error correction. In Proc. CoNLL.

Hwee Tou Ng, Siew Mei Wu, Yuanbin Wu, ChristianHadiwinoto, and Joel Tetreault. 2013. The conll-2013 shared task on grammatical error correction.In Proc. CoNLL.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic eval-uation of machine translation. In Proc. ACL.

Maja Popovic. 2018. Error classification and analy-sis for machine translation quality assessment. InTranslation Quality Assessment, pages 129–158.Springer.

Ingrid Rummo and Kristiina Praakli. 2017. TU eestikeele (voorkeelena) osakonna oppijakeele tekstikor-pus [the language learners corpus of the departmentof estonian language of the university of tartu]. InProc EAAL.

223

Rico Sennrich. 2017. How grammatical is character-level neural machine translation? assessing mt qual-ity with contrastive translation pairs. In Proc. EACL.

Rico Sennrich, Barry Haddow, and Alexandra Birch.2016a. Improving neural machine translation mod-els with monolingual data. In Proc. ACL.

Rico Sennrich, Barry Haddow, and Alexandra Birch.2016b. Neural machine translation of rare wordswith subword units. In Proc. ACL.

Sumeet Singh, Craig Stewart, Graham Neubig, et al.2019. Improving robustness of machine translationwith synthetic noise. arXiv:1902.09508.

Lucia Specia, Carolina Scarton, and Gustavo HenriquePaetzold. 2018. Quality estimation for machinetranslation. Synthesis Lectures on Human LanguageTechnologies, 11(1):1–162.

Toshikazu Tajiri, Mamoru Komachi, and Yuji Mat-sumoto. 2012. Tense and aspect error correction foresl learners using global context. In Proc. ACL.

Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V.Le, Mohammad Norouzi, Wolfgang Macherey,Maxim Krikun, Yuan Cao, Qin Gao, KlausMacherey, et al. 2016. Google’s neural machinetranslation system: Bridging the gap between humanand machine translation. arXiv:1609.08144.

Helen Yannakoudakis, Ted Briscoe, and Ben Medlock.2011. A new dataset and method for automaticallygrading esol texts. In Proc. ACL-HLT.

Daniel Zeman, Mark Fishel, Jan Berka, and Ondrej Bo-jar. 2011. Addicter: what is wrong with my trans-lations? The Prague Bulletin of Mathematical Lin-guistics, 96:79–88.