Name
Title | ||
|---|---|---|
Beyond Error Propagation: Language Branching Also Affects the Accuracy of Sequence Generation |
Abstract | ||
|---|---|---|
Sequence generation tasks, such as neural machine translation (NMT) and abstractive summarization, usually suffer from exposure bias as well as the error propagation problem due to the autoregressive training and generation. Many previous works have discussed the relationship between error propagation and the
<italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">accuracy drop</italic>
problem (i.e., the right part of the generated sentence is often worse than its left part in left-to-right decoding models). In this paper, taking NMT as a typical sequence generation task, we measure the accuracy of the generated sentence with various metrics and conduct a series of analyses to deeply understand the accuracy drop problem. We obtain several interesting findings. First, The role of error propagation on accuracy drop is overstated in the literature, although it is indeed a cause to the accuracy drop problem. Second, Characteristics of a language play a more important role in causing the accuracy drop problem: the left part of the generated sentence in a right-branching language (e.g., English) is more likely to be more accurate than its right part, while the right part is more accurate for a left-branching language (e.g., Japanese). Our discoveries are also confirmed on other generation tasks (e.g., image captioning, abstractive summarization and language modeling) with multiple left/right-branching languages, as well as in various model structures. |
Year | DOI | Venue |
|---|---|---|
2019 | 10.1109/TASLP.2019.2933727 | IEEE/ACM Transactions on Audio, Speech, and Language Processing |
Keywords | Field | DocType |
Task analysis,Decoding,Training,Measurement,Linguistics,Speech processing,Maximum likelihood estimation | Autoregressive model,Speech processing,Automatic summarization,Propagation of uncertainty,Task analysis,Computer science,Machine translation,Speech recognition,Sentence,Language model | Journal |
Volume | Issue | ISSN |
27 | 12 | 2329-9290 |
Citations | PageRank | References |
0 | 0.34 | 8 |
Authors | ||
5 | ||
Name | Order | Citations | PageRank |
|---|---|---|---|
| Lijun Wu | 1 | 124 | 21.21 |
| Xu Tan | 2 | 88 | 23.94 |
| Tao Qin | 3 | 2384 | 147.25 |
| Jian-Huang Lai | 4 | 1783 | 138.20 |
| Tie-yan Liu | 5 | 4662 | 256.32 |