Abstract | ||
---|---|---|
Automatic evaluation of Machine Translation (MT) quality is essential to developing high-quality MT systems. Various evaluation metrics have been proposed, and BLEU is now used as the de facto standard metric. However, when we consider translation between distant language pairs such as Japanese and English, most popular metrics (e.g., BLEU, NIST, PER, and TER) do not work well. It is well known that Japanese and English have completely different word orders, and special care must be paid to word order in translation. Otherwise, translations with wrong word order often lead to misunderstanding and incomprehensibility. For instance, SMT-based Japanese-to-English translators tend to translate 'A because B' as 'B because A.' Thus, word order is the most important problem for distant language translation. However, conventional evaluation metrics do not significantly penalize such word order mistakes. Therefore, locally optimizing these metrics leads to inadequate translations. In this paper, we propose an automatic evaluation metric based on rank correlation coefficients modified with precision. Our meta-evaluation of the NTCIR-7 PATMT JE task data shows that this metric outperforms conventional metrics. |
Year | Venue | Keywords |
---|---|---|
2010 | EMNLP | conventional evaluation metrics,translation quality,wrong word order,word order,distant language pair,different word order,various evaluation metrics,conventional metrics,popular metrics,automatic evaluation,word order mistake,distant language translation |
Field | DocType | Volume |
Rank correlation,De facto standard,BLEU,Language translation,Word order,Computer science,Evaluation of machine translation,Speech recognition,NIST,Artificial intelligence,Natural language processing,Machine learning | Conference | D10-1 |
Citations | PageRank | References |
77 | 2.51 | 13 |
Authors | ||
5 |
Name | Order | Citations | PageRank |
---|---|---|---|
Hideki Isozaki | 1 | 934 | 64.50 |
Tsutomu Hirao | 2 | 394 | 31.80 |
Kevin Duh | 3 | 819 | 72.94 |
Katsuhito Sudoh | 4 | 326 | 34.44 |
Hajime Tsukada | 5 | 449 | 29.46 |