Name
Abstract | ||
|---|---|---|
Evaluating translation models is a trade-off between effort and detail. On the one end of the spectrum there are automatic count-based methods such as BLEU, on the other end linguistic evaluations by humans, which arguably are more informative but also require a disproportionately high effort. To narrow the spectrum, we propose a general approach on how to automatically expose systematic differences between human and machine translations to human experts. Inspired by adversarial settings, we train a neural text classifier to distinguish human from machine translations. A classifier that performs and generalizes well after training should recognize systematic differences between the two classes, which we uncover with neural explainability methods. Our proof-of-concept implementation, DiaMaT, is open source. Applied to a dataset translated by a state-of-the-art neural Transformer model, DiaMaT achieves a classification accuracy of 75% and exposes meaningful differences between humans and the Transformer, amidst the current discussion about human parity. |
Year | DOI | Venue |
|---|---|---|
2019 | 10.18653/v1/n19-4006 | North American Chapter of the Association for Computational Linguistics |
Field | DocType | Volume |
Computer science,sort,Natural language processing,Artificial intelligence,Classifier (linguistics),Machine learning,Adversarial system | Journal | abs/1903.12017 |
Citations | PageRank | References |
0 | 0.34 | 0 |
Authors | ||
5 | ||
Name | Order | Citations | PageRank |
|---|---|---|---|
| Robert Schwarzenberg | 1 | 0 | 0.34 |
| David Harbecke | 2 | 0 | 0.34 |
| Vivien Macketanz | 3 | 1 | 0.71 |
| Eleftherios Avramidis | 4 | 84 | 18.17 |
| Sebastian Möller | 5 | 877 | 141.17 |