Abstract | ||
---|---|---|
Cross-lingual word embeddings (CLWE) have been proven useful in many cross-lingual tasks. However, most existing approaches to learn CLWE including the ones with contextual embeddings are sense agnostic. In this work, we propose a novel framework to align contextual embeddings at the sense level by leveraging cross-lingual signal from bilingual dictionaries only. We operationalize our framework by first proposing a novel sense-aware cross entropy loss to model word senses explicitly. The monolingual ELMo and BERT models pretrained with our sense-aware cross entropy loss demonstrate significant performance improvement for word sense disambiguation tasks. We then propose a sense alignment objective on top of the sense-aware cross entropy loss for cross-lingual model pretraining, and pretrain cross-lingual models for several language pairs (English to German/Spanish/Japanese/Chinese). Compared with the best baseline results, our cross-lingual models achieve 0.52%, 2.09% and 1.29% average performance improvements on zero-shot cross-lingual NER, sentiment classification and XNLI tasks, respectively. |
Year | Venue | DocType |
---|---|---|
2022 | International Conference on Computational Linguistics | Conference |
Volume | Citations | PageRank |
Proceedings of the 29th International Conference on Computational Linguistics | 0 | 0.34 |
References | Authors | |
0 | 5 |
Name | Order | Citations | PageRank |
---|---|---|---|
Linlin Liu | 1 | 0 | 1.01 |
Thien Hai Nguyen | 2 | 111 | 4.50 |
Shafiq R. Joty | 3 | 560 | 56.72 |
Lidong Bing | 4 | 298 | 39.44 |
Luo Si | 5 | 2498 | 169.52 |