Paper Info
Title
Human Parity on CommonsenseQA: Augmenting Self-Attention with External Attention.
Abstract
Most of today's AI systems focus on using self-attention mechanisms and transformer architectures on large amounts of diverse data to achieve impressive performance gains. In this paper, we propose to augment the transformer architecture with an external attention mechanism to bring external knowledge and context to bear. By integrating external information into the prediction process, we hope to reduce the need for ever-larger models and increase the democratization of AI systems. We find that the proposed external attention mechanism can significantly improve the performance of existing AI systems, allowing practitioners to easily customize foundation AI models to many diverse downstream applications. In particular, we focus on the task of Commonsense Reasoning, demonstrating that the proposed external attention mechanism can augment existing transformer models and significantly improve the model's reasoning capabilities. The proposed system, Knowledge External Attention for Reasoning (KEAR), reaches human parity on the open CommonsenseQA research benchmark with an accuracy of 89.4\% in comparison to the human accuracy of 88.9\%.
Year
DOI
Venue
2022
10.24963/ijcai.2022/383
European Conference on Artificial Intelligence
Keywords
DocType
Citations 
Knowledge Representation and Reasoning: Common-Sense Reasoning,Natural Language Processing: Question Answering,Knowledge Representation and Reasoning: Reasoning about Knowledge and Belief
Conference
0
PageRank 
References 
Authors
0.34
0
10
Name
Order
Citations
PageRank
Yichong Xu1177.39
Chenguang Zhu232822.92
shuohang wang318016.63
Siqi Sun4679.15
Hao Cheng500.68
Xiaodong Liu613517.46
Jianfeng Gao75729296.43
Pengcheng He898.97
Michael Zeng946.85
Xuedong Huang101390283.19