Name
Title | ||
|---|---|---|
Efficient Estimate of Sentence's Representation Based on the Difference Semantics Model |
Abstract | ||
|---|---|---|
Sentence representation is an important research hotspot in natural language processing (NLP) since it can map the semantics of sentences into semantics vectors, thereby effectively solving complex semantics computing problems. Recently, sentence representations are mainly obtained by indirect means. Specifically, for sentence representations obtained by unsupervised means, they are often calculated by the weighted sum of embeddings of tokens in sentences; for sentence representations obtained by self-supervised or supervised means, they are often derived from intermediate encodings of sentences in prediction tasks. For example, Google's BERT and MUSE respectively use the embedding of [CLS] in the next sentence prediction task and intermediate encodings of sentences in the translation bridge task as sentence representations. In this paper, we use the observed semantics increment feature of sentences to directly model the semantics function of sentences. To be able to use the existing neural network language model to approximate the semantics function, we first implement the first-order Taylor expansion on the semantics function to obtain a difference semantics model and then add it to BERT as a subtask to perform self-supervised fine-tuning. Finally, we get a new sentence representation model S-BERT. S-BERT achieves the state-of-the-art performance on many datasets in Chinese, English, and Vietnamese. |
Year | DOI | Venue |
|---|---|---|
2021 | 10.1109/TASLP.2021.3123885 | IEEE-ACM TRANSACTIONS ON AUDIO SPEECH AND LANGUAGE PROCESSING |
Keywords | DocType | Volume |
Semantics, Task analysis, Bit error rate, Analytical models, Context modeling, Encoding, Natural language processing, Sentence representation, difference semantics model, BERT | Journal | 29 |
Issue | ISSN | Citations |
1 | 2329-9290 | 0 |
PageRank | References | Authors |
0.34 | 10 | 6 |
Name | Order | Citations | PageRank |
|---|---|---|---|
| Xianwen Liao | 1 | 0 | 0.34 |
| Yongzhong Huang | 2 | 14 | 2.59 |
| Yongzhuang Wei | 3 | 69 | 16.94 |
| Chenhao Zhang | 4 | 0 | 0.34 |
| Fu Wang | 5 | 0 | 0.34 |
| Yong Wang | 6 | 0 | 0.34 |