Name
Title | ||
|---|---|---|
Chinese Word Segmentation with Minimal Linguistic Knowledge: An Improved Conditional Random Fields Coupled with Character Clustering and Automatically Discovered Template Matching |
Abstract | ||
|---|---|---|
This paper addresses three major problems of closed task Chinese word segmentation (CWS): word overlap, tagging sentences interspersed with non-Chinese words, and long named entity (NE) identification. For the first, we use additional bigram features to approximate trigram and tetragram features. For the second, we first apply K-means clustering to identify non-Chinese characters. Then, we employ a two-tagger architecture: one for Chinese text and the other for non-Chinese text. Finally, we post-process our CWS output using automatically generated templates. Our results show that additional bigrams can effectively identify more unknown words. Secondly, using our two-tagger method, segmentation performance on sentences containing non-Chinese words is significantly improved when non-Chinese characters are sparse in the training corpus. Lastly, identification of long NEs and long words is also enhanced by template-based post-processing. Using corpora in closed task of SIGHAN CWS, our best system achieves F-scores of 0.956, 0.947, and 0.965 on the AS, HK, and MSR corpora respectively, compared to the best context scores of 0.952, 0.943, and 0.964 in SIGHAN Bakeoff 2005. In AS, this performance is comparable to the best result (F = 0.956) in the open task |
Year | DOI | Venue |
|---|---|---|
2006 | 10.1109/IRI.2006.252425 | Waikoloa Village, HI |
Keywords | Field | DocType |
computational linguistics,natural language processing,pattern clustering,random processes,text analysis,Chinese text,Chinese word segmentation,K-means clustering,bigram feature,character clustering,conditional random fields,linguistic knowledge,named entity identification,tagging sentences,template matching,tetragram feature,trigram feature,two-tagger method,word overlap | Template matching,Conditional random field,k-means clustering,Trigram,Computer science,Segmentation,Speech recognition,Text segmentation,Artificial intelligence,Bigram,Natural language processing,Cluster analysis | Conference |
ISBN | Citations | PageRank |
0-7803-9788-6 | 0 | 0.34 |
References | Authors | |
3 | 6 | |
Name | Order | Citations | PageRank |
|---|---|---|---|
| Richard Tzong-Han Tsai | 1 | 714 | 54.89 |
| Hong-Jie Dai | 2 | 288 | 21.58 |
| Hsieh-Chuan Hung | 3 | 103 | 4.93 |
| Cheng-Lung Sung | 4 | 239 | 13.67 |
| Min-Yuh Day | 5 | 198 | 29.24 |
| Wen-Lian Hsu | 6 | 1701 | 198.40 |