Paper Info
Title
Tibetan Unknown Word Identification From News Corpora For Supporting Lexicon-Based Tibetan Word Segmentation
Abstract
In Tibetan, as words are written consecutively without delimiters, finding unknown word boundary is difficult. This paper presents a hybrid approach for Tibetan unknown word identification for offline corpus processing. Firstly, Tibetan named entity is preprocessed based on natural annotation. Secondly, other Tibetan unknown words are extracted from word segmentation fragments using MTC, the combination of a statistical metric and a set of context sensitive rules. In addition, the preliminary experimental results on Tibetan News Corpus are reported. Lexicon-based Tibetan word segmentation system SegT with proposed unknown word extension mechanism is indeed helpful to promote the performance of Tibetan word segmentation. It increases the F-score of Tibetan word segmentation by 4.15% on random-selected test set. Our unknown word identification scheme can find new words in any length and in any field.
Year
Venue
Field
2015
PROCEEDINGS OF THE 53RD ANNUAL MEETING OF THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS (ACL) AND THE 7TH INTERNATIONAL JOINT CONFERENCE ON NATURAL LANGUAGE PROCESSING (IJCNLP), VOL 2
Identification scheme,Annotation,Computer science,Speech recognition,Text segmentation,Named entity,Lexicon,Artificial intelligence,Natural language processing,Delimiter,Test set
DocType
Volume
Citations 
Conference
P15-2
0
PageRank 
References 
Authors
0.34
13
4
Name
Order
Citations
PageRank
Minghua Nuo1114.22
Huidan Liu2165.09
Long Congjun300.68
Jian Wu421.58