Paper Info
Title
Design, Compilation, and Preliminary Analyses of Balanced Corpus of Contemporary Written Japanese
Abstract
Compilation of a 100 million words balanced corpus called the Balanced Corpus of Contemporary Written Japanese (or BCCWJ) is underway at the National Institute for Japanese Language and Linguistics. The corpus covers a wide range of text genres including books, magazines, newspapers, governmental white papers, textbooks, minutes of the National Diet, internet text (bulletin board and blogs) and so forth, and when possible, samples are drawn from the rigidly defined statistical populations by means of random sampling. All texts are dually POS-analyzed based upon two different, but mutually related, definitions of 'word.' Currently, more than 90 million words have been sampled and XML annotated with respect to text-structure and lexical and character information. A preliminary linear discriminant analysis of text genres using the data of POS frequencies and sentence length revealed it was possible to classify the text genres with a correct identification rate of 88% as far as the samples of books, newspapers, whitepapers, and internet bulletin boards are concerned. When the samples of blogs were included in this data set, however, the identification rate went down to 68%, suggesting the considerable variance of the blog texts in terms of the textual register and style.
Year
Venue
Field
2010
LREC 2010 - SEVENTH INTERNATIONAL CONFERENCE ON LANGUAGE RESOURCES AND EVALUATION
XML,Computer science,Japanese language,Newspaper,Natural language processing,Artificial intelligence,Sampling (statistics),Linear discriminant analysis,Sentence,Bulletin board,The Internet
DocType
Citations 
PageRank 
Conference
10
0.89
References 
Authors
3
9
Name
Order
Citations
PageRank
Kikuo Maekawa114724.51
Makoto Yamazaki2697.42
Takehiko Maruyama38710.04
Masaya Yamaguchi4655.45
Hideki Ogura5846.97
Wakako Kashino6758.62
Toshinobu Ogiso7797.42
Hanae Koiso88612.10
Yasuharu Den914526.23