Paper Info
Title
Extracting Weighted Language Lexicons from Wikipedia.
Abstract
Language models are used in applications as diverse as speech recognition, optical character recognition and information retrieval. They are used to predict word appearance, and to weight the importance of words in these applications. One basic element of language models is the list of words in a language. Another is the unigram frequency of each word. But this basic information is not available for most languages in the world. Since the multilingual Wikipedia project encourages the production of encyclopedic-like articles in many world languages, we can find there an ever-growing source of text from which to extract these two language modelling elements: word list and frequency. Here we present a simple technique for converting this Wikipedia text into lexicons of weighted unigrams for the more than 270 languages present currently present in Wikipedia. The lexicons produced, and the source code for producing them in a Linux-based system are here made available for free on the Web.
Year
Venue
Keywords
2016
LREC 2016 - TENTH INTERNATIONAL CONFERENCE ON LANGUAGE RESOURCES AND EVALUATION
lexicon,Wikipedia,language model,under-resourced languages
Field
DocType
Citations 
Information retrieval,Computer science,Natural language processing,Artificial intelligence
Conference
2
PageRank 
References 
Authors
0.41
3
1
Name
Order
Citations
PageRank
Gregory Grefenstette11129147.00