Paper Info
Title
Improving Word Embeddings Using Kernel Pca
Abstract
Word-based embedding approaches such as Word2Vec capture the meaning of words and relations between them, particularly well when trained with large text collections; however, they fail to do so with small datasets. Extensions such as fastText reduce the amount of data needed slightly, however, the joint task of learning meaningful morphology, syntactic and semantic representations still requires a lot of data. In this paper, we introduce a new approach to warm-start embedding models with morphological information, in order to reduce training time and enhance their performance. We use word embeddings generated using both word2vec and fastText models and enrich them with morphological information of words, derived from kernel principal component analysis (KPCA) of word similarity matrices. This can be seen as explicitly feeding the network morphological similarities and letting it learn semantic and syntactic similarities. Evaluating our models on word similarity and analogy tasks in English and German, we find that they not only achieve higher accuracies than the original skip-gram and fastText models but also require significantly less training data and time. Another benefit of our approach is that it is capable of generating a high-quality representation of infrequent words as, for example, found in very recent news articles with rapidly changing vocabularies. Lastly, we evaluate the different models on a downstream sentence classification task in which a CNN model is initialized with our embeddings and find promising results.
Year
DOI
Venue
2019
10.18653/v1/w19-4323
4TH WORKSHOP ON REPRESENTATION LEARNING FOR NLP (REPL4NLP-2019)
Field
DocType
Citations 
Computer science,Kernel principal component analysis,Artificial intelligence,Machine learning
Conference
0
PageRank 
References 
Authors
0.34
0
4
Name
Order
Citations
PageRank
Vishwani Gupta100.34
Sven Giesselbach200.68
Stefan Rüping325427.91
Christian Bauckhage41979195.86