Paper Info
Title
Topic extraction from millions of tweets using singular value decomposition and feature selection.
Abstract
Social media offers a wealth of insight into how significant topics-such as the Great East Japan Earthquake, the Arab Spring, and the Boston Bombing-affect individuals. The scale of available data, however, can be intimidating: during the Great East Japan Earthquake, over 8 million tweets were sent each day from Japan alone. Conventional word vector-based social media analysis method using Latent Semantic Analysis, Latent Dirichlet Allocation, or graph community detection often cannot scale to such a large volume of data due to their space and time complexity. To overcome the scalability problem, in this paper, high performance Singular Vector Decomposition (SVD) library redsvd has been used to identify topics over time from the huge data set of over two hundred million tweets sent in the 21 days following the Great East Japan Earthquake. While we begin with word count vectors of authors and words for each time slot (in our case, every hour), authors' clusters from each slot are extracted by SVD and k-means. And then, the original fast feature selection algorithm named CWC has been used to extract discriminative words from each cluster. As a result, authors' clusters recognized as topics as well as issues of conventional social media analysis method for big data can be visualized overcoming the scalability problem.
Year
Venue
Field
2015
Asia-Pacific Signal and Information Processing Association Annual Summit and Conference
Data mining,Latent Dirichlet allocation,Feature selection,Computer science,Vector decomposition,Feature extraction,Word count,Latent semantic analysis,Big data,Scalability
DocType
ISSN
Citations 
Conference
2309-9402
0
PageRank 
References 
Authors
0.34
6
3
Name
Order
Citations
PageRank
Takako Hashimoto15018.47
Tetsuji Kuboyama214029.36
Basabi Chakraborty310923.21