Paper Info
Title
Do Neural Nets Learn Statistical Laws Behind Natural Language?
Abstract
The performance of deep learning in natural language processing has been spectacular, but the reasons for this success remain unclear because of the inherent complexity of deep learning. This paper provides empirical evidence of its effectiveness and of a limitation of neural networks for language engineering. Precisely, we demonstrate that a neural language model based on long short-term memory (LSTM) effectively reproduces Zipf's law and Heaps' law, two representative statistical properties underlying natural language. We discuss the quality of reproducibility and the emergence of Zipf's law and Heaps' law as training progresses. We also point out that the neural language model has a limitation in reproducing long-range correlation, another statistical property of natural language. This understanding could provide a direction for improving the architectures of neural networks.
Year
DOI
Venue
2017
10.1371/journal.pone.0189326
PLOS ONE
Field
DocType
Volume
Zipf's law,Computer science,Computational linguistics,Language acquisition,Natural language,Artificial intelligence,Deep learning,Artificial neural network,Syntax,Law,Language model
Journal
12
Issue
ISSN
Citations 
12
1932-6203
2
PageRank 
References 
Authors
0.37
19
2
Name
Order
Citations
PageRank
Shuntaro Takahashi121.38
Kumiko Tanaka-Ishii226136.69