Paper Info
Title
Selection models of language production support informed text partitioning: an intuitive and practical, bag-of-phrases framework for text analysis.
Abstract
The task of text segmentation, or u0027chunking,u0027 may occur at many levels in text analysis, depending on whether it is most beneficial to break it down by paragraphs of a book, sentences of a paragraph, etc. Here, we focus on a fine-grained segmentation task, which we refer to as text partitioning, where we apply methodologies to segment sentences or clauses into phrases, or lexical constructions of one or more words. In the past, we have explored (uniform) stochastic text partitioning---a process on the gaps between words whereby each space assumes one from a binary state of fixed (word binding) or broken (word separating) by some probability. In that work, we narrowly explored perhaps the most naive version of this process: random, or, uniform stochastic partitioning, where all word-word gaps are prescribed a uniformly-set breakage probability, q. Under this framework, the breakage probability is a tunable parameter, and was set to be pure-uniform: q = 1/2. In this work, we explore phrase frequency distributions under variation of the parameter q, and define non-uniform, or informed stochastic partitions, where q is a function of surrounding information. Using a crude but effective function for q, we go on to apply informed partitions to over 20,000 English texts from the Project Gutenberg eBooks database. In these analyses, we connect selection models to generate a notion of fit goodness for the u0027bag-of-termsu0027 (words or phrases) representations of texts, and find informed (phrase) partitions to be an improvement over the q = 1 (word) and q = 1/2 (phrase) partitions in most cases. This, together with the scalability of the methods proposed, suggests that the bag-of-phrases model should more often than not be implemented in place of the bag-of-words model, setting the stage for a paradigm shift in feature selection, which lies at the foundation of text analysis methodology.
Year
Venue
Field
2016
arXiv: Computation and Language
Feature selection,Computer science,Segmentation,Phrase,Text segmentation,Paragraph,Natural language processing,Artificial intelligence,Language production,Chunking (psychology),Machine learning,Scalability
DocType
Volume
Citations 
Journal
abs/1601.07969
0
PageRank 
References 
Authors
0.34
0
6
Name
Order
Citations
PageRank
Jake Ryland Williams1828.73
James P. Bagrow228126.25
Andrew Reagan3798.30
Sharon E. Alajajian400.34
Christopher M. Danforth557347.23
Peter Sheridan Dodds660350.13