Paper Info
Title
Language independent web news extraction system based on text detection framework.
Abstract
Web news provides a direct and efficient way to construct large text corpora. The creation of text data requires an understanding of HTML code and the preparation of customized parsing rules to identify text content in a webpage. Typically, parsing rules are written manually and cannot be applied to pages with different layouts. In this study, we present a web news extraction system that is based on a text detection framework. The proposed method scans the input HTML page and creates text statistics as a projection profile. Then, text block identification is applied to determine a set of content candidates. To filter noise, text verification determines whether a given text block can be included with content. We evaluate the proposed approach with the L3S-GN1 corpus and 3506 multilingual news data items randomly sampled from 325 websites (15 geographic regions and 11 distinct languages). We also compare the proposed method to 23 well-known state-of-the-art techniques. The experimental results show that the proposed method outperforms the second best method (NReadability) by 7.30% in the macro F-measure rate and is 16.91 times faster than NReadability. In terms of the perfect rate, the proposed method demonstrates 46.38% accuracy, whereas the Boilerpipe algorithm demonstrates only 21.54% accuracy. The proposed method is very useful for constructing a multilingual corpus because it requires no language-specific processing component.
Year
DOI
Venue
2016
10.1016/j.ins.2015.12.025
Information Sciences
Keywords
Field
DocType
Content extraction,Information filtering,Web mining,Block segmentation,HTML
Text mining,Web mining,Noisy text analytics,Web news,Web page,Information retrieval,Computer science,Text corpus,Parsing,Macro
Journal
Volume
Issue
ISSN
342
C
0020-0255
Citations 
PageRank 
References 
3
0.39
27
Authors
1
Name
Order
Citations
PageRank
Yu-Chieh Wu124723.16