Paper Info
Title
Improving The Boilerpipe Algorithm For Boilerplate Removal In News Articles Using Html Tree Structure
Abstract
It is well-known that the lack of quality data is a major problem for information retrieval engines. Web articles are flooded with non-relevant data such as advertising and related links. Moreover, some of these ads are loaded in a randomized way every time you hit a page, so the HTML document will be different and hashing of the content will be not possible. Therefore, we need to filter the non-relevant text of documents. The automatic extraction of relevant text in on-line text (news articles, etc.), is not a trivial task. There are many algorithms for this purpose described in the literature. One of the most popular ones is Boilerpipe and its performance is one of the best. In this paper, we present a method, which improves the precision of the Boilerpipe algorithm using the HTML tree for selection of the relevant content. Our filter greatly increases precision (at least 15%), at the cost of some recall, resulting in an overall F1-measure improvement (around 5%). We make the experiments for the news articles using our own corpus of 2,400 news in Spanish and 1,000 in English.
Year
DOI
Venue
2018
10.13053/CyS-22-2-2959
COMPUTACION Y SISTEMAS
Keywords
Field
DocType
Boilerplate removal, news extraction, HTML tree structure, Boilerpipe
Computer science,Boilerplate text,Algorithm,Tree structure,Artificial intelligence,Hash function,Natural language processing
Journal
Volume
Issue
ISSN
22
2
1405-5546
Citations 
PageRank 
References 
0
0.34
0
Authors
6
Name
Order
Citations
PageRank
Francisco Viveros Jiménez141.79
Miguel A. Sánchez-Pérez200.34
Helena Gómez-Adorno34016.01
Juan Pablo Posadas-Durán4133.72
Grigori Sidorov539860.51
Alexander Gelbukh62843269.19