Paper Info
Title
Web document text and images extraction using DOM analysis and natural language processing
Abstract
Web has emerged as the most important source of information in the world. This has resulted in need for automated software components to analyze web pages and harvest useful information from them. However, in typical web pages the informative content is surrounded by a very high degree of noise in the form of advertisements, navigation bars, links to other content, etc. Often the noisy content is interspersed with the main content leaving no clean boundaries between them. This noisy content makes the problem of information harvesting from web pages much harder. Therefore, it is essential to be able to identify main content of a web page and automatically isolate it from noisy content for any further analysis. Most existing approaches rely on prior knowledge of website specific templates and hand-crafted rules specific to websites for extraction of relevant content. We propose a generic approach that does not require prior knowledge of website templates. While HTML DOM analysis and visual layout analysis approaches have sometimes been used, we believe that for higher accuracy in content extraction, the analyzing software needs to mimic a human user and understand content in natural language similar to the way humans intuitively do in order to eliminate noisy content. In this paper, we describe a combination of HTML DOM analysis and Natural Language Processing (NLP) techniques for automated extractions of main article with associated images from web pages.
Year
DOI
Venue
2009
10.1145/1600193.1600241
ACM Symposium on Document Engineering
Keywords
Field
DocType
html dom analysis,natural language processing,typical web page,prior knowledge,content extraction,relevant content,informative content,web page,images extraction,noisy content,web document text,visual layout analysis approach,main content,natural language,software component,web pages,information content
Web page,Computer science,Information Harvesting,Web modeling,Document Object Model,Natural language processing,Artificial intelligence,Web design,Printer-friendly,Information retrieval,Information extraction,HTML,Database
Conference
Citations 
PageRank 
References 
9
0.57
9
Authors
2
Name
Order
Citations
PageRank
Parag Mulendra Joshi1101.29
Sam Liu2423.30