Paper Info
Title
Assessing the state of the art in biomedical relation extraction: overview of the BioCreative V chemical-disease relation (CDR) task.
Abstract
Manually curating chemicals, diseases and their relationships is significantly important to biomedical research, but it is plagued by its high cost and the rapid growth of the biomedical literature. In recent years, there has been a growing interest in developing computational approaches for automatic chemical-disease relation (CDR) extraction. Despite these attempts, the lack of a comprehensive benchmarking dataset has limited the comparison of different techniques in order to assess and advance the current state-of-the-art. To this end, we organized a challenge task through BioCreative V to automatically extract CDRs from the literature. We designed two challenge tasks: disease named entity recognition (DNER) and chemical-induced disease (CID) relation extraction. To assist system development and assessment, we created a large annotated text corpus that consisted of human annotations of chemicals, diseases and their interactions from 1500 PubMed articles. 34 teams worldwide participated in the CDR task: 16 (DNER) and 18 (CID). The best systems achieved an F-score of 86.46% for the DNER task-a result that approaches the human inter-annotator agreement (0.8875)-and an F-score of 57.03% for the CID task, the highest results ever reported for such tasks. When combining team results via machine learning, the ensemble system was able to further improve over the best team results by achieving 88.89% and 62.80% in F-score for the DNER and CID task, respectively. Additionally, another novel aspect of our evaluation is to test each participating system's ability to return real-time results: the average response time for each team's DNER and CID web service systems were 5.6 and 9.3 s, respectively. Most teams used hybrid systems for their submissions based on machining learning. Given the level of participation and results, we found our task to be successful in engaging the text-mining research community, producing a large annotated corpus and improving the results of automatic disease recognition and CDR extraction.
Year
DOI
Venue
2016
10.1093/database/baw032
DATABASE-THE JOURNAL OF BIOLOGICAL DATABASES AND CURATION
Field
DocType
Volume
Data mining,Disease,Information retrieval,Computer science,Text corpus,Bioinformatics,System development,Web service,Named-entity recognition,Benchmarking,Relationship extraction
Journal
2016
ISSN
Citations 
PageRank 
1758-0463
33
0.97
References 
Authors
26
8
Name
Order
Citations
PageRank
Chih-Hsuan Wei154627.43
Yifan Peng211416.15
Robert Leaman391439.98
Allan Peter Davis444424.76
Carolyn J. Mattingly549529.93
Jiao Li6582.30
Thomas C. Wiegers760330.77
Zhiyong Lu82735171.27