Name
Title | ||
|---|---|---|
Evaluation of machine learning-based information extraction algorithms: criticisms and recommendations. |
Abstract | ||
|---|---|---|
We survey the evaluation methodology adopted in information extraction (IE), as defined in a few different efforts applying
machine learning (ML) to IE. We identify a number of critical issues that hamper comparison of the results obtained by different
researchers. Some of these issues are common to other NLP-related tasks: e.g., the difficulty of exactly identifying the effects
on performance of the data (sample selection and sample size), of the domain theory (features selected), and of algorithm
parameter settings. Some issues are specific to IE: how leniently to assess inexact identification of filler boundaries, the
possibility of multiple fillers for a slot, and how the counting is performed. We argue that, when specifying an IE task,
these issues should be explicitly addressed, and a number of methodological characteristics should be clearly defined. To
empirically verify the practical impact of the issues mentioned above, we perform a survey of the results of different algorithms
when applied to a few standard datasets. The survey shows a serious lack of consensus on these issues, which makes it difficult
to draw firm conclusions on a comparative evaluation of the algorithms. Our aim is to elaborate a clear and detailed experimental
methodology and propose it to the IE community. Widespread agreement on this proposal should lead to future IE comparative
evaluations that are fair and reliable. To demonstrate the way the methodology is to be applied we have organized and run
a comparative evaluation of ML-based IE systems (the Pascal Challenge on ML-based IE) where the principles described in this
article are put into practice. In this article we describe the proposed methodology and its motivations. The Pascal evaluation
is then described and its results presented. |
Year | DOI | Venue |
|---|---|---|
2008 | 10.1007/s10579-008-9079-3 | Language Resources and Evaluation |
Keywords | Field | DocType |
Evaluation methodology,Information extraction,Machine learning | Computer science,Computational linguistics,Algorithm,Domain theory,Information extraction,Artificial intelligence,Natural language processing,Sample selection,Machine learning,Sample size determination | Journal |
Volume | Issue | ISSN |
42 | 4 | 1574-0218 |
Citations | PageRank | References |
10 | 0.53 | 22 |
Authors | ||
8 | ||
Name | Order | Citations | PageRank |
|---|---|---|---|
| Alberto Lavelli | 1 | 615 | 55.37 |
| Mary Elaine Califf | 2 | 399 | 39.04 |
| Fabio Ciravegna | 3 | 1635 | 140.18 |
| Dayne Freitag | 4 | 2176 | 397.46 |
| Claudio Giuliano | 5 | 488 | 33.00 |
| Nicholas Kushmerick | 6 | 2414 | 275.22 |
| Lorenza Romano | 7 | 406 | 22.15 |
| Neil Ireson | 8 | 64 | 10.28 |