Name
Title | ||
|---|---|---|
Receiver operating characteristic analysis for intelligent medical systems-a new approach for finding confidence intervals. |
Abstract | ||
|---|---|---|
Abstract—Intelligent systems are increasingly being deployed in medicine and healthcare, but there is a need for a robust and objective methodology for evaluating such systems. Potentially, re- ceiver operating characteristic (ROC) analysis could form a basis for the objective evaluation of intelligent medical systems. How- ever, it has several weaknesses when applied to the types of data used to evaluate intelligent medical systems. First, small data sets are often used, which are unsatisfactory with existing methods. Second, many existing ROC methods use parametric assumptions which may not always be valid for the test cases selected. Third, system evaluations are often more concerned with particular, clini- cally meaningful, points on the curve, rather than on global indexes such as the more commonly,used area under the curve. A novel, robust and accurate method is proposed, derived from first principles, which calculates the probability density function (pdf) for each point on a ROC curve for any given sample size. Confidence intervals are produced as contours on the pdf. The the- oretical work has been validated by Monte Carlo simulations. It has also been applied to two real-world examples of ROC analysis, taken from the literature (classification of mammograms,and dif- ferential diagnosis of pancreatic diseases), to investigate the confi- dence surfaces produced for real cases, and to illustrate how anal- ysis of system performance can be enhanced. We illustrate the im- pact of sample size on system performance from analysis of ROC pdf’s and 95% confidence boundaries. This work establishes an important new method for generating pdf’s, and provides an ac- curate and robust method of producing confidence intervals for ROC curves for the small sample sizes typical of intelligent med- ical systems. It is conjectured that, potentially, the method could be extended to determine risks associated with the deployment of intelligent medical systems in clinical practice. Index Terms—Confidence interval, intelligent medical system, |
Year | DOI | Venue |
|---|---|---|
2000 | 10.1109/10.846690 | IEEE Trans. Biomed. Engineering |
Keywords | Field | DocType |
monte carlo methods,area under the curve,monte carlo simulations,probability density function,indexation,sample size,testing,first principle,roc analysis,roc curve,artificial intelligence,robustness,monte carlo simulation,intelligent systems,receiver operator characteristic,healthcare,system performance,confidence interval,probability | Data mining,Computer vision,Receiver operating characteristic,Small data,Intelligent decision support system,Computer science,Robustness (computer science),Parametric statistics,Test case,Artificial intelligence,Confidence interval,Sample size determination | Journal |
Volume | Issue | ISSN |
47 | 7 | 0018-9294 |
Citations | PageRank | References |
21 | 2.02 | 3 |
Authors | ||
5 | ||
Name | Order | Citations | PageRank |
|---|---|---|---|
| Julian B. Tilbury | 1 | 21 | 2.02 |
| Peter W. J. Van Eetvelt | 2 | 21 | 2.02 |
| J. M. Garibaldi | 3 | 1425 | 146.38 |
| John S. H. Curnow | 4 | 21 | 2.02 |
| Emmanuel C. Ifeachor | 5 | 255 | 26.56 |