Name
Abstract | ||
|---|---|---|
We address the problem of evaluating the risk of a given model accurately at minimal la- beling costs. This problem occurs in situa- tions in which risk estimates cannot be ob- tained from held-out training data, because the training data are unavailable or do not re- ect the desired test distribution. We study active risk estimation processes in which in- stances are actively selected by a sampling process from a pool of unlabeled test in- stances and their labels are queried. We de- rive the sampling distribution that minimizes the estimation error of the active risk esti- mator when used to select instances from the pool. An analysis of the distribution that governs the estimator leads to condence in- tervals. We empirically study conditions un- der which the active risk estimate is more accurate than a standard risk estimate that draws equally many instances from the test distribution. |
Year | Venue | Keywords |
|---|---|---|
2010 | ICML | empirical study |
Field | DocType | Citations |
Sampling distribution,Risk Estimate,Training set,Sampling process,Standard Risk,Computer science,Artificial intelligence,Statistics,Confidence interval,Machine learning,Estimator | Conference | 11 |
PageRank | References | Authors |
0.85 | 8 | 4 |
Name | Order | Citations | PageRank |
|---|---|---|---|
| Christoph Sawade | 1 | 55 | 6.21 |
| Niels Landwehr | 2 | 506 | 31.54 |
| Steffen Bickel | 3 | 848 | 58.84 |
| Tobias Scheffer | 4 | 1862 | 139.64 |