Paper Info
Title
Field-aware Calibration: A Simple and Empirically Strong Method for Reliable Probabilistic Predictions
Abstract
It is often observed that the probabilistic predictions given by a machine learning model can disagree with averaged actual outcomes on specific subsets of data, which is also known as the issue of miscalibration. It is responsible for the unreliability of practical machine learning systems. For example, in online advertising, an ad can receive a click-through rate prediction of 0.1 over some population of users where its actual click rate is 0.15. In such cases, the probabilistic predictions have to be fixed before the system can be deployed. In this paper, we first introduce a new evaluation metric named field-level calibration error that measures the bias in predictions over the sensitive input field that the decision-maker concerns. We show that existing post-hoc calibration methods have limited improvements in the new field-level metric and other non-calibration metrics such as the AUC score. To this end, we propose Neural Calibration, a simple yet powerful post-hoc calibration method that learns to calibrate by making full use of the field-aware information over the validation set. We present extensive experiments on five large-scale datasets. The results showed that Neural Calibration significantly improves against uncalibrated predictions in common metrics such as the negative log-likelihood, Brier score and AUC, as well as the proposed field-level calibration error.
Year
DOI
Venue
2020
10.1145/3366423.3380154
WWW '20: The Web Conference 2020 Taipei Taiwan April, 2020
Keywords
DocType
ISBN
Probabilistic prediction, Field-aware Calibration, Field-level Calibration Error, Neural Calibration
Conference
978-1-4503-7023-3
Citations 
PageRank 
References 
2
0.39
12
Authors
7
Name
Order
Citations
PageRank
Feiyang Pan1304.71
Xiang Ao211921.43
Pingzhong Tang313332.06
Min Lu4299.98
Dapeng Liu553.20
Xiao Lei6559.63
Qing He775480.58