“Measuring the expected prediction error is very important in risk-sensitive areas where ... appropriate local accuracy measures may provide additional necessary information about the prediction confidence.”

This paper presents a new method for reliability estimation of individual predictions based on sensitivity analysis, using the ideas of Bousquet and Elisseeff [1] and of Kearns and Ron [2].

Using the difference in predictions of initial and sensitivity models, Bosnić and Kononenko compose the reliability estimates based on local variance, local absolute variance, and local bias, to measure instabilities of regression models that arise from the learning algorithm itself. The most promising results are achieved using local bias; it is correlated to the nonabsolute value of the prediction error, and therefore holds the potential for correction of prediction errors. The authors show that the proposed sensitivity estimates compare favorably to density estimates, and also achieve better experimental results. They also show that the suggested methodology is appropriate for regression trees, neural networks, and support vector machines, and test the proposed estimates with these three models. The proposed method is tested on 48 datasets, and the results confirm the usability of their approach.