Code for isotonic regression
For a binary classification task, suppose we have a model that produces real-valued scores for the training and test set. The aim of this MATLAB code is to convert the test scores into meaningful probability estimates. This is done by performing isotonic regression of the training scores with respect to the training labels, and interpolating.The included functions in the code are:
- testCalibration, a sample script that calibrates the outputs of linear regression
- calibrateScores, which performs calibration of a set of test scores by performing an isotonic fit for a set of training scores and labels
- isotonic_regression, the raw PAV algorithm implementation from here
Example usage
Suppose we have a training set of 10 instances. We have a model that produces the following list of scores s, where the corresponding binary labels are in y:
>> s = [ -1 -0.5 0 0.1 0.2 0.45 0.55 0.8 0.9 1 ];
>> y = [ 0 0 1 0 1 1 1 0 1 1 ];
We can convert the scores to probability estimates using isotonic regression.
>> sCal = calibrateScores(s, s, y);
>> disp(sCal);
...
>> plot(s, sCal, 'bo-');
We can further use the resulting isotonic fit to calibrate a set of test scores. This will perform linear interpolation by default.
>> sTest = [ -2 -0.25 0 0.5 2 ];
>> sTestCal = calibrateScores(sTest, s, y);
>> disp(sTestCal);
...
>> plot(s, sCal, 'bo-', sTest, sTestCal, 'ro-');
Detailed descriptions
Sample output on calling testCalibration:
>> testCalibration();
The function calibrateScores has three options for the tieBreak parameter.