How to create a probability calibrator in Machine Learning

João Barros 15 de October de 2026 6 min read

Calibrating probabilities in Machine Learning improves confidence in predictions from classification models (for example, when a model says 0.9, this should correspond to a 90% occurrence). This tutorial shows how to create a probability calibrator in Python using scikit-learn, explaining why and how to avoid common mistakes.

Prerequisites

Python 3.8+ with scikit-learn installed
Basic knowledge of binary classification (probabilities, metrics such as ROC/AUC)
IDE or Jupyter Notebook to run code

Step 1: Understand why to calibrate probabilities

Models like RandomForest or SVM do not always return well-calibrated probabilities. The output of predict_proba reflects the model's confidence according to its own structure, not necessarily the actual observed frequency. Calibrating adjusts the predicted probabilities so that the observed frequency approaches the predicted probabilities: for example, among instances where the model predicts ~0.8, about 80% should be positive.

Common mistakes: calibrating on the same training data (causes overfitting), using Isotonic with few examples (nonparametric overfitting), or forgetting to keep the same class distribution between training, calibration and test. In problems with 10% positives, calibration should preserve that stratification — ideally using stratify in splits or stratified cross-validation.

Step 2: Prepare example data

Use a simple dataset to practice. The breast_cancer dataset from scikit-learn has 569 samples: if you follow the split below (60% train, 20% calibration, 20% test), you will have approximately 341 samples for training and 114 for calibration and test each. Having a few hundred examples in the calibration phase is helpful, especially for Isotonic.

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split

X, y = load_breast_cancer(return_X_y=True)
X_train, X_temp, y_train, y_temp = train_test_split(X, y, test_size=0.4, random_state=42)
X_calib, X_test, y_calib, y_test = train_test_split(X_temp, y_temp, test_size=0.5, random_state=42)

Step 3: Train a base model without calibration

Train a standard classifier and get uncalibrated predictions to serve as a baseline. Here we use RandomForestClassifier because it is common in tabular problems. Measure the Brier score as the probabilistic quality metric — the closer to 0, the better.

from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import brier_score_loss

clf = RandomForestClassifier(random_state=42)
clf.fit(X_train, y_train)
probs_uncal = clf.predict_proba(X_test)[:, 1]
print('Brier (uncalibrated):', brier_score_loss(y_test, probs_uncal))

As a reference, in a typical case with this dataset you might see Brier ~0.06-0.12 depending on the seed; the goal is to reduce that value with calibration.

Step 4: Calibrate with CalibratedClassifierCV (Platt and Isotonic)

scikit-learn provides CalibratedClassifierCV which implements Platt scaling (method 'sigmoid') and Isotonic. Platt fits a logistic regression to the classifier scores and is parametric — less susceptible to overfitting when the calibration sample is small. Isotonic is nonparametric (free monotonic function) and can capture more complex relationships if there are many calibration samples.

from sklearn.calibration import CalibratedClassifierCV

# Platt (sigmoid)
calib_platt = CalibratedClassifierCV(base_estimator=clf, method='sigmoid', cv='prefit')
calib_platt.fit(X_calib, y_calib)
probs_platt = calib_platt.predict_proba(X_test)[:, 1]
print('Brier (Platt):', brier_score_loss(y_test, probs_platt))

# Isotonic
calib_iso = CalibratedClassifierCV(base_estimator=clf, method='isotonic', cv='prefit')
calib_iso.fit(X_calib, y_calib)
probs_iso = calib_iso.predict_proba(X_test)[:, 1]
print('Brier (Isotonic):', brier_score_loss(y_test, probs_iso))

In a hypothetical example, you might observe: Uncalibrated 0.09 → Platt 0.07 → Isotonic 0.05, when there is sufficient data. If Isotonic performs worse, it likely overfitted during calibration.

Step 5: Evaluate calibration (calibration curve and Brier)

Use the Brier score for an aggregate measure and the calibration curve to inspect where deviations occur. The curve groups probabilities into bins (e.g., 10) and compares the mean predicted probability with the observed positive fraction. If the curve is below the diagonal, the model tends to be optimistic; above, pessimistic.

from sklearn.calibration import calibration_curve
import matplotlib.pyplot as plt

def plot_calibration(y_true, probs_list, labels):
    plt.figure(figsize=(6,6))
    for probs, label in zip(probs_list, labels):
        frac_pos, mean_pred = calibration_curve(y_true, probs, n_bins=10)
        plt.plot(mean_pred, frac_pos, marker='o', label=label)
    plt.plot([0,1],[0,1],'k:', label='Perfect')
    plt.xlabel('Mean predicted probability')
    plt.ylabel('Observed positive fraction')
    plt.legend()
    plt.grid(True)
    plt.show()

plot_calibration(y_test, [probs_uncal, probs_platt, probs_iso], ['Uncalibrated','Platt','Isotonic'])

Interpret the points: if Platt follows the diagonal more closely than the uncalibrated model, you gained robustness. If Isotonic shows large fluctuations between bins, it can be a sign of too few samples per bin.

Step 6: Choose method and avoid overfitting

Practical rules: use Platt when you have from a few dozen to a few hundred examples for calibration; use Isotonic when you have several hundred or thousands and suspect a nonlinear relationship. Always prefer stratified cross-validation (cv=5) when there is no independent calibration set. For rare classes, increase the calibration set size or use resampling techniques carefully.

Verify the result

Confirm that the Brier score decreased and that the calibration curve moves closer to the diagonal for the chosen method. Also test the impact on business metrics: for example, if you apply a threshold of 0.7 to act, check changes in false positive/negative rates. If Isotonic worsens on the test set, it is a sign of overfitting during calibration — revert to Platt or increase the calibration set.

Conclusion

Calibrating probabilities in Machine Learning with CalibratedClassifierCV (Platt or Isotonic) is a simple way to make predictions more reliable, especially in applications sensitive to probabilities (for example, risk scoring, clinical screening). Next steps: try cross-validation for calibration, test other classifiers, use stratification and measure impact on decision thresholds. Practical tip: always reserve an independent set for calibration when possible and document Brier and calibration curve differences before/after.