Probability calibration, log loss, and Brier score

A classifier is calibrated when events assigned probability $p$ occur about a fraction $p$ of the time. The binary Brier score is

$BS=\frac{1}{n}\sum_{i=1}^{n}(p_i-y_i)^2,$

and log loss is

$-\frac{1}{n}\sum_i[y_i\log p_i+(1-y_i)\log(1-p_i)].$

from sklearn.metrics import brier_score_loss, log_loss
from sklearn.calibration import calibration_curve

y_true = [0, 0, 0, 1, 1, 1]
prob = [0.05, 0.20, 0.70, 0.55, 0.80, 0.95]
print(brier_score_loss(y_true, prob))
print(log_loss(y_true, prob))
fraction_positive, mean_predicted = calibration_curve(y_true, prob, n_bins=3)

Case study: quality-control triage

If an imaging system sends samples above 0.8 probability for manual review, calibration determines whether “0.8” has operational meaning. Discrimination can be strong while probabilities are overconfident. Fit calibration only on data separate from model training and evaluate under the deployment prevalence.

Functions: sklearn.metrics.brier_score_loss, log_loss, and sklearn.calibration.calibration_curve.

Continue through the concepts

Related notes

Log loss

Log loss: theory, interpretation, Python computation, and a scientific-imaging case study.

Brier score

Brier score: theory, interpretation, Python computation, and a scientific-imaging case study.

Classification metrics from the confusion matrix

Accuracy, balanced accuracy, precision, recall, specificity, F1, MCC, and choosing metrics under class imbalance.

ROC AUC and precision–recall curves

Threshold sweeps, ranking performance, average precision, prevalence, and why ROC and PR answer different questions.