The oldest measure is the Brier score, the mean squared difference between forecast probability and outcome (Brier, 1950). Murphy (1973) showed that it separates into a reliability term, which is the calibration component, a resolution term and an uncertainty term fixed by the data. The Brier score and the negative log-likelihood are proper scoring rules: their expected value is optimised only by reporting the true probabilities (Gneiting and Raftery, 2007). They cannot be gamed, but they mix calibration with discrimination in a single figure.

The expected calibration error isolates calibration. Predictions are sorted into bins by confidence and the gap between mean confidence and observed accuracy is averaged across bins, weighted by bin size (Naeini et al., 2015). The measure is intuitive and pairs naturally with a reliability diagram. It is also fragile. Its value depends on the number of bins and on whether they have equal width or equal mass; in multi-class problems it usually examines only the top predicted class; and Nixon et al. (2019) documented how these choices alter the ranking of methods. Kumar et al. (2019) proved that the binned estimate can understate the true error of a continuous recalibration method and proposed a debiased estimator.

Post-hoc methods adjust the outputs of a trained model using held-out data. Platt scaling fits a logistic function to the scores (Platt, 1999); isotonic regression fits a monotone step function (Zadrozny and Elkan, 2002); and the two were compared across classical learners by Niculescu-Mizil and Caruana (2005). Temperature scaling is the single-parameter relative of Platt scaling for multi-class networks (Guo et al., 2017). Other methods act during training: label smoothing softens the targets (Müller et al., 2019), and averaging several independently trained networks, a deep ensemble, improves both accuracy and uncertainty estimates (Lakshminarayanan et al., 2017).

Three features of clinical image data matter. Datasets are small and classes are unbalanced, so bins at high confidence for a rare disease may hold very few positive cases and any binned estimate is noisy. One patient often contributes several images, and if images from the same patient fall on both sides of a split the test is contaminated. And the deployment population differs from the development population in scanner, protocol and disease prevalence. Ovadia et al. (2019) found that calibration deteriorates as data move away from the training distribution and that post-hoc scaling tuned on in-distribution data is among the methods that degrade, a finding with obvious weight for a model carried from one hospital to another.

The benchmark begins with a written definition of the prediction target, including how labels were obtained and whether the problem is binary, multi-class or multi-label. Data are then divided at the level of the patient into four parts: training, validation for model selection, a calibration split used only to fit post-hoc methods, and a test split that nothing is fitted on. The fourth split is the one most often missing in published work. When recalibration parameters are tuned on the validation data that also chose the model, or on the test set itself, the reported calibration is optimistic. Split files are published so that others evaluate on identical cases.

Each model is trained with a fixed and published configuration, and training is repeated with several random seeds. Calibration is sensitive to choices that are rarely reported, among them the stopping criterion, weight decay and augmentation, so these belong in the record. Repetition matters because the difference between two calibration methods can be smaller than the variation between two runs of the same method.

Post-hoc calibrators are fitted on the calibration split and frozen. The protocol asks for at least the uncalibrated model, temperature scaling and one non-parametric method such as isotonic regression, so that every benchmark shares a common set of reference points. Training-time methods and ensembles are entered as separate models and pass through the same later steps.

No single number is adequate, so the protocol fixes a small set. The Brier score and negative log-likelihood are reported as proper scores. Expected calibration error is reported with equal-mass bins, with the number of bins stated, together with a class-wise variant of the kind described by Nixon et al. (2019), because top-label error hides miscalibration in rare classes. A reliability diagram accompanies each model. Discrimination is reported alongside, following the framework of Steyerberg et al. (2010), who treat overall performance, discrimination and calibration as separate questions. Where the intended clinical decision is known, net benefit from decision curve analysis may be added (Vickers and Elkin, 2006).

Every reported value carries an interval. The protocol specifies a nonparametric bootstrap of the test set (Efron and Tibshirani, 1993) in which patients, not images, are resampled, so that correlated images from one person stay together. Variation across training seeds is reported separately, since it answers a different question: the bootstrap describes uncertainty about this model's calibration, and the seeds describe how much the training procedure itself varies.

The metric set is computed again on at least one test set drawn from a different institution or acquisition device, without refitting the calibrators. Results are also broken down by subgroups that were named before the analysis, typically sex, age band and scanner type where the data allow. A model that is calibrated on average may be overconfident for one group and underconfident for another, and the averages conceal it. Small subgroups will have wide intervals, and the protocol asks that these be shown and not suppressed.

The benchmark is reproducible only if its parts are available. Code for training and for the metrics, the split files, fitted calibration parameters and the per-image predicted probabilities for every model and seed are released under a stated licence. Per-image predictions are the most valuable of these, because they allow a later reader to compute a metric that the original authors did not.