blinklinmult.train.calibration

Per-corpus calibration of the eye-state score.

The frame-wise model ranks closed frames above open ones on the video corpora – threshold-free AUC 0.857 on RN, 0.917 on TalkingFace – but places its decision boundary in the wrong place there. Measured, a genuinely closed RN eye scores a median of 0.19, against 0.98 F1 on the still corpora.

The cause is a class-prior mismatch, not blindness: the sampler shows the model a ~50% closed world (CEW is 49.2% closed, MRL 81.9%) while RN is 1.3-1.6%. A model fitted to the first prior systematically under-predicts under the second.

Because the ranking is already good, one scalar per corpus recovers most of the loss without touching the weights. Two estimators are provided:

• fit_temperature() – one temperature on the logit, the standard recipe (Guo et al., 2017). Monotone, so it cannot change the ranking or the AUC; it moves only where the boundary falls.

• fit_threshold() – the cheaper baseline: pick the threshold that maximises validation F1 directly.

Prefer the threshold unless temperature clearly beats it. They address the same failure, and a threshold is one number a reader can interpret. Temperature earns its place only when the whole probability curve is needed – for a downstream expected-value decision, say – rather than a single cut.

Every parameter here is fitted on validation and applied to test. Fitting on test would report a number no deployment could reproduce. Corpora with no validation split (TalkingFace ships 0 valid windows, and MPEblink annotates no closure at all) therefore cannot be calibrated and must keep the universal parameter – which is exactly why the universal one is reported beside it.

Measured result: calibration does not improve the decision on RN.

Over ten recording-level splits of fw-focal’s RN test signals, fitting on half the recordings and scoring the other half:

method	test F1	beats universal
universal 0.37	0.4439 +/- 0.035	–
per-corpus threshold	0.4269 +/- 0.027	1 of 10
temperature + threshold	0.4261 +/- 0.027	1 of 10

The fitted thresholds are bimodal across splits – 0.30-0.36 on six, 0.55-0.56 on four – so the validation optimum is unstable and does not transfer. The temperature, by contrast, is stable at 0.58-0.71 and does what it claims: it cuts expected calibration error from 0.0355 to 0.0109.

So the probabilities really were mis-scaled, and temperature really does fix them – but the decision was already near its best at the universal threshold, and re-fitting per corpus buys nothing while adding a parameter and its variance.

This module is therefore not wired into the benchmark. It is kept because the ECE numbers are worth reporting and because the video models in Part 3 may sit in a different regime; it should be re-measured there rather than assumed useless. The lesson is that the prior mismatch has to be fixed where it is caused – in the sampler (Task 4) or by temporal context (Task 6) – not patched at the output.

Attributes

logger	Module-level logger.
EPSILON	Clamp keeping logit finite at probabilities of exactly 0 or 1.
DEFAULT_TEMPERATURES	Temperature grid searched by fit_temperature().
DEFAULT_THRESHOLDS	Threshold grid searched by fit_threshold().

Exceptions

CalibrationError

Raised when calibration is asked for something it cannot fit.

Functions

logit(→ numpy.ndarray)	Inverse sigmoid, clamped away from the asymptotes.
apply_temperature(→ numpy.ndarray)	Rescale probabilities by a temperature on the logit.
fit_threshold(→ tuple[float, float])	Pick the decision boundary that maximises F1.
fit_temperature(→ tuple[float, float])	Fit one temperature by validation negative log-likelihood.
expected_calibration_error(→ float)	Gap between confidence and accuracy, averaged over score bins.

Module Contents

blinklinmult.train.calibration.logger

Module-level logger.

blinklinmult.train.calibration.EPSILON = 1e-06

Clamp keeping logit finite at probabilities of exactly 0 or 1.

blinklinmult.train.calibration.DEFAULT_TEMPERATURES

Temperature grid searched by fit_temperature().

Geometric rather than linear: temperature acts multiplicatively on the logit, so 0.1-to-0.2 is the same size of change as 5-to-10 and a linear grid would spend most of its points where they alter the score least.

blinklinmult.train.calibration.DEFAULT_THRESHOLDS

Threshold grid searched by fit_threshold().

exception blinklinmult.train.calibration.CalibrationError

Bases: Exception

Raised when calibration is asked for something it cannot fit.

Initialize self. See help(type(self)) for accurate signature.

blinklinmult.train.calibration.logit(probability: numpy.ndarray) → numpy.ndarray

Inverse sigmoid, clamped away from the asymptotes.

Parameters:

probability (np.ndarray) – Values in [0, 1].

Returns:

Log-odds, finite even at exactly 0 or 1.

Return type:

np.ndarray

blinklinmult.train.calibration.apply_temperature(probability: numpy.ndarray, temperature: float) → numpy.ndarray

Rescale probabilities by a temperature on the logit.

Parameters:

• probability (np.ndarray) – Uncalibrated values in [0, 1].

• temperature (float) – Divisor on the logit. Below 1 sharpens, above 1 softens; 1 is the identity.

Returns:

Calibrated probabilities, same shape.

Return type:

np.ndarray

Raises:

CalibrationError – If the temperature is not positive.

blinklinmult.train.calibration.fit_threshold(probability: numpy.ndarray, target: numpy.ndarray, thresholds: numpy.ndarray | None = None) → tuple[float, float]

Pick the decision boundary that maximises F1.

Parameters:

• probability (np.ndarray) – Validation scores in [0, 1].

• target (np.ndarray) – Validation labels.

• thresholds (np.ndarray | None) – Grid to search; DEFAULT_THRESHOLDS by default.

Returns:

(threshold, its F1).

Return type:

tuple[float, float]

Raises:

CalibrationError – If the inputs disagree in length or hold no positive.

blinklinmult.train.calibration.fit_temperature(probability: numpy.ndarray, target: numpy.ndarray, temperatures: numpy.ndarray | None = None) → tuple[float, float]

Fit one temperature by validation negative log-likelihood.

NLL rather than F1: temperature is a probability correction, and fitting it to a thresholded score would tune it to one operating point and leave the rest of the curve arbitrary. The threshold is then chosen separately, on the calibrated scores.

Parameters:

• probability (np.ndarray) – Validation scores in [0, 1].

• target (np.ndarray) – Validation labels.

• temperatures (np.ndarray | None) – Grid to search; DEFAULT_TEMPERATURES by default.

Returns:

(temperature, its mean NLL).

Return type:

tuple[float, float]

Raises:

CalibrationError – If the inputs disagree in length or are empty.

blinklinmult.train.calibration.expected_calibration_error(probability: numpy.ndarray, target: numpy.ndarray, bins: int = 15) → float

Gap between confidence and accuracy, averaged over score bins.

Reported because F1 alone cannot show why a threshold had to move: a model can be perfectly ranked and badly calibrated at the same time, which is exactly the state measured on RN.

Parameters:

• probability (np.ndarray) – Scores in [0, 1].

• target (np.ndarray) – Binary labels.

• bins (int) – Equal-width bins over [0, 1].

Returns:

Weighted mean |confidence - accuracy|; 0 is perfect.

Return type:

float