blinklinmult.compare.protocol

The comparison protocol: three tables, and why it takes three.

One table cannot compare these four models honestly, because they do not arrive on equal terms.

Table A – threshold-free (average precision). The primary claim. On RN30’s test split only 1.08% of frames are closed (3542 of 328 819), so accuracy is meaningless – a model that answers “open” every time scores 98.9% – and any F1 is a statement about the threshold as much as about the model. AP integrates over every threshold and is the one number no operating-point choice can flatter.

Table B – events at a fitted operating point. What each model achieves when someone actually tunes it. The threshold is swept on validation and applied to test, per model, so every model is judged after the same amount of care.

Table C – events as shipped. What a user gets from pip install today.

Tables B and C exist separately because of a real asymmetry that registry.py documents plainly: the three 1.x models carry threshold=0.5 because “nothing measured it”, while blinkcnn carries a pair swept on validation. Comparing only as-shipped numbers would measure calibration effort; comparing only fitted numbers would hide a defect a user really hits. The gap between B and C is the cost of that defect, made visible.

Fitting never touches test. fit.py says of itself: “the function does not enforce this; it cannot see where the signals came from.” A driver can, and compare_models() refuses when the two paths coincide.

Attributes

Signals

What load_signals() returns.

Exceptions

ProtocolError

Raised when a comparison is asked for on incomparable inputs.

Classes

Comparison

One pairwise result.

CorpusReport

Every model's score on one corpus, and every pairwise comparison.

Functions

compare_average_precision(→ CorpusReport)

Table A: threshold-free comparison of every model on one corpus.

Module Contents

blinklinmult.compare.protocol.Signals

What load_signals() returns.

exception blinklinmult.compare.protocol.ProtocolError

Bases: ValueError

Raised when a comparison is asked for on incomparable inputs.

Initialize self. See help(type(self)) for accurate signature.

class blinklinmult.compare.protocol.Comparison

One pairwise result.

Parameters:

  • first (str) – The model on the left of the difference.

  • second (str) – The model subtracted from it.

  • interval (Interval) – The difference, with its confidence interval.

  • p_raw (float) – Two-sided bootstrap p-value, uncorrected.

  • p_holm (float) – The same after Holm correction across the family.

property conclusive: bool

Whether the corrected result separates the two models.

Both conditions must hold: an interval excluding zero and a corrected p below 0.05. They can disagree at the margin, and a claim should rest on the stricter reading.

Returns:

Whether the difference stands after correction.

Return type:

bool

describe() → str

One line for a table.

Returns:

The difference, its interval, and the corrected p.

Return type:

str

class blinklinmult.compare.protocol.CorpusReport

Every model’s score on one corpus, and every pairwise comparison.

Parameters:

  • corpus (str) – Which corpus.

  • n_clusters (int) – Recordings behind the numbers – the honest sample size, which is why it is reported next to them.

  • n_frames (int) – Valid frames scored.

  • positive_rate (float) – Fraction of frames labelled closed. Printed because at ~1% it is why average precision leads.

  • scores (dict[str, Interval]) – Per-model average precision.

  • comparisons (list[Comparison]) – Every pair, Holm-corrected together.

property resolvable: bool

Whether this corpus can carry an interval at all.

Returns:

False for a corpus like TalkingFace, whose single recording supports a point estimate and nothing more.

Return type:

bool

blinklinmult.compare.protocol.compare_average_precision(stores: dict[str, Signals], corpus: str, rounds: int = DEFAULT_ROUNDS, seed: int = DEFAULT_SEED) → CorpusReport

Table A: threshold-free comparison of every model on one corpus.

Each model is scored on the observed recordings and on every bootstrap resample of them. The same draw is used for every model, so a resample heavy on hard recordings moves all of them together and the difference stays stable where the absolutes do not.

Parameters:

  • stores (dict[str, Signals]) – Per-model signals, from load_signals(). Loading through that function rather than numpy.load() is what stops a blink_presence archive being scored as eye_state.

  • corpus (str) – Name, for the report.

  • rounds (int) – Bootstrap replicates.

  • seed (int) – Seed, recorded in the result.

Returns:

Per-model intervals and every pairwise comparison.

Return type:

CorpusReport

Raises:

ProtocolError – If fewer than two models are given or they disagree on which recordings they cover.