blinklinmult.compare.tests¶

Hypothesis tests and the correction for running six of them.

Two tests, answering different questions, reported side by side because the contrast is the informative part.

mcnemar() conditions on these frames and asks whether two models disagree asymmetrically. two_sided_p() resamples recordings and asks whether one model would beat the other on a fresh sample of subjects. The second is the question a deployment decision turns on; the first will report a far smaller p-value, because frames within a blink are near-copies and McNemar treats them as independent evidence. That divergence is not a contradiction and it is not a bug – it is the difference between “these classifiers differ on this data” and “this model is better”, and being able to say which one a number supports is the point of reporting both.

holm_adjust() then corrects for the fact that four models make six comparisons, and six chances to see something is six chances to be fooled.

Exceptions¶

TestError

Raised when a hypothesis test is given inputs it cannot interpret.

Functions¶

`mcnemar`(→ tuple[int, int, float])	Exact McNemar test on two models' per-frame correctness.
`holm_adjust`(→ dict[str, float])	Holm-Bonferroni correction across a family of comparisons.
`bonferroni_adjust`(→ dict[str, float])	Bonferroni correction: multiply every p-value by the family size.
`benjamini_hochberg_adjust`(→ dict[str, float])	Benjamini-Hochberg correction, controlling the false discovery rate.

Module Contents¶

exception blinklinmult.compare.tests.TestError[source]¶

Bases: ValueError

Raised when a hypothesis test is given inputs it cannot interpret.

Initialize self. See help(type(self)) for accurate signature.

blinklinmult.compare.tests.mcnemar(first_correct: numpy.ndarray, second_correct: numpy.ndarray) → tuple[int, int, float][source]¶

Exact McNemar test on two models’ per-frame correctness.

Only the discordant frames carry information: those both models get right, or both get wrong, say nothing about which is better. Under the null the discordant frames split like a fair coin.

Exact binomial, not the chi-squared approximation. At the measured 1.08% positive rate on RN30 the discordant frames concentrate on a few thousand positives, and for some model pairs the count is small enough that the approximation misleads. The exact test costs nothing here.

Its p-value is anti-conservative and must be reported as such. Frames inside one blink are near-duplicates, so the effective sample size is far below the frame count and this will overstate significance. See the module docstring.

Parameters:

first_correct (np.ndarray) – Boolean, one entry per valid frame.
second_correct (np.ndarray) – Boolean, aligned with the first.

Returns:

Frames only the first got right, frames only the second got right, and the two-sided p-value.

Return type:

tuple[int, int, float]

Raises:

TestError – If the two arrays are not the same length, which would mean they are not aligned frame for frame.

blinklinmult.compare.tests.holm_adjust(p_values: collections.abc.Mapping[str, float]) → dict[str, float][source]¶

Holm-Bonferroni correction across a family of comparisons.

Four models make six pairwise comparisons, and six independent chances to clear a 5% bar means a ~26% chance of at least one false positive if the bar is not raised.

Holm rather than Benjamini-Hochberg. BH controls the proportion of false claims among rejections, which suits screening thousands of hypotheses where a few false leads are cheap and get followed up. Here each rejection becomes a claim in a table a reader takes as true, so the guarantee worth having is familywise: at most a 5% chance of any false claim in the family. Holm delivers that without assuming the tests are independent – and they are not, since all six share models – while being uniformly more powerful than plain Bonferroni.

Adjusted p-values are returned rather than reject/accept flags, so the significance level stays the reader’s choice.

Parameters:: p_values (Mapping[str, float]) – Raw p-values, keyed by comparison.
Returns:: Adjusted p-values, in the input’s key order.
Return type:: dict[str, float]
Raises:: TestError – If any p-value falls outside [0, 1].

blinklinmult.compare.tests.bonferroni_adjust(p_values: collections.abc.Mapping[str, float]) → dict[str, float][source]¶

Bonferroni correction: multiply every p-value by the family size.

The simplest family-wise correction, and strictly worse than :func:`holm_adjust`: Holm gives the identical guarantee while rejecting at least as much, and usually more. It is provided for comparison rather than for use – seeing the three side by side is what makes the choice legible.

Parameters:: p_values (Mapping[str, float]) – Raw p-values, keyed by comparison.
Returns:: Adjusted p-values, in the input’s key order.
Return type:: dict[str, float]
Raises:: TestError – If any p-value falls outside [0, 1].

blinklinmult.compare.tests.benjamini_hochberg_adjust(p_values: collections.abc.Mapping[str, float]) → dict[str, float][source]¶

Benjamini-Hochberg correction, controlling the false discovery rate.

A different guarantee from Holm and Bonferroni, not a weaker version of one. Those bound the chance of making any false claim; this bounds the expected proportion of false claims among the claims made. With many tests that is far more powerful, and it is the right target when the output is a shortlist to investigate rather than a set of conclusions.

Not what this repo uses. Six model comparisons that go into a table a reader takes as true call for the family-wise guarantee – see holm_adjust(). On the worked example in the tests, BH calls all six comparisons significant where Holm calls none, which is the trade-off in one line.

Parameters:: p_values (Mapping[str, float]) – Raw p-values, keyed by comparison.
Returns:: Adjusted p-values, in the input’s key order.
Return type:: dict[str, float]
Raises:: TestError – If any p-value falls outside [0, 1].