blinklinmult.train.model¶

The BlinkLinMulT architecture: an eye-crop CNN feeding a LinMulT sequence model.

Three families share one construction path, because they are the paper’s three rows of the same table:

cnn: The per-frame backbone with a classification head and no sequence model — the DenseNet121 eye-state baseline.
lint: Backbone embeddings through a single-stream linmult.LinT — the BlinkLinT baseline.
linmult: Backbone embeddings and handcrafted eye features through the cross-modal linmult.LinMulT — the published BlinkLinMulT.

One sample is one eye. The encoder embeds a single eye’s window; the two eyes of a frame are separate samples, recombined only at evaluation time. That matches how the corpora annotate (per eye) and lets the corpora that supply one eye, or label the whole face, be represented honestly.

What changed from 1.x. The old BlinkLinMulT.forward embedded frames with a Python loop over the time axis, one backbone call per timestep, and hard-coded a 1024-d DenseNet embedding into the LinMulT construction. Here every crop in the batch is embedded in a single fused call (fold_time_into_batch()), and the embedding width is read from the constructed backbone rather than assumed — so swapping DenseNet121 for ResNet50 is a config edit, not a code edit.

Heads. Both tasks are sequence-level: they predict one value per timestep, not one per clip, so both heads are linmult sequence heads. The frame- and clip-level decisions the tasks are scored on are aggregations of the sequence (see blinklinmult.train.metrics), which keeps the temporal localisation the model learns rather than discarding it at the head.

Attributes¶

`logger`	Module-level logger.
`SEQUENCE_HEAD`	Name of the single head every supervised target reads.
`EVENT_HEAD`	Name of the optional head stacked on the ESR signal.
`EVENT_HEAD_TYPES`	Event-head architectures.
`EVENT_CONV_KERNEL`	Temporal receptive field of the `conv` event head, in frames.
`SUPPORTED_TARGETS`	Targets the single head can serve.
`HEAD_OUTPUT_DIM`	one value per timestep.
`IMAGENET_MEAN`	Per-channel mean the pretrained backbones were trained against.
`IMAGENET_STD`	Per-channel standard deviation the pretrained backbones were trained against.
`TIMM_WEIGHTS`	Backbones served by timm, mapped to the pretrained tag used.
`DENSENET_CUT`	First DenseNet block dropped by `densenet121_truncated`.
`MIN_IMAGE_SIZE`	Smallest eye crop the convolutional backbones accept.
`ENCODER_PREFIX`	Prefix the eye encoder's weights carry inside a Lightning checkpoint.

Exceptions¶

ModelError

Raised when a model cannot be built as configured.

Classes¶

`TimmBackbone`	Wraps a timm model so it pools to `(N, D)` like the torchvision path.
`EyeEncoder`	Embeds a window of eye crops into a per-timestep feature sequence.
`CnnHead`	Per-frame classification head for the `cnn` family.
`EventHead`	Predicts blink intervals from the ESR closure signal.
`BlinkModel`	The full model: eye encoder, optional sequence model, and task heads.

Functions¶

`build_backbone`(→ tuple[torch.nn.Module, int])	Construct an image backbone and report its embedding width.
`load_encoder_weights`(→ None)	Initialise an eye encoder from a trained frame-wise checkpoint.
`check_targets`(→ None)	Reject targets the single head cannot serve.
`build_model`(→ BlinkModel)	Construct the model for the configured family.

Module Contents¶

blinklinmult.train.model.logger¶: Module-level logger.

blinklinmult.train.model.SEQUENCE_HEAD = 'eye_openness'¶

Name of the single head every supervised target reads.

Both tasks are decided from one per-timestep sequence, exactly as in the paper: eye state recognition reads it frame by frame, and blink presence detection takes the max over a window. Two independent heads would fit the annotation slightly better but could disagree – a window scored “no frame shows a closed eye” and “a blink occurred” is incoherent, and nothing in a two-head model forbids it. One head makes that inconsistency unrepresentable.

blinklinmult.train.model.EVENT_HEAD = 'blink_event'¶

Name of the optional head stacked on the ESR signal.

Reads SEQUENCE_HEAD’s per-timestep closure score and predicts the blink interval from it, rather than predicting both from one set of parameters.

Why stack rather than share or split. Sharing one head makes the two annotations compete: a blink interval covers 3-4x more frames than actual closure (measured P(closed | inside a blink) = 0.237 / 0.310 / 0.344 on RN15 / RN30 / TalkingFace), so a corpus that annotates only intervals pulls the shared parameters toward a wider target. On the video benchmark 72% of batches did exactly that, and RN30’s ESR F1 fell to 0.4107 against the frame-wise model’s 0.6614.

Two parallel heads would remove the competition but allow incoherence – one head reporting “no frame shows a closed eye” while the other reports “a blink occurred”. Stacking keeps the event prediction a function of the closure signal, so that contradiction stays unrepresentable, which was the reason the single head was chosen in the first place.

blinklinmult.train.model.EVENT_HEAD_TYPES: frozenset[str]¶

Event-head architectures.

Both see neighbouring timesteps, which a pointwise map cannot. That matters because the closure-to-interval mapping is not a per-frame decision: a blink is a run of elevated closure roughly 200-330 ms long, and the interval covers 3-4x more frames than the closure itself. A head reading one scalar at a time could only learn a monotone rescaling – a threshold, which the extractor already applies for free.

conv is a temporal convolution over the ESR sequence: local, cheap, and exactly the smoothing-and-widening rule the mapping needs. attention is one self-attention layer, which can additionally relate distant timesteps – useful if blink boundaries depend on the wider context rather than a fixed window.

blinklinmult.train.model.EVENT_CONV_KERNEL = 9¶

Temporal receptive field of the conv event head, in frames.

At 25-30 fps a blink spans roughly 5-10 frames, so a 9-frame window covers one whole blink and its immediate surroundings – enough to turn a closure apex into the interval around it. Padded to preserve length, so the output stays aligned with the input timestep for timestep.

blinklinmult.train.model.SUPPORTED_TARGETS: frozenset[str]¶: Targets the single head can serve.

blinklinmult.train.model.HEAD_OUTPUT_DIM = 1¶

one value per timestep.

A sample carries a single eye, so the eye axis lives in the sample identity rather than in the head. Frame-level predictions are recovered by aggregating the two eyes afterwards – see FrameAggregator.

Type:: Width of the head

blinklinmult.train.model.IMAGENET_MEAN = (0.485, 0.456, 0.406)¶: Per-channel mean the pretrained backbones were trained against.

blinklinmult.train.model.IMAGENET_STD = (0.229, 0.224, 0.225)¶: Per-channel standard deviation the pretrained backbones were trained against.

exception blinklinmult.train.model.ModelError¶

Bases: ValueError

Raised when a model cannot be built as configured.

Initialize self. See help(type(self)) for accurate signature.

class blinklinmult.train.model.TimmBackbone(network: torch.nn.Module)¶

Bases: torch.nn.Module

Wraps a timm model so it pools to (N, D) like the torchvision path.

timm’s forward_features returns the spatial map, and its own pooling is entangled with a classifier this model does not use. Pooling here keeps both backbone families behind one interface.

Parameters:: network (nn.Module) – A timm model created with num_classes=0.

forward(images: torch.Tensor) → torch.Tensor¶

Embed a batch of crops.

Parameters:: images (torch.Tensor) – (N, 3, H, W).
Returns:: (N, D).
Return type:: torch.Tensor

blinklinmult.train.model.TIMM_WEIGHTS: dict[str, str]¶

Backbones served by timm, mapped to the pretrained tag used.

The tag is pinned rather than left to timm’s default so a run is reproducible: timm moves its default weights between releases, and efficientnet_b0 alone ships several checkpoints that differ by several points of ImageNet accuracy.

blinklinmult.train.model.DENSENET_CUT = 'denseblock4'¶: First DenseNet block dropped by densenet121_truncated.

blinklinmult.train.model.build_backbone(name: str, pretrained: bool = True, wide_stem: bool = False, freeze_hint: bool = False) → tuple[torch.nn.Module, int]¶

Construct an image backbone and report its embedding width.

The width is read off the constructed network rather than hard-coded, so a library change to a backbone’s final layer cannot silently produce a model whose projection expects the wrong size.

Parameters:

name (str) – Backbone name.
pretrained (bool) – Start from ImageNet weights.
wide_stem (bool) – Drop one stride-2 reduction, so a 64px crop yields a 4x4 feature map instead of 2x2. See _widen_stem(); this expects fine-tuning.
freeze_hint (bool) – Whether the caller intends to freeze the backbone, used only to warn about the combination with wide_stem.

Returns:

The feature extractor, mapping (N, 3, H, W) to (N, D), and its width D.

Return type:

tuple[nn.Module, int]

Raises:

ModelError – If the backbone name is unknown.

blinklinmult.train.model.MIN_IMAGE_SIZE = 32¶

Smallest eye crop the convolutional backbones accept.

All three backbones downsample by 32 in total, so a smaller crop collapses to a zero-sized feature map inside their final pooling layer. Torch reports that as Calculated output size: (512x0x0) from somewhere deep in avg_pool2d, which says nothing about the crop size being the cause — hence the explicit check.

blinklinmult.train.model.ENCODER_PREFIX = 'model.encoder.'¶

Prefix the eye encoder’s weights carry inside a Lightning checkpoint.

BlinkLightningModule holds a BlinkModel as self.model, whose encoder is self.encoder – so a checkpoint’s state_dict names them model.encoder.backbone.* and model.encoder.project.*.

blinklinmult.train.model.load_encoder_weights(encoder: torch.nn.Module, path: str | pathlib.Path) → None¶

Initialise an eye encoder from a trained frame-wise checkpoint.

The frame-wise BlinkCNN and the sequence models share exactly one component: this encoder, backbone and projection together. Everything after it differs (BlinkCNN has a per-frame head; LinT and LinMulT have a transformer), so only the encoder transfers — and it transfers as a starting point, not frozen. The sequence models fine-tune it at optimizer.backbone_lr.

Loading is strict about shape and lenient about nothing else: a checkpoint trained with a different backbone or a different backbone_output_dim has tensors that cannot be assigned, and silently skipping them would leave a randomly-initialised encoder that looks pretrained. Any mismatch raises.

Parameters:

encoder (nn.Module) – The freshly built EyeEncoder.
path (str | Path) – A Lightning .ckpt written by a frame-wise run.

Raises:

ModelError – If the file is missing, holds no encoder weights, or its tensors do not fit this encoder.

class blinklinmult.train.model.EyeEncoder(config: blinklinmult.train.config.ModelConfig, image_size: int)¶

Bases: torch.nn.Module

Embeds a window of eye crops into a per-timestep feature sequence.

Parameters:

config (ModelConfig) – Architecture settings.
image_size (int) – Side length of a square eye crop.

Raises:

ModelError – If the backbone cannot be built, or the crop is too small for it.

train(mode: bool = True) → EyeEncoder¶

Keep a frozen encoder in eval mode.

Lightning calls train() on the whole module at the start of every epoch, which would put a frozen encoder’s BatchNorm back into training mode. Its running statistics would then keep updating even though no gradient flows, so the “frozen” arm would drift away from the checkpoint it was supposed to hold fixed – and the comparison against the fine-tuned arm would measure something other than what it claims.

Parameters:: mode (bool) – Requested mode, ignored while frozen.
Returns:: self, as torch.nn.Module.train() does.
Return type:: EyeEncoder

forward(eye_images: torch.Tensor) → torch.Tensor¶

Embed one batch of windows.

Parameters:: eye_images (torch.Tensor) – (B, T, C, H, W) as loaded.
Returns:: (B, T, backbone_output_dim).
Return type:: torch.Tensor

blinklinmult.train.model.check_targets(target_names: list[str]) → None¶

Reject targets the single head cannot serve.

Parameters:: target_names (list[str]) – Targets this run supervises.
Raises:: ModelError – If a target is not one this model predicts.

class blinklinmult.train.model.CnnHead(input_dim: int, config: blinklinmult.train.config.ModelConfig, target_names: list[str])¶

Bases: torch.nn.Module

Per-frame classification head for the cnn family.

No sequence model: each timestep is classified from its own embedding, which is the DenseNet121 eye-state baseline of the paper. The output is still a sequence so that every family shares one training step.

Parameters:

input_dim (int) – Encoder output width.
config (ModelConfig) – Architecture settings.
target_names (list[str]) – Targets to predict.

forward(sequence: torch.Tensor) → dict[str, torch.Tensor]¶

Classify every timestep.

Parameters:: sequence (torch.Tensor) – (B, T, D) per-timestep embeddings.
Returns:: The same (B, T, 1) logits under every supervised target key – one decision, read two ways.
Return type:: dict[str, torch.Tensor]

class blinklinmult.train.model.EventHead(kind: str, hidden_dim: int, dropout: float = 0.1, num_heads: int = 4, detach: bool = True)¶

Bases: torch.nn.Module

Predicts blink intervals from the ESR closure signal.

Takes the (B, T, 1) per-timestep closure score and emits (B, T, 1) interval logits. See EVENT_HEAD for why this is stacked on the ESR head rather than sharing it or sitting beside it.

The input is detached by default. Without that, the event loss reaches the ESR head through this module and the parameter competition returns by a longer path – the thing stacking was meant to remove. Detached, the ESR head is supervised purely by closure and this module learns to interpret whatever signal it is handed.

Attaching is left configurable rather than forbidden: a little event gradient may help the ESR head sharpen blink boundaries, which is exactly what RN15 under-specifies at 15 fps with a median 1.22 closed frames per blink. That is worth measuring rather than assuming either way.

Parameters:

kind (str) – One of EVENT_HEAD_TYPES.
hidden_dim (int) – Width of the internal representation.
dropout (float) – Dropout applied before the output projection.
num_heads (int) – Attention heads, for the attention variant.
detach (bool) – Cut the gradient to the ESR head.

Raises:

ModelError – If kind is not a known variant.

forward(esr_logits: torch.Tensor, mask: torch.Tensor | None = None) → torch.Tensor¶

Map a closure sequence to interval logits.

Parameters:

esr_logits (torch.Tensor) – (B, T, 1) from the ESR head.
mask (torch.Tensor | None) – (B, T), True where the timestep is valid. Used by the attention variant to ignore padding.

Returns:

(B, T, 1) interval logits.

Return type:

torch.Tensor

class blinklinmult.train.model.BlinkModel(config: blinklinmult.train.config.ModelConfig, target_names: list[str], image_size: int, eye_feature_dim: int | None = None)¶

Bases: torch.nn.Module

The full model: eye encoder, optional sequence model, and task heads.

Parameters:

config (ModelConfig) – Architecture settings.
target_names (list[str]) – Targets this run supervises, in head order.
image_size (int) – Side length of a square eye crop.
eye_feature_dim (int | None) – Width of the handcrafted eye features, or None when the run has none.

Raises:

ModelError – If the family and the available modalities disagree.

property uses_eye_features: bool¶

Whether this model consumes the handcrafted feature stream.

Returns:: True only for the linmult family.
Return type:: bool

forward(eye_images: torch.Tensor, eye_image_mask: torch.Tensor, eye_features: torch.Tensor | None = None, eye_feature_mask: torch.Tensor | None = None, eye_embedding: torch.Tensor | None = None) → dict[str, torch.Tensor]¶

Predict every supervised target over a batch of windows.

Parameters:

eye_images (torch.Tensor) – (B, T, C, H, W) eye crops.
eye_image_mask (torch.Tensor) – (B, T), True = valid timestep.
eye_features (torch.Tensor | None) – (B, T, F) handcrafted features; required by the linmult family.
eye_feature_mask (torch.Tensor | None) – (B, T) validity mask.
eye_embedding (torch.Tensor | None) – (B, T, D) precomputed eye embeddings, replacing the encoder pass. Supplied only when a frozen encoder makes them constant across epochs; see blinklinmult.data.embeddings. None encodes live.

Returns:

Raw logits per target, (B, T, D).

Return type:

dict[str, torch.Tensor]

Raises:

ModelError – If the cross-modal family is called without its second modality.

blinklinmult.train.model.build_model(config: blinklinmult.train.config.ModelConfig, target_names: list[str], image_size: int, eye_feature_dim: int | None = None) → BlinkModel¶

Construct the model for the configured family.

Parameters:

config (ModelConfig) – Architecture settings.
target_names (list[str]) – Targets this run supervises.
image_size (int) – Side length of a square eye crop.
eye_feature_dim (int | None) – Handcrafted feature width, if any.

Returns:

The model.

Return type:

BlinkModel