blinklinmult.data.omni

Bridge from a DatasetSpec to OmniLoader.

OmniLoader unifies several datasets that annotate different things into one masked, batchable stream. To do that it needs, per dataset, a omniloader.DatasetSchema declaring which keys that dataset provides and what shape they have. This module generates those declarations from the same DatasetSpec objects the builder wrote the files with, so the schema a corpus is read under can never drift from the one it was written under.

Shape mapping. OmniLoader describes values structurally: feature_dim declares a flat trailing axis, shape declares a structured one. One eye’s crops are (T, 3, H, W), so they are declared with shape=(3, H, W) and travel in native image form the whole way — stored, padded, masked, mixed and batched without a reshape.

That needs omniloader >= 1.1, which added structured trailing shapes. Before it, images had to be flattened into a wide feature_dim and reshaped back at the model boundary; the schema then described a 12288-wide feature rather than an image, and nothing validated that the width factorised correctly.

Cross-dataset agreement. OmniLoader requires that specs sharing a name agree on shape and dtype across datasets. The corpora differ in frame rate and therefore in native window length, so unified_specs() resolves one shared time_dim for the whole run — the longest any participating corpus needs — and every shorter corpus is padded to it with an all-False mask. That is how a 15 fps and a 30 fps corpus train together while still covering the same real duration.

Attributes

logger

Module-level logger.

Functions

eye_image_shape(→ tuple[int, int, int])

Trailing shape of one frame's eye crop.

dataset_schema(→ omniloader.DatasetSchema)

Build the OmniLoader schema for one corpus.

unified_specs(→ tuple[int, int, int | None])

Resolve the shared shape a joint run brings every corpus to.

build_schemas(→ tuple[list[omniloader.DatasetSchema], ...)

Resolve the run's shared shape and build every corpus's schema.

Module Contents

blinklinmult.data.omni.logger

Module-level logger.

blinklinmult.data.omni.eye_image_shape(image_size: int) → tuple[int, int, int]

Trailing shape of one frame’s eye crop.

Parameters:

image_size (int) – Side length of a square eye crop.

Returns:

(C, H, W).

Return type:

tuple[int, int, int]

blinklinmult.data.omni.dataset_schema(spec: blinklinmult.data.schema.DatasetSpec, time_dim: int, image_size: int, feature_dim: int | None, embedding_dim: int | None = None) → omniloader.DatasetSchema

Build the OmniLoader schema for one corpus.

Parameters:

  • spec (DatasetSpec) – The corpus.

  • time_dim (int) – Shared window length for the run, in frames. Every dataset in a joint run declares the same T; OmniLoader pads each corpus’s native window to it and masks the difference.

  • image_size (int) – Shared eye-crop size for the run.

  • feature_dim (int | None) – Shared handcrafted-descriptor width, or None when no corpus in the run supplies them.

  • embedding_dim (int | None) – Width of a precomputed eye embedding, when the run reads one from a cache, else None. Declaring it is what lets the field survive collation; see the comment at its TensorSpec.

Returns:

Features and targets this corpus provides. Keys it lacks are simply absent, which is how OmniLoader knows to mask them.

Return type:

DatasetSchema

Raises:

SchemaError – If the corpus declares a feature width that disagrees with the run’s shared width.

blinklinmult.data.omni.unified_specs(specs: list[blinklinmult.data.schema.DatasetSpec], time_dim: int | None = None, image_size: int | None = None, stills: bool = False) → tuple[int, int, int | None]

Resolve the shared shape a joint run brings every corpus to.

Parameters:

  • specs (list[DatasetSpec]) – Corpora participating in the run, already carrying the run’s window length (see with_window()).

  • time_dim (int | None) – Window length in frames to use. None takes the maximum any corpus derives from its own rate, so no corpus’s window is truncated by default.

  • image_size (int | None) – Eye-crop size. None requires every corpus to agree, since resizing a crop after extraction would resample an already-resampled image.

  • stills (bool) – Whether the run serves single frames. Suppresses the truncation warning: in still mode the frames are chosen before OmniLoader sees them (see blinklinmult.data.stills), so nothing is cropped and warning about it would be false.

Returns:

(time_dim, image_size, feature_dim).

Return type:

tuple[int, int, int | None]

Raises:

SchemaError – If the list is empty, crop sizes disagree with no explicit choice, or the corpora supply descriptors of differing widths.

blinklinmult.data.omni.build_schemas(specs: list[blinklinmult.data.schema.DatasetSpec], time_dim: int | None = None, image_size: int | None = None, stills: bool = False, embedding_dim: int | None = None) → tuple[list[omniloader.DatasetSchema], int, int, int | None]

Resolve the run’s shared shape and build every corpus’s schema.

Parameters:

  • specs (list[DatasetSpec]) – Corpora participating in the run.

  • time_dim (int | None) – Shared window length in frames, or None to infer from the corpora’s rates.

  • image_size (int | None) – Shared crop size, or None to infer.

  • stills (bool) – Whether the run serves single frames.

  • embedding_dim (int | None) – Width of a cached eye embedding, or None when the run encodes live.

Returns:

(schemas, time_dim, image_size, feature_dim), with schemas in the same order as specs.

Return type:

tuple