blinklinmult.data

Training-data layer: the per-corpus HDF5 artifacts and their loaders.

This package owns everything between the corpus-specific scripts in blinklinmult.preprocess and the training loop in blinklinmult.train:

  • schema — the typed per-corpus contract.

  • builder — the config-driven HDF5 writer.

  • omni — the bridge to OmniLoader’s schema.

  • collate — the batch-to-model boundary.

  • datamodule — the Lightning DataModule that mixes every corpus into one stream.

Each corpus’s HDF5 file is a reproducibility boundary: training reads those files and nothing else. Each is published to a public Hugging Face dataset repo (make push-<name>) so a run reproduces without re-running extraction, and each is rebuilt from source by make preprocess-<name>.

Submodules