blinklinmult.data.shards

Per-clip shards, so a long corpus build can be resumed rather than restarted.

MPEblink decodes 1 842 untrimmed videos and takes the better part of a day. A single-file build has no notion of progress: H5Writer opens its temporary file with mode "w", so an interrupted run leaves a partial file that is indistinguishable from a complete one except by sample count, and the next run starts from zero.

Sharding fixes that by making the clip the unit of durability. Each clip writes its own small HDF5 file; a clip whose shard already exists is skipped, and the shards are merged into the corpus file once every clip has one. A shard is only renamed into place after it closes cleanly, so a shard that exists is a clip that finished – there is no partial-shard ambiguity to reason about.

The cost is one extra pass over the data at merge time, which is I/O-bound and minutes rather than hours.

Attributes

logger	Module-level logger.
SHARD_DIRNAME	Directory, beside the corpus file, holding one HDF5 per completed clip.

Functions

shard_dir(→ pathlib.Path)	Where a corpus's shards live.
shard_path(→ pathlib.Path)	The shard for one clip.
completed_clips(→ set[str])	Which clips already have a shard.
merge_shards(→ int)	Combine every shard into the corpus file.
shard_attrs(→ dict)	Root attributes from any one shard, for the merged file.
pending(→ list)	The entries whose clips have no shard yet.

Module Contents

blinklinmult.data.shards.logger

Module-level logger.

blinklinmult.data.shards.SHARD_DIRNAME = 'shards'

Directory, beside the corpus file, holding one HDF5 per completed clip.

blinklinmult.data.shards.shard_dir(h5_path: pathlib.Path) → pathlib.Path

Where a corpus’s shards live.

Parameters:

h5_path (Path) – The corpus’s final .h5.

Returns:

The shard directory, which may not exist yet.

Return type:

Path

blinklinmult.data.shards.shard_path(h5_path: pathlib.Path, clip_id: str) → pathlib.Path

The shard for one clip.

Parameters:

• h5_path (Path) – The corpus’s final .h5.

• clip_id (str) – Identifier unique within the corpus, e.g. test_1.

Returns:

The shard’s path.

Return type:

Path

blinklinmult.data.shards.completed_clips(h5_path: pathlib.Path) → set[str]

Which clips already have a shard.

A shard is renamed into place only after it closes cleanly, so its presence means that clip finished. A run therefore resumes by skipping these rather than by trusting a progress counter that a crash could have left stale.

Parameters:

h5_path (Path) – The corpus’s final .h5.

Returns:

Clip identifiers already built.

Return type:

set[str]

blinklinmult.data.shards.merge_shards(h5_path: pathlib.Path, attrs: dict, remove: bool = True) → int

Combine every shard into the corpus file.

Samples are copied group by group, which preserves each dataset’s compression and dtype exactly – re-encoding here would silently change what a shard already got right.

Parameters:

• h5_path (Path) – The corpus’s final .h5.

• attrs (dict) – Root attributes for the merged file, as the shards’ writer would have written them.

• remove (bool) – Delete the shard directory once merged. False keeps it, which is what a debugging run wants.

Returns:

Samples written.

Return type:

int

Raises:

FileNotFoundError – If there are no shards to merge.

blinklinmult.data.shards.shard_attrs(h5_path: pathlib.Path) → dict

Root attributes from any one shard, for the merged file.

Every shard is written by the same writer from the same declaration, so they agree; reading the first avoids reconstructing them at the merge site and lets them drift.

Parameters:

h5_path (Path) – The corpus’s final .h5.

Returns:

The attributes.

Return type:

dict

Raises:

FileNotFoundError – If there are no shards.

blinklinmult.data.shards.pending(entries: collections.abc.Iterator, h5_path: pathlib.Path, clip_id) → list

The entries whose clips have no shard yet.

Parameters:

• entries (Iterator) – Everything the build would process.

• h5_path (Path) – The corpus’s final .h5.

• clip_id – Callable mapping an entry to its clip identifier.

Returns:

The entries still to do, in the original order.

Return type:

list