personalitylinmult.data.builder

Build the single HDF5 training artifact from extracted FI features.

The builder reads a data config, and writes only the features that config names. Nothing else is opened, read, or written: if the config lists WavLM and DINOv2, the resulting file contains WavLM and DINOv2 and no trace of the other extractors. The config that produced a file is embedded verbatim in its root attributes, so a built file is always traceable back to its recipe.

The resulting file is the reproducibility boundary for training: it is published to a public Hugging Face dataset repo so a run reproduces without re-extraction, and the training loop reads it and nothing else.

Example

uv run python -m personalitylinmult.data.builder --config config/FI/data/wedrx.yaml

Attributes

logger	Module-level logger.
DEFAULT_ROOT	Repository root; all config paths are interpreted relative to it.
COMPRESSION	Compression filter. Paired with the shuffle filter for float data.
COMPRESSION_LEVEL	a deliberate middle ground between size and read CPU.

Exceptions

BuildError

Raised when the dataset cannot be built as configured.

Functions

git_sha(→ str)	Return the current commit SHA, for traceability of a built file.
file_sha256(→ str)	Hash a file's contents.
load_labels(→ tuple[list[str], list[str], dict[str, ...)	Read label.csv into ids, subsets, and per-trait arrays.
resolve_coverage(→ None)	Assert a feature exists for every clip, or fail with the missing ids.
probe_feature(→ tuple[int, int])	Determine a feature's stored time dimension and verify its width.
build(→ personalitylinmult.data.schema.BuildStats)	Build the HDF5 dataset described by spec.
dataset_spec_from_config(...)	Load a data config into a DatasetSpec.
main(→ None)	CLI entry point.

Module Contents

personalitylinmult.data.builder.logger

Module-level logger.

personalitylinmult.data.builder.DEFAULT_ROOT

Repository root; all config paths are interpreted relative to it.

personalitylinmult.data.builder.COMPRESSION = 'gzip'

Compression filter. Paired with the shuffle filter for float data.

personalitylinmult.data.builder.COMPRESSION_LEVEL = 4

a deliberate middle ground between size and read CPU.

Type:

Compression level

exception personalitylinmult.data.builder.BuildError

Bases: RuntimeError

Raised when the dataset cannot be built as configured.

Initialize self. See help(type(self)) for accurate signature.

personalitylinmult.data.builder.git_sha(root: pathlib.Path) → str

Return the current commit SHA, for traceability of a built file.

A -dirty suffix is appended when the working tree has uncommitted changes, so an artifact built from an uncommitted state is never mistaken for one that a clean checkout of that SHA would reproduce.

Parameters:

root (Path) – Repository root.

Returns:

The commit SHA (with a -dirty suffix when the tree is dirty), or "unknown" outside a git checkout.

Return type:

str

personalitylinmult.data.builder.file_sha256(path: pathlib.Path) → str

Hash a file’s contents.

Parameters:

path (Path) – File to hash.

Returns:

Hex digest.

Return type:

str

personalitylinmult.data.builder.load_labels(label_csv: pathlib.Path) → tuple[list[str], list[str], dict[str, numpy.ndarray]]

Read label.csv into ids, subsets, and per-trait arrays.

Parameters:

label_csv (Path) – Path to the preprocess stage’s label.csv.

Returns:

(video_ids, subsets, labels) where labels maps each trait in ALL_TRAITS to an (N,) float32 array.

Return type:

tuple

Raises:

BuildError – If the file is missing a required column or a subset name is unrecognised.

personalitylinmult.data.builder.resolve_coverage(spec: personalitylinmult.data.schema.FeatureSpec, root: pathlib.Path, video_ids: list[str]) → None

Assert a feature exists for every clip, or fail with the missing ids.

A partially-extracted feature must never be zero-filled into the dataset: that would train and evaluate on silence while looking perfectly healthy. EmotiEffNet and FaRL are only ~7% extracted at the time of writing, and this check is what keeps them out until they are finished.

Parameters:

• spec (FeatureSpec) – Feature to check.

• root (Path) – Repository root.

• video_ids (list[str]) – Clips the dataset requires.

Raises:

BuildError – If any clip’s feature is missing.

personalitylinmult.data.builder.probe_feature(spec: personalitylinmult.data.schema.FeatureSpec, root: pathlib.Path, video_ids: list[str], time_dim_hint: int | None = None) → tuple[int, int]

Determine a feature’s stored time dimension and verify its width.

Reading every clip here would double the build’s I/O — and for WavLM, whose pickles hold all 12 layers at ~0.34 s per clip, that is nearly an hour of pure waste. Instead:

• max_time_dim is taken from time_dim_hint when the caller can derive it (any feature with a rate_hz has a known frame count), or probed from a sample otherwise.

• Every clip’s width and length is still checked exactly, during the write pass, where the data is being read anyway. A clip that disagrees fails the build; nothing is silently truncated.

Parameters:

• spec (FeatureSpec) – Feature to probe.

• root (Path) – Repository root.

• video_ids (list[str]) – Clips to sample.

• time_dim_hint (int | None) – Known stored time dimension, if any.

Returns:

(max_time_dim, dim).

Return type:

tuple[int, int]

Raises:

BuildError – If a sampled clip’s width contradicts the configured dim.

personalitylinmult.data.builder.build(spec: personalitylinmult.data.schema.DatasetSpec, root: pathlib.Path, config_yaml: str, limit: int | None = None, overwrite: bool = False) → personalitylinmult.data.schema.BuildStats

Build the HDF5 dataset described by spec.

Parameters:

• spec (DatasetSpec) – What to build. Only its features are read.

• root (Path) – Repository root for resolving relative paths.

• config_yaml (str) – The config source, embedded for traceability.

• limit (int | None) – Build only the first N clips, for tests and CI.

• overwrite (bool) – Rebuild even if an identical file already exists.

Returns:

Counters describing the run.

Return type:

BuildStats

Raises:

BuildError – If labels or features are missing or inconsistent.

personalitylinmult.data.builder.dataset_spec_from_config(config_path: pathlib.Path) → tuple[personalitylinmult.data.schema.DatasetSpec, str]

Load a data config into a DatasetSpec.

The config names its features by file stem; each is loaded from <config_dir>/../features/<name>.yaml. Only the named features are read. Paths on the returned spec (h5_path, label_csv) stay relative – resolving them against a repository root is the caller’s job (see build()), not this function’s.

Parameters:

config_path (Path) – Path to the data config YAML.

Returns:

The spec and the verbatim config text.

Return type:

tuple[DatasetSpec, str]

Raises:

BuildError – If the config or a referenced feature file is missing or malformed.

personalitylinmult.data.builder.main() → None

CLI entry point.