personalitylinmult.data.schema¶

Typed contract shared by the HDF5 builder and the training dataloader.

This module is deliberately dependency-light (no torch, no h5py) so that the same FeatureSpec objects can describe a feature on the way into the HDF5 file and on the way out of it. One schema, no drift.

Layout of the built file (see personalitylinmult.data.builder):

/                        attrs: schema_version, created_utc, git_sha,
                                source_label_csv_sha256, builder_config_yaml
/video_id                (N,)     variable-length utf-8, row order == index order
/subset                  (N,)     uint8, see SUBSET_CODES
/labels/<trait>          (N,)     float32
/features/<name>/data    (N,T,D)  float16, chunks=(1,T,D)
/features/<name>/mask    (N,T)    bool,    chunks=(1,T)
/features/<name>/length  (N,)     int32, the clip's length before padding
    attrs on /features/<name>: dim, max_time_dim, source_dir, source_format,
                               dtype_orig, layers, rate_hz

Padding is post-padding. A clip’s real frames occupy [0, length) and the zeros fill [length, max_time_dim) – never the other way round. So data[i, :length[i]] is the clip, mask[i] is True exactly on [0, length[i]), and a longer clip is truncated from the tail. Anything reading this file may rely on that.

Masks are written for every sequence feature, including ones that are dense for every sample today. The network contract is feature + mask per feature, and a uniform contract survives windowing (where density is a property of the source clip, not of the cropped batch) at a storage cost of roughly 0.05%.

Attributes¶

`SCHEMA_VERSION`	Bumped whenever the on-disk HDF5 layout changes incompatibly.
`BIG_FIVE_TRAITS`	Targets of the big-five regression task.
`INTERVIEW_TRAITS`	Target of the interview-variable regression task.
`ALL_TRAITS`	Every label column stored in the HDF5.
`SUBSETS`	Dataset splits, in code order.
`SUBSET_CODES`	Maps a split name to the `uint8` code stored in `/subset`.
`SOURCE_FORMATS`	Recognised on-disk feature formats.
`CROP_MODES`	Analysis-window crop strategies.

Exceptions¶

SchemaError

Raised when a feature or dataset configuration is self-inconsistent.

Classes¶

`FeatureSpec`	Describes one feature: where it comes from and how it is stored.
`DatasetSpec`	The full description of one built HDF5 dataset.
`BuildStats`	Counters describing one builder run, for logging and tests.

Module Contents¶

personalitylinmult.data.schema.SCHEMA_VERSION = 1¶: Bumped whenever the on-disk HDF5 layout changes incompatibly.

personalitylinmult.data.schema.BIG_FIVE_TRAITS: tuple[str, Ellipsis] = ('openness', 'conscientiousness', 'extraversion', 'agreeableness', 'emotional_stability')¶

Targets of the big-five regression task.

emotional_stability (= 1 - neuroticism) is used rather than neuroticism so that all five traits point the same way: higher is more of the named quality. TrainConfig rejects neuroticism as a big-five target.

personalitylinmult.data.schema.INTERVIEW_TRAITS: tuple[str, Ellipsis] = ('interview',)¶

Target of the interview-variable regression task.

A distinct task from big-five regression: the interview variable is a hireability judgement, not a personality trait, and is trained separately.

personalitylinmult.data.schema.ALL_TRAITS: tuple[str, Ellipsis] = ('openness', 'conscientiousness', 'extraversion', 'agreeableness', 'neuroticism',...¶

Every label column stored in the HDF5.

All seven are stored regardless of task, because storing a scalar per clip is free and it means a task change never requires a rebuild. neuroticism is retained as the raw annotation for traceability even though training uses emotional_stability.

personalitylinmult.data.schema.SUBSETS: tuple[str, Ellipsis] = ('train', 'valid', 'test')¶: Dataset splits, in code order.

personalitylinmult.data.schema.SUBSET_CODES: dict[str, int]¶: Maps a split name to the uint8 code stored in /subset.

personalitylinmult.data.schema.SOURCE_FORMATS: frozenset[str]¶

Recognised on-disk feature formats.

pickle_layers — a pickled list of per-layer (T, D) arrays (WavLM).
npy — a single (T, D) array (emotion2vec+).
safetensors — features/frame_ids and optionally mask (DINOv2, EmotiEffNet, FaRL).
hdf5_per_id — one HDF5 holding a (T, D) sequence dataset per video_id (the token-level text features).

personalitylinmult.data.schema.CROP_MODES: frozenset[str]¶

Analysis-window crop strategies.

head takes [0, window); center takes the middle window; random takes a uniformly random offset and is only applied to the training split (center is substituted elsewhere) so evaluation stays deterministic.

exception personalitylinmult.data.schema.SchemaError[source]¶

Bases: ValueError

Raised when a feature or dataset configuration is self-inconsistent.

Initialize self. See help(type(self)) for accurate signature.

class personalitylinmult.data.schema.FeatureSpec[source]¶

Describes one feature: where it comes from and how it is stored.

The same object drives extraction (builder) and consumption (dataset), so a feature can never be written under one interpretation and read under another.

Parameters:

name (str) – Feature name. Becomes the HDF5 group name and the key the model sees in names=, so it must be unique within a dataset.
source_dir (str) – Directory of per-clip files, or the containing directory for hdf5_per_id. Relative to the repository root.
source_format (str) – One of SOURCE_FORMATS.
dim (int) – Feature dimensionality D.
rate_hz (float | None) – Nominal sampling rate, used to convert window_seconds into a frame count. None means the time axis is not wall-clock (e.g. text tokens) and the feature is never cropped by the analysis window.
max_time_dim (int | None) – Stored time dimension. Clips longer than this are truncated and shorter ones are zero-padded (with the mask marking the padding invalid), so the stored length is uniform. None means “measure it from the data”, which is only appropriate when the length is not known in advance.
layers (tuple[int, ...]) – For pickle_layers only: which transformer layers to keep. One layer stores (N, T, D); several store (N, L, T, D) for linmult’s learned weighted_sum.
source_file (str | None) – For hdf5_per_id: the HDF5 filename inside source_dir.
dtype (str) – On-disk dtype. float16 halves size and I/O; measured across the real features the largest magnitude is ~31 against fp16’s 65504 ceiling, so the range is safe.

Raises:

SchemaError – If the spec is internally inconsistent.

__post_init__() → None[source]¶

Validate the spec’s internal consistency.

Raises:: SchemaError – If any field combination is invalid.

property is_multi_layer: bool¶

Whether this feature stores several transformer layers per sample.

Returns:: True when the stored array carries a layer axis, which linmult consumes as a 4-D (B, L, T, F) weighted-sum input.
Return type:: bool

window_frames(window_seconds: float | None) → int | None[source]¶

Frames spanned by an analysis window for this feature.

Parameters:: window_seconds (float | None) – Window length in seconds, or None for no windowing.
Returns:: Frame count, or None when this feature is not windowed (no rate_hz, or no window requested).
Return type:: int | None

classmethod from_dict(d: dict[str, Any]) → FeatureSpec[source]¶

Build a spec from a plain dict, e.g. parsed YAML.

Parameters:: d (dict) – Feature configuration values.
Returns:: The validated spec.
Return type:: FeatureSpec
Raises:: SchemaError – If required keys are missing or the spec is invalid.

class personalitylinmult.data.schema.DatasetSpec[source]¶

The full description of one built HDF5 dataset.

Parameters:

features (list[FeatureSpec]) – Features to include. Only these are read from disk and written to the file — the selection is the contract, not a filter applied later.
h5_path (str) – Destination path of the built file.
label_csv (str) – Source label.csv produced by the preprocess stage.

Raises:

SchemaError – If no features are given or names collide.

__post_init__() → None[source]¶

Validate the dataset spec.

Raises:: SchemaError – If the feature list is empty or contains duplicates.

property feature_names: list[str]¶

Feature names in configured order.

Returns:: Names, in the order the model receives its inputs.
Return type:: list[str]

feature(name: str) → FeatureSpec[source]¶

Look up a feature by name.

Parameters:: name (str) – Feature name.
Returns:: The matching spec.
Return type:: FeatureSpec
Raises:: KeyError – If no feature has that name.

class personalitylinmult.data.schema.BuildStats[source]¶

Counters describing one builder run, for logging and tests.

Parameters:

n_samples (int) – Rows written.
n_features (int) – Features written.
bytes_written (int) – Size of the resulting file.
per_feature_time_dim (dict[str, int]) – Stored T per feature.