annie.dataset.scanning¶

Dataset scanning: aggregate each video with its annotations into the manifest.

The scanner turns a SourceRegistry into one row per video (a VideoEntry) carrying that video’s raw detections, all of its tracks, its resolved protagonist track, and any label values from CSV label sources. Browse consumes the sorted manifest; the Dataset tab shows the overview counts; the Browse filter bar is built from the discovered label columns.

Because there is no Scan button any more, this runs whenever the set of sources changes (add/remove on the Dataset tab). It stays linear in the number of files:

A video is matched by its stem (filename minus extension).
A .vdet matches by exact stem; a __track{N}.csv matches by the part before __track. Both fall back to a longest-stem-first prefix scan only on a miss.
A label/protagonist table row matches a video by exact key-column value — or, for a label table, by a nested-layout path field (match_field) and optionally by substring (match_substring, see match_keys()).

Classes¶

`ScanResult`	The outcome of a dataset scan.
`JoinSummary`	How well a table's keys link to the videos (shown while configuring a table).

Functions¶

`resolve_video_stem`(→ str \| None)	Find the video stem an annotation belongs to, longest-stem-first.
`match_keys`(→ tuple[dict[str, str], set[str]])	Link table keys to match targets (video ids or path-field values).
`join_summary`(→ JoinSummary)	Summarise a prospective join for the table dialog.
`build_manifest`(→ ScanResult)	Scan all sources in `registry` into a per-video manifest.
`scan_dataset`(→ ScanResult)	Scan the classic four MOSEI paths into a per-video manifest (convenience).

Module Contents¶

class annie.dataset.scanning.ScanResult[source]¶

The outcome of a dataset scan.

entries¶: The sorted, per-video manifest the Browse tab renders.

num_vdet_files¶: Total vdet files discovered.

num_track_files¶: Total track files discovered.

protagonist_available¶: Whether protagonist assignments were found.

label_columns¶: The label-column names available for tags/filters.

layout_collisions¶: Video ids that more than one file produced under a nested layout (only the first file of each was kept).

layout_error¶: Why a configured nested layout could not be used, else None.

nested_videos_hint¶: The flat video folder holds no videos itself but has some in sub-folders, so a nested layout is probably needed. Found during the scan so the Dataset tab never walks the folder while it redraws.

property by_video_id: dict[str, annie.core.models.VideoEntry]¶

The manifest indexed by video_id, built once on first access.

Callers that resolve a video by id inside a re-rendered UI (the Segment-review task does it on every keystroke) would otherwise walk the whole manifest per lookup. Where two entries share a video_id — the same video with different annotation suffixes — the first in manifest order wins, matching the behaviour of the linear next(...) search this replaced.

Returns:: A mapping of video id to its first VideoEntry.

property counts: dict[str, int | bool]¶

Overview metrics for the Dataset cards.

Returns:: Counts keyed by num_videos, num_vdet_files, num_track_files, videos_vdet_and_track, videos_with_vdet, videos_with_track, and protagonist_available (bool).

label_values(column: str) → list[str][source]¶

Return the sorted distinct values a label column takes across entries.

Parameters:: column – The label column name.
Returns:: Sorted distinct non-empty values for that column.

annie.dataset.scanning.resolve_video_stem(annotation_stem: str, video_stems_longest_first: list[str]) → str | None[source]¶

Find the video stem an annotation belongs to, longest-stem-first.

A video stem s matches when the annotation stem equals s exactly or begins with s followed by an underscore separator. Checking longest-first ensures X2 wins over X for X2__track0.

Parameters:

annotation_stem – The annotation filename minus extension.
video_stems_longest_first – Known video stems, pre-sorted longest-first.

Returns:

The matching video stem, or None if no video matches.

annie.dataset.scanning.match_keys(keys: collections.abc.Iterable[str], targets: collections.abc.Iterable[str], *, substring: bool) → tuple[dict[str, str], set[str]][source]¶

Link table keys to match targets (video ids or path-field values).

Exact matching (the default) is a set lookup. With substring a target links to a key it contains: keys are grouped by length and each target is slid over once per distinct key length, so the cost stays near-linear rather than keys × targets. When a target contains several keys the longest wins; an equal-length tie goes to the key that comes first in keys (the table’s winning-row order). Such targets are reported as ambiguous.

Parameters:

keys – The table’s keys, in priority order.
targets – The strings to link.
substring – Match keys contained in a target instead of equal to it.

Returns:

(target -> key, ambiguous targets); unmatched targets are absent.

class annie.dataset.scanning.JoinSummary[source]¶

How well a table’s keys link to the videos (shown while configuring a table).

videos_matched¶: Videos that get a row.

videos_total¶: All videos considered.

keys_used¶: Distinct keys that link to at least one video.

keys_total¶: Distinct non-empty keys in the table.

ambiguous¶: Matched videos whose target contained more than one key.

annie.dataset.scanning.join_summary(keys: collections.abc.Iterable[str], video_targets: dict[str, str], *, substring: bool) → JoinSummary[source]¶

Summarise a prospective join for the table dialog.

Parameters:

keys – The table’s keys in priority order (see match_keys()).
video_targets – Video id to the string its key must match (the id itself or a path-field value); an empty string means the video has no target.
substring – Whether keys match by substring.

Returns:

A JoinSummary.

annie.dataset.scanning.build_manifest(registry: annie.dataset.sources.SourceRegistry) → ScanResult[source]¶

Scan all sources in registry into a per-video manifest.

The video source is the spine: with none configured the manifest is empty. Every other source attaches to a video by stem (vdet/track) or key column (label/protagonist CSV).

Parameters:: registry – The configured data sources.
Returns:: A ScanResult with the per-video manifest, overview counts, and the discovered label columns. Missing/unset sources contribute nothing rather than raising.

Scan the classic four MOSEI paths into a per-video manifest (convenience).

A thin wrapper that assembles a SourceRegistry from the given folders and protagonist file (with the default uuid/track_id columns) and delegates to build_manifest().

Parameters:

videos_dir – Folder of source videos.
vdet_dir – Folder of .vdet files.
track_dir – Folder of __track{N}.csv files.
participants_file – Protagonist heuristic CSV (uuid,track_id).

Returns:

A ScanResult with the per-video manifest and overview counts.