annie.dataset.scanning ====================== .. py:module:: annie.dataset.scanning .. autoapi-nested-parse:: Dataset scanning: aggregate each video with its annotations into the manifest. The scanner turns a :class:`~annie.sources.SourceRegistry` into **one row per video** (a :class:`~annie.models.VideoEntry`) carrying that video's raw detections, all of its tracks, its resolved protagonist track, and any label values from CSV label sources. Browse consumes the sorted manifest; the Dataset tab shows the overview counts; the Browse filter bar is built from the discovered label columns. Because there is no Scan button any more, this runs whenever the set of sources changes (add/remove on the Dataset tab). It stays linear in the number of files: * A video is matched by its **stem** (filename minus extension). * A ``.vdet`` matches by exact stem; a ``__track{N}.csv`` matches by the part before ``__track``. Both fall back to a longest-stem-first prefix scan only on a miss. * A label/protagonist table row matches a video by exact key-column value — or, for a label table, by a nested-layout path field (``match_field``) and optionally by substring (``match_substring``, see :func:`match_keys`). Classes ------- .. autoapisummary:: annie.dataset.scanning.ScanResult annie.dataset.scanning.JoinSummary Functions --------- .. autoapisummary:: annie.dataset.scanning.resolve_video_stem annie.dataset.scanning.match_keys annie.dataset.scanning.join_summary annie.dataset.scanning.build_manifest annie.dataset.scanning.scan_dataset Module Contents --------------- .. py:class:: ScanResult The outcome of a dataset scan. .. attribute:: entries The sorted, per-video manifest the Browse tab renders. .. attribute:: num_vdet_files Total vdet files discovered. .. attribute:: num_track_files Total track files discovered. .. attribute:: protagonist_available Whether protagonist assignments were found. .. attribute:: label_columns The label-column names available for tags/filters. .. attribute:: layout_collisions Video ids that more than one file produced under a nested layout (only the first file of each was kept). .. attribute:: layout_error Why a configured nested layout could not be used, else ``None``. .. attribute:: nested_videos_hint The flat video folder holds no videos itself but has some in sub-folders, so a nested layout is probably needed. Found during the scan so the Dataset tab never walks the folder while it redraws. .. py:property:: by_video_id :type: dict[str, annie.core.models.VideoEntry] The manifest indexed by ``video_id``, built once on first access. Callers that resolve a video by id inside a re-rendered UI (the Segment-review task does it on every keystroke) would otherwise walk the whole manifest per lookup. Where two entries share a ``video_id`` — the same video with different annotation suffixes — the first in manifest order wins, matching the behaviour of the linear ``next(...)`` search this replaced. :returns: A mapping of video id to its first :class:`~annie.core.models.VideoEntry`. .. py:property:: counts :type: dict[str, int | bool] Overview metrics for the Dataset cards. :returns: Counts keyed by ``num_videos``, ``num_vdet_files``, ``num_track_files``, ``videos_vdet_and_track``, ``videos_with_vdet``, ``videos_with_track``, and ``protagonist_available`` (bool). .. py:method:: label_values(column: str) -> list[str] Return the sorted distinct values a label column takes across entries. :param column: The label column name. :returns: Sorted distinct non-empty values for that column. .. py:function:: resolve_video_stem(annotation_stem: str, video_stems_longest_first: list[str]) -> str | None Find the video stem an annotation belongs to, longest-stem-first. A video stem ``s`` matches when the annotation stem equals ``s`` exactly or begins with ``s`` followed by an underscore separator. Checking longest-first ensures ``X2`` wins over ``X`` for ``X2__track0``. :param annotation_stem: The annotation filename minus extension. :param video_stems_longest_first: Known video stems, pre-sorted longest-first. :returns: The matching video stem, or ``None`` if no video matches. .. py:function:: match_keys(keys: collections.abc.Iterable[str], targets: collections.abc.Iterable[str], *, substring: bool) -> tuple[dict[str, str], set[str]] Link table keys to match targets (video ids or path-field values). Exact matching (the default) is a set lookup. With ``substring`` a target links to a key it *contains*: keys are grouped by length and each target is slid over once per distinct key length, so the cost stays near-linear rather than keys × targets. When a target contains several keys the **longest** wins; an equal-length tie goes to the key that comes first in ``keys`` (the table's winning-row order). Such targets are reported as ambiguous. :param keys: The table's keys, in priority order. :param targets: The strings to link. :param substring: Match keys contained in a target instead of equal to it. :returns: ``(target -> key, ambiguous targets)``; unmatched targets are absent. .. py:class:: JoinSummary How well a table's keys link to the videos (shown while configuring a table). .. attribute:: videos_matched Videos that get a row. .. attribute:: videos_total All videos considered. .. attribute:: keys_used Distinct keys that link to at least one video. .. attribute:: keys_total Distinct non-empty keys in the table. .. attribute:: ambiguous Matched videos whose target contained more than one key. .. py:function:: join_summary(keys: collections.abc.Iterable[str], video_targets: dict[str, str], *, substring: bool) -> JoinSummary Summarise a prospective join for the table dialog. :param keys: The table's keys in priority order (see :func:`match_keys`). :param video_targets: Video id to the string its key must match (the id itself or a path-field value); an empty string means the video has no target. :param substring: Whether keys match by substring. :returns: A :class:`JoinSummary`. .. py:function:: build_manifest(registry: annie.dataset.sources.SourceRegistry) -> ScanResult Scan all sources in ``registry`` into a per-video manifest. The video source is the spine: with none configured the manifest is empty. Every other source attaches to a video by stem (vdet/track) or key column (label/protagonist CSV). :param registry: The configured data sources. :returns: A :class:`ScanResult` with the per-video manifest, overview counts, and the discovered label columns. Missing/unset sources contribute nothing rather than raising. .. py:function:: scan_dataset(videos_dir: str | pathlib.Path | None = None, vdet_dir: str | pathlib.Path | None = None, track_dir: str | pathlib.Path | None = None, participants_file: str | pathlib.Path | None = None) -> ScanResult Scan the classic four MOSEI paths into a per-video manifest (convenience). A thin wrapper that assembles a :class:`~annie.sources.SourceRegistry` from the given folders and protagonist file (with the default ``uuid``/``track_id`` columns) and delegates to :func:`build_manifest`. :param videos_dir: Folder of source videos. :param vdet_dir: Folder of ``.vdet`` files. :param track_dir: Folder of ``__track{N}.csv`` files. :param participants_file: Protagonist heuristic CSV (``uuid,track_id``). :returns: A :class:`ScanResult` with the per-video manifest and overview counts.