annie.dataset.sources

Generic, extensible data-source registry (domain).

Annie is dataset-agnostic: a dataset is described as an ordered list of data sources the user adds on the Dataset tab, rather than four fixed folders. Each source is one of:

• SourceKind.VIDEO — a folder of source videos (the mandatory spine; one Browse row per video). Singleton.

• SourceKind.VDET — a folder of raw-detection .vdet files. Singleton.

• SourceKind.TRACK — a folder of __track{N}.csv files. Singleton.

• SourceKind.CSV — a label/metadata CSV joined to videos by a chosen key column. A CSV plays one of two roles (CsvRole): labels (its selected value columns become Browse tags/filters; many allowed) or protagonist (one value column is the active-track id; singleton).

This module is pure domain — no NiceGUI, no config, no torch. The registry is the single source of truth the scanner (annie.scanning) consumes.

Attributes

FOLDER_KINDS	Folder-backed kinds. Each is a singleton in a registry.
VIDEO_SUFFIXES	Accepted source-video suffixes (lower-case, dotted).
VDET_SUFFIXES	Suffix identifying a raw-detection file.
TRACK_GLOB	Glob identifying derived single-track files.
KIND_LABELS	Short human label per kind, for the Dataset source list and the add menu.
KIND_ICONS	Tabler icon per kind for the Dataset cards.
TASK_LABELS	Human label per task, for the Dataset task-wise sections and the Annotator switch.

Classes

SourceKind	The kind of a data source.
CsvRole	How a CSV source is interpreted.
SegmentationBand	A named start/end column pair defining one segmentation of a clip.
DataSource	One configured data source.
SourceRegistry	An ordered set of data sources with singleton rules.
TaskKind	A supervision task the Annotator can offer, driven by the sources present.
TaskRequirement	One prerequisite for a task, and whether the registry satisfies it.
TaskReadiness	Whether a task's sources are all present, with the per-requirement detail.

Functions

task_readiness(→ list[TaskReadiness])

Report, per task, which required sources are present in registry.

Module Contents

class annie.dataset.sources.SourceKind

Bases: enum.StrEnum

The kind of a data source.

Initialize self. See help(type(self)) for accurate signature.

class annie.dataset.sources.CsvRole

Bases: enum.StrEnum

How a CSV source is interpreted.

Initialize self. See help(type(self)) for accurate signature.

class annie.dataset.sources.SegmentationBand

A named start/end column pair defining one segmentation of a clip.

A segmentation CSV may carry several competing segmentations of the same clip (e.g. a ground-truth span and a WhisperX forced-alignment span); each is one band, rendered side by side in the Segment-review task for comparison.

name

Human label shown beside the band’s preview (e.g. "GT", "cut").

start_column

CSV column holding the band’s start time, in seconds.

end_column

CSV column holding the band’s end time, in seconds.

annie.dataset.sources.FOLDER_KINDS: tuple[SourceKind, ...]

Folder-backed kinds. Each is a singleton in a registry.

annie.dataset.sources.VIDEO_SUFFIXES: tuple[str, ...] = ('.mp4',)

Accepted source-video suffixes (lower-case, dotted).

annie.dataset.sources.VDET_SUFFIXES: tuple[str, ...] = ('.vdet',)

Suffix identifying a raw-detection file.

annie.dataset.sources.TRACK_GLOB = '*__track*.csv'

Glob identifying derived single-track files.

annie.dataset.sources.KIND_LABELS: dict[SourceKind, str]

Short human label per kind, for the Dataset source list and the add menu.

annie.dataset.sources.KIND_ICONS: dict[SourceKind, str]

Tabler icon per kind for the Dataset cards.

class annie.dataset.sources.DataSource

One configured data source.

kind

The SourceKind.

path

Folder (for folder kinds) or file (for CSV) the source points at.

role

For a CSV source, its CsvRole; ignored otherwise.

key_column

For a CSV source, the column joined to the video id.

value_columns

For a CSV source, the selected value columns (label tags, or a single protagonist track-id column).

column_types

For a CSV source, the chosen data type per value column ("str" / "int" / "float"); columns absent here default to str.

segment_column

For a CsvRole.SEGMENTATION CSV, the column that tells apart the several rows sharing one key_column value: the two combine into the clip identity {video_id}_{segment_id} that each accept/drop is saved under. None for other roles.

bands

For a CsvRole.SEGMENTATION CSV, the ordered start/end column pairs to render and compare (see SegmentationBand). Empty otherwise.

layout

For a SourceKind.VIDEO folder, an optional nested VideoLayout. None means a flat folder (one video per file, id = file stem).

sheet

For a table that is an Excel workbook, the worksheet it reads. None for a CSV.

match_field

For a label table, the nested-layout path field (e.g. "subject") its key column matches; None matches the whole video id.

match_substring

For a label table, link a row to every video whose match target contains the key instead of equalling it.

column_names

For a label table, the short tag name shown for a value column; columns absent here keep their header as the name.

order_column

For a label table whose key repeats, the column whose order decides which row counts (e.g. a timestamp). None when keys are unique.

order_descending

Direction for order_column; True keeps the largest / newest row.

property is_folder: bool

Whether this source points at a folder (vs a single CSV file).

property is_protagonist: bool

Whether this is the CSV that assigns each video’s protagonist track.

property is_segmentation: bool

Whether this CSV holds per-clip segments for the Segment-review task.

property available: bool

Whether the source path currently exists (folder for folders, file for CSV).

property label: str

A short label for the Dataset source list.

tag_name(column: str) → str

The tag / filter name shown for one of this table’s value columns.

count() → int

Count the items the source contributes (cheap; no matching).

Returns:

The number of videos / vdet / track files, or CSV data rows. 0 if the path is missing.

class annie.dataset.sources.SourceRegistry

An ordered set of data sources with singleton rules.

Folder kinds (video/vdet/track) and the protagonist CSV are singletons: adding one replaces any existing source of that role. Label CSVs are unlimited, keyed by path (re-adding the same file replaces it).

add(source: DataSource) → None

Add source, enforcing the singleton rules in place of duplicates.

Parameters:

source – The source to add.

remove(source: DataSource) → None

Remove source if present (identity by path + kind).

get(kind: SourceKind) → DataSource | None

Return the first source of kind, or None.

property video: DataSource | None

The video-folder source, or None.

property vdet: DataSource | None

The vdet-folder source, or None.

property track: DataSource | None

The track-folder source, or None.

property protagonist: DataSource | None

The protagonist CSV source, or None.

property label_sources: list[DataSource]

All label CSV sources (excludes the protagonist and segmentation CSVs).

property segmentation_sources: list[DataSource]

All segmentation CSV sources (drive the Segment-review task).

property has_video: bool

Whether a usable (existing) video folder is configured.

available_kinds_to_add() → list[SourceKind]

Kinds offered by the + menu (singletons drop out once present).

class annie.dataset.sources.TaskKind

Bases: enum.StrEnum

A supervision task the Annotator can offer, driven by the sources present.

Initialize self. See help(type(self)) for accurate signature.

annie.dataset.sources.TASK_LABELS: dict[TaskKind, str]

Human label per task, for the Dataset task-wise sections and the Annotator switch.

class annie.dataset.sources.TaskRequirement

One prerequisite for a task, and whether the registry satisfies it.

label

Human name of the required source (e.g. "videos").

present

Whether an available source satisfying it is configured.

class annie.dataset.sources.TaskReadiness

Whether a task’s sources are all present, with the per-requirement detail.

task

The TaskKind.

requirements

Each prerequisite and its present/absent state, in order.

property ready: bool

Whether every requirement is satisfied (the Annotator may offer the task).

annie.dataset.sources.task_readiness(registry: SourceRegistry) → list[TaskReadiness]

Report, per task, which required sources are present in registry.

This is the single source of truth behind both the Dataset tab’s task-wise readiness sections and the Annotator’s decision of which tasks to offer.

Parameters:

registry – The configured sources.

Returns:

One TaskReadiness per TaskKind, requirements in order.