annie.dataset.sources ===================== .. py:module:: annie.dataset.sources .. autoapi-nested-parse:: Generic, extensible data-source registry (domain). Annie is dataset-agnostic: a dataset is described as an ordered list of **data sources** the user adds on the Dataset tab, rather than four fixed folders. Each source is one of: * :attr:`SourceKind.VIDEO` — a folder of source videos (the mandatory spine; one Browse row per video). Singleton. * :attr:`SourceKind.VDET` — a folder of raw-detection ``.vdet`` files. Singleton. * :attr:`SourceKind.TRACK` — a folder of ``__track{N}.csv`` files. Singleton. * :attr:`SourceKind.CSV` — a label/metadata CSV joined to videos by a chosen **key column**. A CSV plays one of two roles (:class:`CsvRole`): ``labels`` (its selected value columns become Browse tags/filters; many allowed) or ``protagonist`` (one value column is the active-track id; singleton). This module is pure domain — no NiceGUI, no config, no torch. The registry is the single source of truth the scanner (:mod:`annie.scanning`) consumes. Attributes ---------- .. autoapisummary:: annie.dataset.sources.FOLDER_KINDS annie.dataset.sources.VIDEO_SUFFIXES annie.dataset.sources.VDET_SUFFIXES annie.dataset.sources.TRACK_GLOB annie.dataset.sources.KIND_LABELS annie.dataset.sources.KIND_ICONS annie.dataset.sources.TASK_LABELS Classes ------- .. autoapisummary:: annie.dataset.sources.SourceKind annie.dataset.sources.CsvRole annie.dataset.sources.SegmentationBand annie.dataset.sources.DataSource annie.dataset.sources.SourceRegistry annie.dataset.sources.TaskKind annie.dataset.sources.TaskRequirement annie.dataset.sources.TaskReadiness Functions --------- .. autoapisummary:: annie.dataset.sources.task_readiness Module Contents --------------- .. py:class:: SourceKind Bases: :py:obj:`enum.StrEnum` The kind of a data source. Initialize self. See help(type(self)) for accurate signature. .. py:class:: CsvRole Bases: :py:obj:`enum.StrEnum` How a CSV source is interpreted. Initialize self. See help(type(self)) for accurate signature. .. py:class:: SegmentationBand A named start/end column pair defining one segmentation of a clip. A segmentation CSV may carry several competing segmentations of the same clip (e.g. a ground-truth span and a WhisperX forced-alignment span); each is one band, rendered side by side in the Segment-review task for comparison. .. attribute:: name Human label shown beside the band's preview (e.g. ``"GT"``, ``"cut"``). .. attribute:: start_column CSV column holding the band's start time, in seconds. .. attribute:: end_column CSV column holding the band's end time, in seconds. .. py:data:: FOLDER_KINDS :type: tuple[SourceKind, ...] Folder-backed kinds. Each is a **singleton** in a registry. .. py:data:: VIDEO_SUFFIXES :type: tuple[str, ...] :value: ('.mp4',) Accepted source-video suffixes (lower-case, dotted). .. py:data:: VDET_SUFFIXES :type: tuple[str, ...] :value: ('.vdet',) Suffix identifying a raw-detection file. .. py:data:: TRACK_GLOB :value: '*__track*.csv' Glob identifying derived single-track files. .. py:data:: KIND_LABELS :type: dict[SourceKind, str] Short human label per kind, for the Dataset source list and the add menu. .. py:data:: KIND_ICONS :type: dict[SourceKind, str] Tabler icon per kind for the Dataset cards. .. py:class:: DataSource One configured data source. .. attribute:: kind The :class:`SourceKind`. .. attribute:: path Folder (for folder kinds) or file (for CSV) the source points at. .. attribute:: role For a CSV source, its :class:`CsvRole`; ignored otherwise. .. attribute:: key_column For a CSV source, the column joined to the video id. .. attribute:: value_columns For a CSV source, the selected value columns (label tags, or a single protagonist track-id column). .. attribute:: column_types For a CSV source, the chosen data type per value column (``"str"`` / ``"int"`` / ``"float"``); columns absent here default to str. .. attribute:: segment_column For a :attr:`CsvRole.SEGMENTATION` CSV, the column that tells apart the several rows sharing one ``key_column`` value: the two combine into the clip identity ``{video_id}_{segment_id}`` that each accept/drop is saved under. ``None`` for other roles. .. attribute:: bands For a :attr:`CsvRole.SEGMENTATION` CSV, the ordered start/end column pairs to render and compare (see :class:`SegmentationBand`). Empty otherwise. .. attribute:: layout For a :attr:`SourceKind.VIDEO` folder, an optional nested :class:`~annie.dataset.layout.VideoLayout`. ``None`` means a flat folder (one video per file, id = file stem). .. attribute:: sheet For a table that is an Excel workbook, the worksheet it reads. ``None`` for a CSV. .. attribute:: match_field For a label table, the nested-layout path field (e.g. ``"subject"``) its key column matches; ``None`` matches the whole video id. .. attribute:: match_substring For a label table, link a row to every video whose match target *contains* the key instead of equalling it. .. attribute:: column_names For a label table, the short tag name shown for a value column; columns absent here keep their header as the name. .. attribute:: order_column For a label table whose key repeats, the column whose order decides which row counts (e.g. a timestamp). ``None`` when keys are unique. .. attribute:: order_descending Direction for ``order_column``; ``True`` keeps the largest / newest row. .. py:property:: is_folder :type: bool Whether this source points at a folder (vs a single CSV file). .. py:property:: is_protagonist :type: bool Whether this is the CSV that assigns each video's protagonist track. .. py:property:: is_segmentation :type: bool Whether this CSV holds per-clip segments for the Segment-review task. .. py:property:: available :type: bool Whether the source path currently exists (folder for folders, file for CSV). .. py:property:: label :type: str A short label for the Dataset source list. .. py:method:: tag_name(column: str) -> str The tag / filter name shown for one of this table's value columns. .. py:method:: count() -> int Count the items the source contributes (cheap; no matching). :returns: The number of videos / vdet / track files, or CSV data rows. ``0`` if the path is missing. .. py:class:: SourceRegistry An ordered set of data sources with singleton rules. Folder kinds (video/vdet/track) and the protagonist CSV are singletons: adding one replaces any existing source of that role. Label CSVs are unlimited, keyed by path (re-adding the same file replaces it). .. py:method:: add(source: DataSource) -> None Add ``source``, enforcing the singleton rules in place of duplicates. :param source: The source to add. .. py:method:: remove(source: DataSource) -> None Remove ``source`` if present (identity by path + kind). .. py:method:: get(kind: SourceKind) -> DataSource | None Return the first source of ``kind``, or ``None``. .. py:property:: video :type: DataSource | None The video-folder source, or ``None``. .. py:property:: vdet :type: DataSource | None The vdet-folder source, or ``None``. .. py:property:: track :type: DataSource | None The track-folder source, or ``None``. .. py:property:: protagonist :type: DataSource | None The protagonist CSV source, or ``None``. .. py:property:: label_sources :type: list[DataSource] All label CSV sources (excludes the protagonist and segmentation CSVs). .. py:property:: segmentation_sources :type: list[DataSource] All segmentation CSV sources (drive the Segment-review task). .. py:property:: has_video :type: bool Whether a usable (existing) video folder is configured. .. py:method:: available_kinds_to_add() -> list[SourceKind] Kinds offered by the ``+`` menu (singletons drop out once present). .. py:class:: TaskKind Bases: :py:obj:`enum.StrEnum` A supervision task the Annotator can offer, driven by the sources present. Initialize self. See help(type(self)) for accurate signature. .. py:data:: TASK_LABELS :type: dict[TaskKind, str] Human label per task, for the Dataset task-wise sections and the Annotator switch. .. py:class:: TaskRequirement One prerequisite for a task, and whether the registry satisfies it. .. attribute:: label Human name of the required source (e.g. ``"videos"``). .. attribute:: present Whether an available source satisfying it is configured. .. py:class:: TaskReadiness Whether a task's sources are all present, with the per-requirement detail. .. attribute:: task The :class:`TaskKind`. .. attribute:: requirements Each prerequisite and its present/absent state, in order. .. py:property:: ready :type: bool Whether every requirement is satisfied (the Annotator may offer the task). .. py:function:: task_readiness(registry: SourceRegistry) -> list[TaskReadiness] Report, per task, which required sources are present in ``registry``. This is the single source of truth behind both the Dataset tab's task-wise readiness sections and the Annotator's decision of which tasks to offer. :param registry: The configured sources. :returns: One :class:`TaskReadiness` per :class:`TaskKind`, requirements in order.