annie.dataset.storage ===================== .. py:module:: annie.dataset.storage .. autoapi-nested-parse:: SQLite-backed review-status store (Browse tab curation state). This holds *curation* state per video — a good/bad verdict, an optional note, an "add to annotator" flag, and the protagonist-track correction — plus, on export, the protagonist ``_manual`` CSV. The store is a real file (by default a per-session ``~/.annie/sessions/annie_.db``; pin one with ``ANNIE_DB_PATH``) so it survives restarts, and is keyed by :attr:`annie.models.VideoEntry.key`. Every video is **liked (good) by default**: a row only exists once the user interacts, and the UI treats "no row" as good. The store is exportable to CSV/JSON in one call and importable via upsert, so a reviewer's curation travels with them. Attributes ---------- .. autoapisummary:: annie.dataset.storage.Verdict annie.dataset.storage.Decision Classes ------- .. autoapisummary:: annie.dataset.storage.ReviewRecord annie.dataset.storage.EventRecord annie.dataset.storage.EventCategory annie.dataset.storage.EventType annie.dataset.storage.ReviewStore Module Contents --------------- .. py:data:: Verdict A review verdict. ``None`` (no row) is treated as ``"good"`` by the UI. .. py:data:: Decision A Segment-review decision on a clip. ``None`` means the clip is not yet reviewed. .. py:class:: ReviewRecord One persisted review row. .. attribute:: row_key Stable per-row identity (:attr:`annie.models.VideoEntry.key`). .. attribute:: video_id The video this review belongs to. .. attribute:: annotation_suffix The annotation suffix, or ``None`` for an exact match. .. attribute:: verdict ``"good"``, ``"bad"``, or ``None`` if untouched (treated as good). .. attribute:: note Free-text reviewer note (empty string when none). .. attribute:: annotate Whether the video is queued for the Annotator tab. .. attribute:: active_track The session's protagonist-track override, or ``None`` if the reviewer has not corrected this video (the heuristic value then stands). .. attribute:: decision The Segment-review accept/drop for a clip row, or ``None`` if the clip is not (or not a) Segment-review sample. .. attribute:: updated_at ISO-8601 UTC timestamp of the last change. .. py:class:: EventRecord One persisted Event-annotation event: a labelled interval on a track. .. attribute:: event_id Stable UUID identity, so an event survives edits to its own fields. .. attribute:: video_id The video this event belongs to. .. attribute:: row_key The :attr:`annie.core.models.VideoEntry.key` of the queue row, tying the event to the same identity curation uses. .. attribute:: track The track (category) name the event sits on. .. attribute:: start_frame Inclusive start frame of the interval. .. attribute:: end_frame Inclusive end frame of the interval (``>= start_frame``). .. attribute:: label Free-text label shown on the event box (empty when unnamed). .. attribute:: note Free-text note (empty when none). .. attribute:: attributes Arbitrary string key/value pairs; stored as a JSON object. .. attribute:: color Per-event colour override (hex string), or ``None`` to inherit the event category's colour, else the track's, on the timeline. .. attribute:: event_type The event category the event was made with (and stays linked to), or ``None`` for a plain event. .. attribute:: updated_at ISO-8601 UTC timestamp of the last change. .. py:class:: EventCategory One participant category (a timeline lane) in the Event-annotation task. DB-scoped, not per-video: the categories in a review DB are shared by every video of that dataset. An event belongs to a category when its ``track`` equals the category ``name``. .. attribute:: name The category / participant label (e.g. ``"Mother"``); unique within the DB. .. attribute:: ordinal Vertical lane order, ``0`` at the top. .. attribute:: color Optional hex colour override for the lane; ``None`` uses the palette by ordinal. .. attribute:: updated_at ISO-8601 UTC timestamp of the last change. .. py:class:: EventType One event category in the Event-annotation task (e.g. ``"Smile"``). DB-scoped like :class:`EventCategory`. A new event made with it gets its name as the label, and stays linked: renaming the type renames its events; its colour and attributes apply wherever the event sets none of its own (see :mod:`annie.dataset.events`). .. attribute:: name The event category's name; unique within the DB. .. attribute:: ordinal Order in the Category block (``0`` first; Shift+1 selects it). .. attribute:: color Hex colour of its events, or ``None`` for the participant lane's colour. .. attribute:: attributes Default key/value attributes of its events. .. attribute:: updated_at ISO-8601 UTC timestamp of the last change. .. py:class:: ReviewStore(db_path: str | pathlib.Path) A thin, connection-per-call wrapper over the ``review`` table. The store opens a fresh connection for each operation, which keeps it safe to call from the render worker threads and the UI thread alike without sharing a connection across threads. Open (creating if needed) the review database at ``db_path``. :param db_path: Filesystem path to the SQLite file. Parent directories are created as needed. .. py:method:: has_data() -> bool Return whether any review, event, participant or event category has been stored. Used to decide whether switching to another database would leave progress behind (see :meth:`copy_to`). .. py:method:: copy_to(db_path: str | pathlib.Path) -> ReviewStore Copy this whole database to ``db_path`` and return a store opened on the copy. Uses SQLite's online backup API, so the copy is consistent even while this store is in use. An existing file at ``db_path`` is overwritten; this database is left untouched. :param db_path: Destination SQLite file. Parent directories are created as needed. :returns: A :class:`ReviewStore` backed by the copy. .. py:method:: get(row_key: str) -> ReviewRecord | None Return the review record for ``row_key``, or ``None`` if unreviewed. :param row_key: The row identity to look up. :returns: The stored :class:`ReviewRecord`, or ``None``. .. py:method:: all() -> list[ReviewRecord] Return every review record, ordered by row key. :returns: All stored review records. .. py:method:: records_by_key() -> dict[str, ReviewRecord] Return every stored review record keyed by row key, in one query. A filter pass over the whole manifest needs each video's review state; reading them all at once replaces one connection per video with a single one. :returns: ``row_key -> ReviewRecord`` for every touched video (unreviewed ones are absent). .. py:method:: list_by_verdict(verdict: Verdict) -> list[ReviewRecord] Return all records with the given verdict (the good or bad list). :param verdict: ``"good"`` or ``"bad"``. :returns: Matching review records ordered by row key. .. py:method:: upsert(row_key: str, video_id: str, annotation_suffix: str | None, *, verdict: Verdict | None = None, note: str | None = None, annotate: bool | None = None, active_track: int | None = None, decision: Decision | None = None) -> ReviewRecord Insert or update a review row, preserving fields left as ``None``. Passing ``verdict=None`` / ``note=None`` / ``annotate=None`` / ``active_track=None`` / ``decision=None`` keeps any existing value, so the good/bad toggle, the note, the annotator flag, the protagonist correction, and the Segment-review accept/drop update independently. :param row_key: Stable row identity (primary key). :param video_id: The video id, stored for export/grouping. :param annotation_suffix: The annotation suffix, or ``None``. :param verdict: New verdict, or ``None`` to leave unchanged. :param note: New note, or ``None`` to leave unchanged. :param annotate: New annotator flag, or ``None`` to leave unchanged. :param active_track: New protagonist-track override, or ``None`` to leave unchanged. :param decision: New Segment-review decision, or ``None`` to leave unchanged. :returns: The resulting :class:`ReviewRecord`. .. py:method:: set_verdict(row_key: str, video_id: str, annotation_suffix: str | None, verdict: Verdict | None) -> ReviewRecord Set (or clear) the good/bad verdict for a row. :param row_key: Stable row identity. :param video_id: The video id. :param annotation_suffix: The annotation suffix, or ``None``. :param verdict: ``"good"``, ``"bad"``, or ``None`` to clear. :returns: The updated :class:`ReviewRecord`. .. py:method:: set_annotate(row_key: str, video_id: str, annotation_suffix: str | None, value: bool) -> ReviewRecord Set the "add to annotator" flag for a video. :param row_key: Stable row identity. :param video_id: The video id. :param annotation_suffix: The annotation suffix, or ``None``. :param value: Whether the video is queued for the Annotator tab. :returns: The updated :class:`ReviewRecord`. .. py:method:: set_annotate_many(videos: collections.abc.Iterable[tuple[str, str]], value: bool) -> int Set the "add to annotator" flag for many videos in one transaction. Backs Browse's "Add all to Annotator" action: calling :meth:`set_annotate` per video would open a connection and commit for each one, which is slow on a large filtered selection. Only the ``annotate`` column is written, so any stored verdict, note, or protagonist correction survives; videos with no row yet get one with the defaults (liked, no note). :param videos: ``(row_key, video_id)`` pairs to update. :param value: Whether the videos are queued for the Annotator tab. :returns: The number of videos written. .. py:method:: set_active_track(row_key: str, video_id: str, annotation_suffix: str | None, track_id: int) -> ReviewRecord Persist the reviewer's protagonist-track correction for a video. The choice lives only in this session's database; the ``_manual`` CSV is written separately by the Annotator's "Export corrected CSV" action. :param row_key: Stable row identity. :param video_id: The video id. :param annotation_suffix: The annotation suffix, or ``None``. :param track_id: The chosen active track index. :returns: The updated :class:`ReviewRecord`. .. py:method:: active_tracks() -> dict[str, int] Return every stored protagonist override as ``row_key -> track_id``. :returns: A mapping of row key to corrected track index for the rows a reviewer has changed this session (rows without an override are omitted). .. py:method:: annotator_keys() -> set[str] Return the row keys flagged for the Annotator tab. :returns: The set of ``row_key`` values whose ``annotate`` flag is set. .. py:method:: set_decision(row_key: str, video_id: str, decision: Decision) -> ReviewRecord Persist a Segment-review accept/drop decision for a clip. The clip is keyed by its composite ``{video_id}_{segment_id}`` ``row_key``, so re-opening the source resumes a half-finished pass from the database. :param row_key: The clip identity (``{video_id}_{segment_id}``). :param video_id: The parent video id, stored for export/grouping. :param decision: ``"accept"`` or ``"drop"``. :returns: The updated :class:`ReviewRecord`. .. py:method:: clear_decision(row_key: str) -> None Return a clip to the *undecided* state, dropping any accept/drop. This is the "Undecided" escape hatch in Segment review: a misclick or a changed mind must be able to put a clip back into the undecided pool that the progress bar counts and "jump to next undecided" walks. It cannot go through :meth:`upsert`, where ``decision=None`` means "leave unchanged" — this writes the ``NULL`` explicitly. A clip that was never decided is left untouched. :param row_key: The clip identity (``{video_id}_{segment_id}``). .. py:method:: decisions() -> dict[str, Decision] Return every stored Segment-review decision as ``row_key -> decision``. :returns: A mapping of clip key to ``"accept"``/``"drop"`` for the clips a reviewer has decided (undecided clips are omitted). .. py:method:: list_by_decision(decision: Decision) -> list[ReviewRecord] Return all records with the given Segment-review decision. :param decision: ``"accept"`` or ``"drop"``. :returns: Matching review records ordered by row key (the accepted or dropped set). .. py:method:: add_event(video_id: str, row_key: str, track: str, start_frame: int, end_frame: int, *, label: str = '', note: str = '', attributes: dict[str, str] | None = None, color: str | None = None, event_type: str | None = None) -> EventRecord Insert a new event on a track and return it (with its fresh id). The event's track is also registered in ``event_track`` if absent, so a lane created implicitly by dropping an event onto it persists like an explicit one. :param video_id: The video the event belongs to. :param row_key: The queue row identity the event ties to. :param track: The track (category) name. :param start_frame: Inclusive start frame. :param end_frame: Inclusive end frame (``>= start_frame``). :param label: Optional event label. :param note: Optional event note. :param attributes: Optional arbitrary string key/value pairs. :param color: Optional per-event colour override (hex), or ``None`` to inherit the track. :param event_type: The event category it is made with, or ``None`` for a plain event. :returns: The stored :class:`EventRecord`. .. py:method:: update_event(event_id: str, **fields: object) -> EventRecord | None Update named columns of an event, returning the updated record. Only the columns named in ``fields`` are written (plus ``updated_at``); any others keep their stored value. ``attributes`` is accepted as a dict and stored as JSON. Passing an unknown column raises, so a typo fails loudly rather than silently doing nothing. :param event_id: The event to update. :param \*\*fields: Column/value pairs among ``track``, ``start_frame``, ``end_frame``, ``label``, ``note``, ``attributes``, ``color``, ``event_type``. :returns: The updated :class:`EventRecord`, or ``None`` if no such event exists. .. py:method:: delete_event(event_id: str) -> None Delete one event by id (a no-op if it does not exist). :param event_id: The event to delete. .. py:method:: get_event(event_id: str) -> EventRecord | None Return one event by id, or ``None`` if it does not exist. :param event_id: The event to fetch. :returns: The stored :class:`EventRecord`, or ``None``. .. py:method:: events_for(row_key: str) -> list[EventRecord] Return a video's events, ordered by track then start frame. :param row_key: The queue row identity to fetch events for. :returns: The events on that video, grouped by track and sorted within each track. .. py:method:: all_events() -> list[EventRecord] Return every stored event, ordered by video then track then start frame. Backs the session-wide export (every annotated video in one file). :returns: All events in the store. .. py:method:: add_track(row_key: str, track: str) -> None Create an empty track (lane) for a video, appended after existing ones. Idempotent: re-adding an existing track leaves its ordinal untouched. :param row_key: The queue row identity. :param track: The track (category) name. .. py:method:: rename_track(row_key: str, old: str, new: str) -> None Rename a track and move its events onto the new name, keeping the ordinal. A no-op when ``old`` and ``new`` are equal. If ``new`` already exists, the two lanes merge: ``old``'s events move onto ``new`` and the now-empty ``old`` lane is removed. :param row_key: The queue row identity. :param old: The current track name. :param new: The new track name. .. py:method:: delete_track(row_key: str, track: str) -> None Delete a track and all of its events for a video. :param row_key: The queue row identity. :param track: The track (category) name to remove. .. py:method:: tracks_for(row_key: str) -> list[str] Return a video's track names in their vertical order. :param row_key: The queue row identity. :returns: The track names ordered by ``ordinal`` (their top-to-bottom lane order). .. py:method:: event_row_keys() -> set[str] Return the row keys that have at least one event or track. Lets the tab know which queued videos already carry annotation work. :returns: The set of row keys present in either event table. .. py:method:: categories() -> list[EventCategory] Return the dataset's participant categories, in lane order. :returns: The categories ordered by ``ordinal`` then ``name`` (top to bottom). .. py:method:: add_category(name: str, color: str | None = None) -> EventCategory Add a category at the end of the lane order (idempotent on the name). Re-adding an existing name is a no-op that returns the existing category, so a double-click never creates a duplicate. :param name: The category / participant label (must be non-empty; caller validates). :param color: Optional hex colour override for the lane. :returns: The stored :class:`EventCategory` (existing one if the name was already present). .. py:method:: rename_category(old: str, new: str) -> None Rename a category and move every event on it onto the new name. A no-op when ``old == new``. If ``new`` already exists, the two lanes merge: ``old``'s events move onto ``new`` and the now-empty ``old`` category is removed. This is the safe way to fix a label everywhere at once — events follow, so nothing is orphaned. :param old: The current category name. :param new: The new name. .. py:method:: delete_category(name: str) -> None Delete a category and all events on it, then close the ordinal gap. The caller is responsible for confirming when the category still holds events (the UI guards this); at the store level the delete always cascades to keep the data consistent — an event can never reference a category that no longer exists. :param name: The category to remove. .. py:method:: category_event_count(name: str) -> int Return how many events currently sit on a category (for the delete guard). :param name: The category name. :returns: The number of events whose ``track`` equals ``name``. .. py:method:: reorder_categories(names: list[str]) -> None Set the lane order to ``names`` (first = top). Unlisted categories keep going after. :param names: The category names in the desired top-to-bottom order. Names not present in the DB are ignored; categories omitted from the list are appended after the listed ones in their previous relative order. .. py:method:: set_category_color(name: str, color: str | None) -> None Set (or clear) a category's lane colour override. :param name: The category name. :param color: A hex colour, or ``None`` to fall back to the palette by ordinal. .. py:method:: seed_categories_from_events() -> int Seed the category table from distinct event track names, if it is empty. Back-compat for a DB annotated under the old single-lane model: its events sit on the ``"events"`` track (or whatever names were used), but there are no category rows yet. This creates one category per distinct existing track name so those events keep a lane the user can then rename to a participant. Does nothing once any category exists. :returns: The number of categories created (``0`` if the table was already populated or there are no events). .. py:method:: event_types() -> list[EventType] Return the dataset's event categories, in their order (Shift+1 is the first). :returns: The event categories ordered by ``ordinal`` then ``name``. .. py:method:: add_event_type(name: str, color: str | None = None, attributes: dict[str, str] | None = None) -> EventType Add an event category at the end of the order (idempotent on the name). :param name: The event category's name (must be non-empty; caller validates). :param color: Optional hex colour of its events. :param attributes: Optional default key/value attributes of its events. :returns: The stored :class:`EventType` (the existing one if the name was already present). .. py:method:: rename_event_type(old: str, new: str) -> None Rename an event category; its events follow. Events made with it point at the new name, and those whose label is still the old name are relabelled (a label the user changed is kept). If ``new`` already exists, the two merge: ``old``'s events move onto ``new`` and ``old`` is removed. :param old: The current name. :param new: The new name. .. py:method:: delete_event_type(name: str) -> None Delete an event category without changing how its events look. Each of its events first takes over what it inherited: the category's colour (unless the event has its own) and its default attributes (the event's own values win). Then the link is cleared, so the events stay as they were, just no longer linked. :param name: The event category to remove. .. py:method:: event_type_event_count(name: str) -> int Return how many events are linked to an event category (for the delete warning). :param name: The event category's name. :returns: The number of events whose ``event_type`` equals ``name``. .. py:method:: reorder_event_types(names: list[str]) -> None Set the event categories' order to ``names`` (unlisted ones keep going after). :param names: Event category names in the desired order; unknown names are ignored. .. py:method:: set_event_type_color(name: str, color: str | None) -> None Set (or clear) an event category's colour; its events follow unless they have their own. :param name: The event category's name. :param color: A hex colour, or ``None`` for the participant lane's colour. .. py:method:: set_event_type_attributes(name: str, attributes: dict[str, str]) -> None Replace an event category's default attributes. :param name: The event category's name. :param attributes: The new default key/value attributes. .. py:method:: set_note(row_key: str, video_id: str, annotation_suffix: str | None, note: str) -> ReviewRecord Set the free-text note for a row. :param row_key: Stable row identity. :param video_id: The video id. :param annotation_suffix: The annotation suffix, or ``None``. :param note: The note text. :returns: The updated :class:`ReviewRecord`. .. py:method:: export_json_text() -> str Render all review records as a JSON array string. .. py:method:: export_csv_text() -> str Render all review records as CSV text. .. py:method:: export_json(path: str | pathlib.Path) -> pathlib.Path Write all review records to a JSON array file. :param path: Destination JSON path. :returns: The path written. .. py:method:: export_csv(path: str | pathlib.Path) -> pathlib.Path Write all review records to a CSV file. :param path: Destination CSV path. :returns: The path written. .. py:method:: import_records(records: collections.abc.Iterable[dict[str, object]]) -> int Upsert review records from dicts (e.g. a re-imported export). :param records: Iterable of dicts with at least ``row_key`` and ``video_id``. :returns: The number of records imported.