annie.dataset.datasets¶

Dataset config files: save, load, and discover a SourceRegistry (service).

A config is a small JSON file describing a dataset as a list of data sources, so a set of folders/CSVs can be loaded in one click instead of re-picking them every restart. Paths may be relative (resolved against the config file’s directory, so configs that reference the bundled examples travel with the repo) or absolute.

When running from a source checkout, example configs under data/config/ are auto-discovered and listed in the Dataset tab dropdown. When installed from PyPI the data/ directory is absent and discovery returns an empty list silently; users load configs via the file picker or save their own.

Attributes¶

LAST_CONFIG_FILENAME

File under ANNIE_HOME remembering the config most recently loaded or saved.

Functions¶

`repo_data_dir`(→ pathlib.Path)	Return the bundled `data` directory.
`bundled_config_dir`(→ pathlib.Path)	Return the source-checkout config directory (`data/config`).
`discover_configs`(→ list[pathlib.Path])	List `*.json` config files for the Dataset tab dropdown.
`config_name`(→ str)	Return a config's display name (its `name` field, or the file stem).
`load_config`(→ tuple[str, ...)	Load a config file into a `(name, registry, db_path)` triple.
`to_config_dict`(→ dict)	Serialise a registry to a config dict.
`save_config`(→ pathlib.Path)	Write a registry to a config file as JSON.
`remember_last_config`(→ None)	Record `path` as the config to reopen on the next start.
`last_config`(→ pathlib.Path \| None)	Return the config remembered by `remember_last_config()`, if it still exists.

Module Contents¶

annie.dataset.datasets.repo_data_dir() → pathlib.Path[source]¶

Return the bundled data directory.

Prefers the explicit ANNIE_DATA_DIR override (settings.data_dir), which non-editable installs (Docker, PyPI) need because the package lives in site-packages and data does not. Otherwise falls back to three levels up from this file, which is valid only from a git clone. Callers must handle a non-existent path gracefully.

annie.dataset.datasets.bundled_config_dir() → pathlib.Path[source]¶

Return the source-checkout config directory (data/config).

May not exist when Annie is installed from PyPI.

annie.dataset.datasets.discover_configs(directory: str | pathlib.Path | None = None) → list[pathlib.Path][source]¶

List *.json config files for the Dataset tab dropdown.

Returns an empty list silently when a directory does not exist, so PyPI installs degrade gracefully without bundled examples.

With no argument this scans the bundled config directory and, when ANNIE_CONFIG_DIR is set (e.g. in Docker to a persistent volume path), appends that directory’s configs deduplicated by stem. When an explicit directory is passed it is treated as the sole search root — the ANNIE_CONFIG_DIR merge is not applied — so callers get exactly what they ask for.

Parameters:: directory – Explicit directory to scan exclusively; defaults to the bundled dir merged with ANNIE_CONFIG_DIR.
Returns:: Sorted config file paths, or [] if the directories are absent.

annie.dataset.datasets.config_name(path: str | pathlib.Path) → str[source]¶

Return a config’s display name (its name field, or the file stem).

Parameters:: path – The config file.
Returns:: The configured name, or the filename stem on any read/parse error.

annie.dataset.datasets.load_config(path: str | pathlib.Path) → tuple[str, annie.dataset.sources.SourceRegistry, pathlib.Path | None][source]¶

Load a config file into a (name, registry, db_path) triple.

Relative source paths — and the optional db path — are resolved against the config file’s directory. Column types for CSV sources are auto-detected from a 500-row sample when not declared in the config.

Parameters:: path – The config file to load.
Returns:: The dataset name, a populated SourceRegistry, and an optional Path to a pre-configured review database (None if the config has no "db" field).
Raises:: ValueError – If the JSON is malformed or a source kind is unknown.

annie.dataset.datasets.to_config_dict(registry: annie.dataset.sources.SourceRegistry, name: str, *, relative_to: str | pathlib.Path | None = None, db_path: str | pathlib.Path | None = None) → dict[source]¶

Serialise a registry to a config dict.

Parameters:

registry – The sources to serialise.
name – The dataset name to store.
relative_to – If given, store paths relative to this directory where possible.
db_path – Optional path to a review database to embed in the config so that loading it re-opens the same DB automatically.

Returns:

A JSON-serialisable config dict.

Write a registry to a config file as JSON.

Parameters:

path – Destination .json path.
registry – The sources to save.
name – The dataset name to store.
relative_to – If given, store paths relative to this directory where possible.
db_path – Optional review-database path to embed in the config.

Returns:

The path written.

annie.dataset.datasets.LAST_CONFIG_FILENAME = 'last_config.json'¶: File under ANNIE_HOME remembering the config most recently loaded or saved.

annie.dataset.datasets.remember_last_config(path: str | pathlib.Path) → None[source]¶

Record path as the config to reopen on the next start.

Parameters:: path – The config file just loaded or saved.

annie.dataset.datasets.last_config() → pathlib.Path | None[source]¶

Return the config remembered by remember_last_config(), if it still exists.

A config that is gone from its remembered path is still found when it moved together with the whole Annie home (e.g. the Windows installer moving your work to a new data folder): the same file name is looked up in the config folder.

Returns:: The config path, or None when nothing is remembered, the pointer file is unreadable, or the config has since been deleted or moved elsewhere.