omniloader.data.factory

Build datasets and schemas from declarative config entries.

This lets datasets be described in JSON/YAML (adapter + constructor args + schema) so tools like the CLI and OmniConfig can assemble a dataset mix without Python glue. Only the registered data adapters are ever constructed — never models.

A dataset entry looks like:

{
  "adapter": "hdf5",
  "args": {"h5_path": "data/mosei.h5", "subset": "train"},
  "schema": {
    "features": [
      {"name": "video", "feature_dim": 1024, "time_dim": 300},
      {"name": "eye_image", "shape": [3, 64, 64], "time_dim": 15}
    ],
    "targets": [{"name": "sentiment", "placeholder": -5.0}]
  }
}

shape declares a structured (e.g. image) trailing shape instead of a flat feature vector; see TensorSpec.

Attributes

ADAPTERS

Functions

schema_from_dict(→ omniloader.schema.spec.DatasetSchema)	Build a DatasetSchema from a {"features": [...], "targets": [...]} dict.
build_dataset(→ tuple[omniloader.loader.SizedDataset, ...)	Construct one (dataset, schema) pair from a config entry.
build_datasets(...)	Construct parallel lists of datasets and schemas from config entries.

Module Contents

omniloader.data.factory.ADAPTERS: dict[str, type]

omniloader.data.factory.schema_from_dict(data: collections.abc.Mapping[str, Any]) → omniloader.schema.spec.DatasetSchema

Build a DatasetSchema from a {"features": [...], "targets": [...]} dict.

Each spec dict is forwarded to TensorSpec/TensorSpec; the dtype field may be a string name (e.g. "int64").

Parameters:

data – The schema mapping.

Returns:

The constructed schema.

omniloader.data.factory.build_dataset(entry: collections.abc.Mapping[str, Any]) → tuple[omniloader.loader.SizedDataset, omniloader.schema.spec.DatasetSchema]

Construct one (dataset, schema) pair from a config entry.

Parameters:

entry – A dict with adapter (a key of ADAPTERS), args (the adapter constructor kwargs) and schema.

Returns:

The dataset instance and its schema.

Raises:

ValueError – If adapter is not registered.

omniloader.data.factory.build_datasets(entries: collections.abc.Sequence[collections.abc.Mapping[str, Any]]) → tuple[list[omniloader.loader.SizedDataset], list[omniloader.schema.spec.DatasetSchema]]

Construct parallel lists of datasets and schemas from config entries.

Parameters:

entries – A list of dataset entries (see build_dataset()).

Returns:

A (datasets, schemas) tuple ready for OmniLoader.