prepare_data
Create and register a dataset as a DataRef ID by running generator code with a seed or capturing an external URI, for train, validation, or test splits.
Instructions
Prepare a dataset and return a DataRef ID.
Two regimes:
generated: runs the generator (via executor), stores output, computes hash. Requires generator_code_ref, generator_seed. Generator contract: the file must expose generate_data(config, output_path) — config carries 'seed' AND 'generator_seed' (same value — either spelling works) plus generator_params flattened at the top level; a seed/generator_seed key inside generator_params is superseded so the recorded seed always equals the seed delivered. The dataset is written to output_path. A generator may print a single-line {"error": "..."} JSON object to stdout to report a structured failure. The generator runs under the executor's default interpreter ([executor] python) — not a bundle env; only packages installed there are importable.
captured: registers an external URI, computes hash if accessible. Requires source_uri.
For stream capture (captured + temporal), provide capture_window_start and capture_window_end.
The DataRef is stored in the state DB and the data is stored in data storage (separate from code storage). Access is read-only.
Returns: {"data_ref_id": "data-ref-...", "split": ..., "regime": ...}
generator_params and capture_source_metadata may be sent as JSON-encoded strings.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| split | Yes | Dataset split: train | validation | test. | |
| regime | Yes | generated (run generator via executor; needs generator_code_ref + generator_seed) | captured (register external URI; needs source_uri). | |
| version | No | Version tag for the captured data source. | |
| source_uri | No | External data URI — required for regime='captured'. | |
| generator_seed | No | Generator seed — required for regime='generated'. | |
| generator_params | No | Generator parameters; object or JSON-encoded. | |
| capture_window_end | No | Temporal bound for stream capture (captured + temporal). | |
| generator_code_ref | No | Path/ref to generator code — required for regime='generated'. The file must expose generate_data(config, output_path) — two positional args; see the tool description for the full contract. | |
| capture_window_start | No | Temporal bound for stream capture (captured + temporal). | |
| capture_source_metadata | No | Extra provenance about the captured source; object or JSON-encoded. |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| split | No | ||
| regime | No | ||
| data_ref_id | No | ||
| storage_uri | No | ||
| content_hash | No | ||
| reproducibility_risk | No |