create dataset from logs
create_dataset_from_logsCurate production gateway logs into training and eval datasets, auto-dropping errored, truncated, duplicate, or human-failed exchanges and applying a quality gate for fine-tuning or evaluation.
Instructions
Curates logged gateway traffic into a managed training dataset (optionally with a disjoint eval holdout split), auto-dropping errored/truncated/empty/duplicate/human-failed/benchmark-contaminated exchanges and applying a chosen quality gate — use it to turn production logs into fine-tuning or eval data. POST /v1/datasets/from_logs (API-key scope: evals:write). Returns: 201 with snake_case: { summary: { total: , kept, dropped: { unparseable, errored, truncated, empty, duplicate, human_failed, contaminated }, folded: { folded_turns, conversations } }, quality: { mode, criterion_id?, criterion_name?, kappa?, human_pass_kept?, ungraded_excluded?, judged?, judge_passed?, judge_failed?, judge_unparsed?, judge_spend_usd? }, training_name, training_count, eval_name?: '-eval', eval_count? }. Notes: Requires an OWNER/ADMIN minting user for dataset creation (403, enforced in the dataset service). 400 (invalid_json) on unparseable JSON. Other 400s: name missing/over 80 chars; unknown quality mode; judge mode without criterion_id; judge criterion not found / trace-unit / misaligned / borderline / unmeasured / drift-flagged / segment-bound but build not scoped to that segment (each eligibility refusal is also written to the refusal ledger, kind dataset_judge_trust); no usable exchanges after curation (nothing created). Fetch is capped at 50,000 most recent matching rows. MONEY: judge mode is gated up front at ~$0.02 per conversation to judge (402 'Insufficient balance for judge gating' before any spend) and every judge call is then metered as usage; a scoring failure mid-run FAILS THE WHOLE BUILD (no dataset created) but rows already judged were billed (idempotent ids — retry does not re-bill). Human FAIL grades (or rows sharing an agent run with a trace-scoped FAIL) never enter a dataset in any mode. Multi-turn chats are folded into one weighted line per conversation. If the training set is created but the eval split fails, the response is a 400 that says the training dataset already exists. The eval split shares no example with the training set, so it is valid as an eval source (POST /v1/evals with sample_filters.dataset_id).
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| name | Yes | Dataset name, 1-80 chars after trimming. Also used as the file name ('<name>.jsonl') and, with holdout_pct, the eval split is named '<name>-eval'. | |
| filters | No | Which logged traffic feeds the build. Nested keys: model (string, exact model name), tag (string, the task label sent as X-Omnia-Tag), segment (string, an auto-detected traffic segment / prompt family as shown on GET /v1/logs rows), finish_reason (string), cache_hit (boolean), start (integer unix seconds, inclusive lower bound), end (integer unix seconds). Falsy values (empty string, 0) are ignored. Success-only is always enforced regardless of filters. | |
| quality | No | The quality ladder: { mode: 'cleaned'|'graded'|'judge', criterion_id?: string }. 'cleaned' (default) = mechanical curation only. 'graded' = keep only exchanges a human graded pass (free). 'judge' = a CALIBRATED judge (criterion_id REQUIRED; request-unit; trust 'trustworthy' or 'under-measured'; not drift-flagged; if segment-bound, filters.segment must equal its segment) keeps only passing conversations, judged at each conversation's terminal turn; human grades override the judge for free. mode must be a string, criterion_id a string when present. | |
| sources | No | Include-list of source models: { models: string[] }. Only exchanges served by these models feed the build; an empty array means no restriction. Each item must be a non-empty string, else 400 'sources.models must be an array of model names'. | |
| holdout_pct | No | Percentage (0-50) of curated lines carved into a second, DISJOINT '<name>-eval' dataset linked back to the training set. 0/omitted = no eval split. Values above 50 are capped at 50. | |
| decontaminate | No | Drop rows whose prompt shares a 13-word shingle with a public benchmark test split. Default true. Only an explicit boolean is honored. The provenance records what was checked; an unavailable index is recorded as 'not checked', never as clean. |