create_experiment
Create and run an experiment that processes dataset rows through a workflow (prompt, model, or passthrough) and scores outputs with evaluator pipelines.
Instructions
Create and run an experiment. Processes a dataset's inputs through a workflow chain (prompt / model / passthrough) and scores results with evaluator pipelines.
REQUIRED: dataset_id, workflow, evaluator_workflow_ids.
WORKFLOW TYPES (these are how each dataset row produces an output):
"prompt": Use a saved prompt. Config: { prompt_id, version (optional) }
"completion": Direct model completion. Config: { model, temperature, max_tokens, top_p, response_format, tools, ... }
"duplicate": Passthrough — skip generation and just score the dataset's existing outputs. Use when your dataset already has outputs (e.g. logs imported from prod) and you only want to evaluate them.
"condition": Branch based on field values. Config: { condition_policy: { "event.": { operator, value } } }
EVALUATOR_WORKFLOW_IDS: Pass PIPELINE IDs (from list_evaluation_pipelines or create_evaluation_pipeline — the "id" field, NOT "workflow_id"). These pipelines score each row after the workflow completes.
EXAMPLE — Compare two models on a dataset: { "name": "GPT-4o vs Claude", "dataset_id": "ds_abc", "workflow": [ { "type": "completion", "config": { "model": "openai/gpt-4o", "temperature": 0 } } ], "evaluator_workflow_ids": [""] }
EXAMPLE — Score existing dataset outputs without re-running a model: { "name": "Score existing outputs", "dataset_id": "ds_with_outputs", "workflow": [ { "type": "duplicate", "config": { "name": "passthrough" } } ], "evaluator_workflow_ids": [""] }
EXAMPLE — Test a saved prompt version: { "name": "Prompt v3", "dataset_id": "ds_abc", "workflow": [ { "type": "prompt", "config": { "prompt_id": "prompt_xyz", "version": "3" } } ], "evaluator_workflow_ids": [""] }
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| name | No | Experiment name. | |
| workflow | Yes | Workflow tasks executed in order for each dataset row. | |
| batch_size | No | Rows processed per batch (default: 100). | |
| dataset_id | Yes | Dataset ID to run the experiment against. | |
| concurrency | No | Concurrent workers (default: 15). | |
| description | No | Experiment description. | |
| enable_tracing | No | Create trace logs for each row (default: true). | |
| evaluator_workflow_ids | Yes | Evaluator PIPELINE IDs (the 'id' from list_evaluation_pipelines / create_evaluation_pipeline, NOT 'workflow_id'). At least one required. |