run_eval_dataset
Run a golden dataset against a target model to detect regressions. Scores outputs with a judge model and records eval scores for A/B comparison across runs.
Instructions
Run a golden dataset against a target model and produce a regression verdict (POST /v1/eval-datasets/:id/run, Pro+ only). Feeds each item's inputText to targetModel, has gpt-4o-mini score the outputs against the default criteria plus expectedOutput, and records eval_scores (regression A/B). Results can be compared across runs with compare_eval_runs. Run records are excluded from production cost / analytics / alert aggregation. Cost: item count x criteria count LLM calls. 503 in environments without OPENAI_API_KEY provisioned.
Input Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| datasetId | Yes | Id of the dataset to run (list_eval_datasets.datasets[].id) | |
| judgeModel | No | Judge model (omit = gpt-4o-mini). Only OpenAI models present in the pricing table. | |
| targetModel | Yes | Model to measure regressions for (only OpenAI models present in the pricing table, e.g. 'gpt-4o-mini'). Unknown models get 400. | |
| idempotencyKey | No | Opaque key for retry dedup (UUID recommended, 200 char cap). Re-POSTing the same key returns the existing run (double-billing prevention). |