create eval
create_evalQueue an eval run that compares models pairwise, applies a calibrated criterion, or screens cheaper models on logged traffic—so you can measure a change before shipping it.
Instructions
Queue an eval run — a pairwise model comparison, an absolute criterion (calibrated judge) run, or a one-click screening of cheaper models against your own logged traffic — so a customer can measure a model, prompt, tool or index change before shipping it. POST /v1/evals (API-key scope: evals:write). SPENDS MONEY: this starts billable work on the workspace wallet. Returns: 201 with the queued run in the same shape as GET /v1/evals/{id} (status PENDING, results null, progress_ratio 0, arms[] describing each candidate key). Poll GET /v1/evals/{id} until status is DONE|ERROR|CANCELLED, or gate a pipeline directly with GET /v1/evals/{id}/gate. Notes: MONEY: a run spends wallet credit (every generation for baseline + each arm, plus judge calls; best-of-N arms pay N×). Plain runs disclose cost and are gated lazily per tick by the runner; screening runs enforce a creation-time funds gate → 402 {error:{type:"insufficient_quota", code:"insufficient_balance"}}. The key's minting user must be workspace OWNER/ADMIN → otherwise 403. 400 on: invalid JSON, schema violations (name length, rubric length, sample_count range, >6 candidates, duplicates, candidate == baseline, bad assertion, bad arm label/override), unknown model, missing criterion_id, population-binding refusal, trace_replay without stored baseline, or too little population ("Not enough logged traffic for this filter (need at least 5 distinct prompts…)"). 404 for a dataset/criterion not in this workspace. Screening-specific: 422 {code:"unprocessable"} when there is nothing to screen (no/too little logged traffic, or nothing cheaper than the incumbent — the message carries the import hint), 400 for an unusable candidate_models list, 402 for funds. All errors are {error:{message, type, code}}. Input keys are snake_case; internally converted to camelCase (candidate_models→candidateModels, candidates[]→candidateModels keys + armOverrides map keyed by label, sample_filters.dataset_id→datasetId, trace_replay→traceReplay, max_output_tokens→genMaxOutputTokens).
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| name | Yes | Run name, 1..80 chars after trimming (required unless screening=true, where it is auto-generated). | |
| rubric | No | Judge rubric, 10..2000 chars. Required for eval_kind="comparison" (400 if shorter than 10 chars); ignored for criterion runs (the criterion's frozen judge prompt is the rubric). | |
| eval_kind | No | "comparison" (default): each candidate arm is judged pairwise against the baseline in both orderings → win rate with Wilson CI. "criterion": every model (baseline and candidates) is graded absolutely by a saved, calibrated criterion → observed and calibration-corrected pass rates; requires criterion_id. Default: "comparison". | |
| screening | No | Screening mode. When true every other field becomes an optional override and the server auto-fills like the dashboard's one-click: incumbent = your dominant logged model, baseline = its stored answers ("__stored__"), candidates = the cheapest model of each distinct family, judge family-checked, fixed quality rubric, sample_count = clamp(min(40, population), 5..500). Only name, sample_count, candidate_models (1..6 explicit picks, validated against the catalog: unknown id / the incumbent itself / duplicate / >6 / empty list = 400 naming the offender), judge_model, and sample_filters.dataset_id (screen an imported dataset's stored answers instead of logged traffic) are honoured in this mode; rubric, rubric_type, eval_kind, criterion_id, baseline_model, candidates[], assertions, other sample_filters and max_output_tokens are ignored. Screening additionally enforces a creation-time funds gate (402). | |
| assertions | No | Up to 10 deterministic output checks run against every generated answer at finalize (free, exact — these are what CI should gate on via min_assertion_pass_rate). Each: {type, value?}. Types: "json_valid" (no value), "json_schema" (value = JSON Schema string ≤4000 chars, must parse), "regex_match" (value = pattern ≤200 chars, must compile), "contains" / "not_contains" (value = substring, required), "max_length" / "min_length" (value = non-negative integer as string), "completed" (finish reason was not a length cut-off), "tool_called" (value = tool name ≤200 chars; matches the canonical "[tool call] name(args)" notation), "no_tool_call". Invalid configs are rejected 400 with the specific reason. | |
| candidates | No | Versioned arms — an alternative to candidate_models (when present, candidate_models is ignored and this list defines the arms; max 6 total). Each item: {model: string (required, catalog id that runs the arm), label?: string (arm key shown in the report; must match /^[A-Za-z0-9][A-Za-z0-9 _.:+\-]{0,63}$/ and must NOT contain "/"), system?: string (replace the logged system prompt on every sampled prompt; "" strips it; max 20000 chars; omit to keep the logged one), tools?: array of tool-definition objects (OpenAI format; replaces the logged tool definitions; [] offers none; max 64; omit to keep), n?: integer 2..8 (best-of-N: sample N times and keep the judge-preferred answer via N−1 pairwise knockout verdicts on the run's own rubric/judge; the arm pays for all N generations plus the selection verdicts)}. A bare {model} is identical to listing the id in candidate_models. Any arm that sets system, tools or n MUST carry a label (400 "candidates[]: an arm that overrides system or tools needs a label"); the label becomes the arm key in candidate_models/arms/results, and the override is stored as armOverrides[label] = {model, system?, tools?, n?}. Two arms may share one model (e.g. old prompt vs new prompt); metering follows the model that actually ran. | |
| judge_model | No | Catalog chat model used as the judge (validated; 400 if not offered). Precedence when omitted: the workspace's default judge, then the house default judge. Ignored for criterion runs (the criterion's judge is frozen). In screening mode an explicit judge is honoured even if it shares a family with a contestant (the report discloses judge_shares_family) instead of being swapped. | |
| rubric_type | No | How the judge reads the rubric: "direct" (default) judges answers on the rubric alone; "adherence" also requires each sample to carry a reference answer (the logged reply), so the population must have text replies. Default: "direct". | |
| criterion_id | No | Id of a workspace criterion (calibrated judge). Required when eval_kind="criterion" (400 otherwise). The criterion's judge model and prompt override judge_model/rubric and are frozen into the run (criterion_snapshot). A trace-unit criterion requires baseline_model="__stored__" and no candidates (it grades completed agent runs), and refuses if it was aligned on an older transcript instrument version. Population binding is enforced: sampling a tag/segment different from the criterion's calibrated population is refused (400); sampling with no population filter while the judge is scoped is allowed with a stored warning. | |
| sample_count | No | Number of prompts to sample from the population, integer 5..500. Defaults to 20 when omitted (non-screening). The run is refused at creation (400) if the filtered population cannot supply at least 5 distinct prompts (with references when rubric_type="adherence" or baseline is "__stored__"). Screening: clamped to 5..500, default min(40, population). Default: 20. | |
| baseline_model | Yes | The incumbent arm: a catalog model id this workspace is offered (validated, 400 "Model '…' is not available."), or the sentinel "__stored__" to judge candidates against the incumbent's STORED logged answers (nothing is regenerated for the baseline; no savings figure is computed). "__stored__" is required for trace_replay and for the certified-switch (noninferiority) gate shape. Required unless screening=true (then forced to "__stored__"). | |
| sample_filters | No | Which population prompts are sampled from (success-only logged requests by default). Keys: tag?: string (only requests logged with this tag); model?: string (only requests served by this model); segment?: string (an auto-detected traffic segment = the prompt FAMILY shown as `segment` on GET /v1/logs rows — one application surface's traffic, stable under interpolated dates/ids); dataset_id?: string (sample from a managed dataset — e.g. a holdout split — instead of live logs; must belong to this workspace, 404 otherwise; the run needs ≥5 usable rows); trace_replay?: boolean (replay bake-off: sample WHOLE completed agent runs from the last 7 days, one teacher-forced sample per step, max 12 steps per run; REQUIRES baseline_model="__stored__" (400 otherwise) and ≥5 replayable steps). Empty-string values are treated as absent. Screening mode reads only dataset_id here. | |
| candidate_models | No | Array of catalog model ids (max 6, no duplicates, none equal to baseline_model). Each candidate answers every sample and is judged, so cost is linear in this count. A comparison needs at least 1 (400 otherwise); a criterion run may have 0 (grade the baseline alone). Ignored when candidates[] is present (candidates[] replaces it). | |
| max_output_tokens | No | Per-answer generation output cap, integer 256..16384 (default 4096) — sized so thinking models can finish reasoning and answer; the runaway-spend guard. Silently ignored in screening mode. Default: 4096. |