create_eval_criterion
Create a custom evaluation criterion with name, rubric, and score range. Choose between LLM-based scoring or deterministic evaluators for instant results.
Instructions
Create one custom eval criterion in your account (Pro+ only). name + rubric + scaleMin + scaleMax are required. Same name already existing in the account = 409. A name matching a global default is structurally allowed (UNIQUE (account_id, name) separates it from account_id IS NULL). type defaults to 'llm_judge' (judge LLM scoring). Specifying a deterministic evaluator type (exact_match / contains / regex / json_schema / json_path) scores without calling an LLM — free and instant (pass -> scaleMax / fail -> scaleMin). Deterministic types require config. The path an AI agent takes when it decides "add this criterion" during dogfood evals.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| name | Yes | Criterion name (1-50 chars, starts with an alphanumeric, [A-Za-z0-9 _\-.] only). E.g. 'helpfulness' / 'concise' | |
| type | No | Evaluator type (default 'llm_judge'). Deterministic evaluators score without an LLM call — free and instant: 'exact_match' / 'contains' / 'regex' / 'json_schema' / 'json_path' | |
| scope | No | Evaluation scope (default call). call = per call; trajectory = scores multiple calls + steps in the same trace as one trajectory (llm_judge only) | |
| config | No | Type-specific settings (not needed for llm_judge). exact_match: {expectedOutput}, contains: {substring, caseSensitive?}, regex: {pattern, flags?}, json_schema: {schema}, json_path: {path, expectedValue?}. Categorical scoring also requires config.categories (2-10 entries, worst to best). | |
| rubric | Yes | Scoring rubric text (10-2000 chars; the narrative the judge LLM bases scores on. Required as a human-readable explanation even for deterministic evaluators) | |
| scaleMax | Yes | Score upper bound (1-100, must be greater than scaleMin) | |
| scaleMin | Yes | Score lower bound (1-100, must be less than scaleMax) | |
| scoreType | No | Scoring type (default numeric). boolean = pass/fail; categorical requires config.categories (llm_judge only) |