create_evaluator
Create a grader to score LLM outputs with customizable criteria: LLM-based, code-based, or human evaluation. Specify type, scoring format, and optional passing conditions.
Instructions
Create a new evaluator (grader). Evaluators score LLM outputs.
REQUIRED: name, type, score_value_type.
TYPES:
"llm": LLM-based evaluation. Requires llm_config with model + evaluator_definition.
"code": Code-based evaluation. Requires code_config with eval_code_snippet.
"human": Manual human evaluation. No automation config needed.
SCORE VALUE TYPES: numerical, boolean, percentage, single_select, multi_select, json, text
FOR LLM EVALUATORS: llm_config must include:
model (required): e.g. "gpt-4o-mini"
evaluator_definition (required): Jinja2 prompt template. MUST contain {{output}}. Use {{input}} for user question, {{expected_output}} for ground truth.
scoring_rubric (recommended): Scoring instructions appended after definition.
temperature, max_tokens, top_p, etc. (optional)
EXAMPLE - Boolean LLM grader: { "name": "Hallucination Check", "type": "llm", "score_value_type": "boolean", "llm_config": { "model": "gpt-4o-mini", "evaluator_definition": "Score whether this output hallucinates.\nInput: {{input}}\nOutput: {{output}}\nReturn true or false.", "temperature": 0 } }
EXAMPLE - Numerical LLM grader with rubric: { "name": "Response Quality", "type": "llm", "score_value_type": "numerical", "score_config": { "min_score": 1, "max_score": 5 }, "passing_conditions": { "primary_score": { "operator": "gte", "value": 3 } }, "llm_config": { "model": "gpt-4o", "evaluator_definition": "Evaluate the quality of this response.\nInput: {{input}}\nOutput: {{output}}", "scoring_rubric": "1=terrible, 2=poor, 3=ok, 4=good, 5=excellent" } }
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| name | Yes | Evaluator name. | |
| type | Yes | Evaluator type: llm (requires llm_config), code (requires code_config), or human. | |
| llm_config | No | LLM automation config. Required for type="llm". Must include model + evaluator_definition. | |
| code_config | No | Code automation config. Required for type="code". | |
| description | No | Evaluator description. | |
| score_config | No | Score type configuration. | |
| evaluator_slug | No | Unique slug identifier. Auto-generated if not provided. | |
| score_value_type | Yes | Score format: numerical, boolean, percentage, single_select, multi_select, json, text. | |
| passing_conditions | No | Conditions for passing. Example: { "primary_score": { "operator": "gte", "value": 3 } } | |
| categorical_choices | No | Choices for single_select/multi_select score types. |