Generate Check
generate_checkGenerate a check from a natural language description. AI creates the model prompt or Python code, then saves the check for evaluation.
Instructions
Generate a check from a natural language description. Uses AI to create the prompt template (model checks) or Python code (code checks), then saves the check.
Use this when you only have a description of what to evaluate. When
you already know the exact prompt template or Python code the check
should use, call create_or_update_check directly instead. The
generated prompt/code is returned in the response — review it and
refine with create_or_update_check if needed.
Args:
name: Name for the generated check.
description: Natural language description of what to evaluate
(e.g., "check if the response is toxic"). The more specific
the description, the better the generated check.
output_type: "pass_fail" (boolean verdict), "score" (numeric), or
"analysis" (free-form qualitative feedback; model checks only).
check_type: "model" (LLM judge) or "code" (deterministic Python).
requires_scenario_input: Set true when the evaluation must compare
the output against the scenario input. The generated check
will reference {scenario_input} and only works on runs whose
scenarios provide it.
requires_scenario_result: Set true when the evaluation must
compare the output against the expected result. The generated
check will reference {scenario_result} and only works on runs
whose scenarios provide it.
Input Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| name | Yes | ||
| check_type | No | model | |
| description | Yes | ||
| output_type | No | pass_fail | |
| requires_scenario_input | No | ||
| requires_scenario_result | No |
Output Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| result | Yes |