Create or Update Check
create_or_update_checkIdempotent
Create or update a named quality check (upsert) for model-based, code-based, or audio evaluation, with pass/fail, score, or analysis output types.
Instructions
Create or update a quality check by name (upsert). Supports model-based, code-based, and audio checks.
Saving to an existing name creates a new version of that check (see
get_check's "available_versions"). Before writing a prompt_template or
code_contents from scratch, fetch a worked example with get_templates:
"boolean_check_prompt", "score_check_prompt", "analysis_check_prompt",
or "check_code".
Args:
name: Unique name for the check.
description: What the check evaluates.
check_type: "model" (an LLM judge driven by prompt_template) or
"code" (a deterministic Python class in code_contents).
output_type: "pass_fail" (boolean verdict), "score" (numeric, e.g.
a 1-5 rubric), or "analysis" (free-form qualitative feedback;
only valid with check_type="model"). For check_type="code" the
server infers pass_fail vs score from the value evaluate()
returns (bool vs int/float) — output_type is used only to
validate the request, not sent to the server. Note: list_checks
and get_check report this as output_data_type in the server
vocabulary, where "bool" means pass_fail and "int" means score.
prompt_template: Required when check_type="model". The judge
prompt. Inject the runtime data the judge needs with these
placeholders:
- {model_output}: the model output being evaluated. In a
multi-turn conversation this is ONLY the final assistant
message, not the full conversation.
- {scenario_input}: the scenario input / source text.
- {scenario_result}: the reference/expected output.
- {model_input}: what was sent to the model (prompt or
messages).
- {message_history}: the full multi-turn conversation — the
model_input messages plus the assistant's model_output. Use
this when the check must judge the whole conversation.
- {tool_calls}: the tool/function calls the model just made.
- {tools}: the tool definitions/schema available to the model.
- {model_output_metadata}: metadata attached to the most
recent model output.
- {simulation_message_history}: full conversation history
reconstructed from trace metadata. Only populated for traced
(ingested) conversations; for simulations and evaluations
use {message_history}.
The legacy {generation} placeholder is deprecated — use
{model_output} instead.
code_contents: Required when check_type="code" (output_type
"pass_fail" or "score" only). Python source defining
`class Check(CodeBasedCheck)` with a
`@staticmethod def evaluate(...) -> CheckResponse` method.
Start from `from okareo.checks import CodeBasedCheck,
CheckResponse`. evaluate() may declare any subset of these
parameters: model_output, scenario_input, scenario_result,
metadata, model_input. Return CheckResponse(score=...,
explanation=...) where score is a bool for pass_fail or an
int/float for score. See get_templates("check_code") for
complete examples.
is_audio: Set to true for audio/voice evaluation. Only valid with
check_type="model".
tags: Optional list of string tags to organize the check. Tags are
stored with the check and returned by get_check.
Input Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| name | Yes | ||
| tags | No | ||
| is_audio | No | ||
| check_type | Yes | ||
| description | Yes | ||
| output_type | Yes | ||
| code_contents | No | ||
| prompt_template | No |
Output Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| result | Yes |