Evaluate With LLM Judge
evaluate_with_llm_judgeScore an agent output with an LLM judge using your own Anthropic or OpenAI key. Get a 0–1 score, rationale, and cost, with a worst-case cost cap checked before the call.
Instructions
Score an output with an LLM judge on your own key: a 0..1 score, a rationale, the spend.
What it does. Calls Anthropic or OpenAI with IRIS_ANTHROPIC_API_KEY or IRIS_OPENAI_API_KEY; Iris never proxies. correctness needs expected, relevance input, faithfulness source_material. Worst-case cost is checked against max_cost_usd first. Full detail: iris://capabilities.
When not to use it. For length, keyword, PII or cost checks (evaluate_output, free).
Returns. JSON: id, reference_trace_id, score, passed, verdict, pass_threshold, self_reported_pass, disagreement, rationale, dimensions, model, provider, template, input_tokens, output_tokens, cost_usd, latency_ms, raw_response_id, warnings.
Errors. IRIS_JUDGE_NOT_ENABLED (recovery has the steps); IRIS_JUDGE_UNKNOWN_MODEL; IRIS_UNKNOWN_TRACE; IRIS_BUDGET_EXCEEDED; IRIS_PROVIDER_ERROR. Failures return {"error":{code,message,recovery}} with isError; follow recovery.
Siblings. evaluate_output — the free deterministic path; verify_citations — citation grounding.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| input | No | User question / prompt that produced the output (required for relevance; improves helpfulness and safety) | |
| model | Yes | Model ID. Supported: anthropic = claude-fable-5-1 | claude-opus-5-5 | claude-sonnet-5 (and the priced 4.x line); openai = gpt-5 | gpt-5-mini | gpt-4.1-mini (and the priced 4o, o3/o4 and retired o1 ids); an unknown id is refused with the full priced list. Required — cost varies a hundredfold across models | |
| output | Yes | The agent output text to evaluate | |
| expected | No | Reference answer (required for correctness template) | |
| provider | No | Auto-detected from model when omitted | |
| template | Yes | The question the judge answers; correctness needs expected, relevance needs input, faithfulness and task_completed read source_material | |
| trace_id | No | Keep this evaluation beside a stored trace (id from log_trace / get_traces). It is listed with the trace and never replaces its verdict; an unknown id is rejected BEFORE the judge is called | |
| timeout_ms | No | Per-request timeout; default 60_000 | |
| agent_model | No | The model that produced the output, for the same-family warning, when no linked trace records it | |
| temperature | No | Sampling temperature; default 0 (deterministic) | |
| max_cost_usd | No | Cost cap in USD for this call, at most the operator's IRIS_LLM_JUDGE_MAX_COST_USD_PER_EVAL (default 0.25). The worst case (two attempts, full max_output_tokens) is computed before the call and refused if it exceeds the cap | |
| source_material | No | Provided RAG sources (required for faithfulness template) | |
| max_output_tokens | No | Judge output token cap; default 512 |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| id | Yes | ||
| model | Yes | ||
| score | Yes | ||
| passed | Yes | ||
| verdict | No | ||
| cost_usd | Yes | ||
| provider | Yes | ||
| template | Yes | ||
| warnings | No | ||
| rationale | Yes | ||
| dimensions | Yes | ||
| latency_ms | Yes | ||
| disagreement | No | ||
| input_tokens | Yes | ||
| output_tokens | Yes | ||
| pass_threshold | Yes | ||
| raw_response_id | No | ||
| reference_trace_id | No | ||
| self_reported_pass | No |