Store one agent execution (input, output, tool calls, spans, cost, tokens) and get the trace_id later calls key on.

What it does. Writes one immutable trace to local storage; nothing is deduplicated. Only agent_name is required; store tool_calls, tools, cost and input/output so the rules can judge them. evaluate: true (with output) also scores it in this call, exactly as evaluate_output would. Full detail: iris://capabilities.

When not to use it. To score a stored trace (evaluate_output with its trace_id). To change one (delete_trace, then log again).

Returns. JSON: trace_id, status, cost_usd, cost_source, cost_estimate, evaluation.

Errors. IRIS_STORAGE_ERROR; IRIS_INVALID_ARGUMENT when evaluate is true without output. Unknown or malformed arguments are refused, naming the valid keys. Failures return {"error":{code,message,recovery}} with isError; follow recovery.

Siblings. evaluate_output — score the stored output; get_traces — query what was logged; delete_trace — remove one trace.

Score an output with deterministic rules: a ship verdict with its basis, per-rule evidence, what was unjudged.

What it does. eval_type picks a bundle or all (the default). A rule missing its input SKIPS, never passes: input is REQUIRED when eval_type="relevance"; tool_calls, tools, cost_usd, expected feed the rest. trace_id alone scores the stored trace. Full detail: iris://capabilities.

When not to use it. For semantic judgment (evaluate_with_llm_judge). As an input firewall: the rules read the output.

Returns. JSON: id, trace_id, reference_trace_id, verdict, coverage, provenance, run_id, interpretations, erased_at, eval_type, score, passed, rule_results, rules_evaluated, rules_skipped, insufficient_data, critical_failures, critical_skipped, categories, note, rules_changed.

Errors. IRIS_UNKNOWN_TRACE (nothing written); IRIS_STORAGE_ERROR. Failures return {"error":{code,message,recovery}} with isError; follow recovery.

Siblings. log_trace — record the run first; list_rules — what each rule needs.

Query stored traces with filters, pagination and sorting; optionally with the dashboard summary.

What it does. Read-only, local. Exact-match agent_name and framework, inclusive since and until, min_score and max_score on each trace's latest evaluation, and q: full-text search, ranked, with a snippet. limit is 1..1000 (default 50). A crossed range is refused, never returned as an empty page. Full detail: iris://capabilities.

When not to use it. To score a trace (evaluate_output) or create one (log_trace). As a live stream: poll with backoff.

Returns. JSON: untrusted, traces, total, limit, offset, summary, search.

Errors. IRIS_STORAGE_ERROR. Out-of-range bounds are refused, naming the values; no match is total 0, not an error. Failures return {"error":{code,message,recovery}} with isError; follow recovery.

Siblings. log_trace — record an execution; evaluate_output — score one output; delete_trace — remove one trace.

Did this change make the agent worse? Compares two runs: worse, better, equivalent, or too little evidence to tell.

What it does. Pairs by case_key (McNemar exact), else Newcombe. A rule counts only where it ran; no better if the second run was judged on less. Refuses unlike runs unless force. Local, no model call. Full detail: iris://capabilities.

When not to use it. To score one output (evaluate_output). As a gate: it reports; blocking is your policy.

Returns. JSON: comparable, incomparable_because, forced, method, before, after, difference, paired, worse, better, improvement_withheld, call, smallest_detectable, equivalent_within, coverage, critical_rises, rules_tested, regressions, improvements, discordant, discordant_total, summary, dataset.

Errors. IRIS_INVALID_ARGUMENT (empty run id, no baseline, bad margin or dataset); IRIS_STORAGE_ERROR. Failures return {"error":{code,message,recovery}} with isError; follow recovery.

Siblings. log_trace — record into a run; get_traces — find a run's traces.

How reliably does the agent answer the same question? Per-case pass rates, flaky cases, and an interval that respects repeats.

What it does. Groups evaluations by case_key (or session) and reports each case's pass rate with a Wilson interval, the flaky cases least reliable first, and an overall rate over cases, with an exact interval, rather than over pooled attempts. Deterministic, local, no model call. Full detail: iris://capabilities.

When not to use it. To compare two runs (compare_runs). To score an output (evaluate_output).

Returns. JSON: case_key, group_by, question, cases, attempts, overall, pooled, flaky_cases, by_case, summary.

Errors. IRIS_STORAGE_ERROR. No matching evaluations is cases 0, not an error. Failures return {"error":{code,message,recovery}} with isError; follow recovery.

Siblings. compare_runs — compare two runs against each other; log_trace — record an execution with a case_key; get_traces — read the traces themselves.

Re-score every trace in a run under the current rules into a new run, so a rules change is compared, not overwritten.

What it does. Writes the verdicts into a NEW run; the source run is never modified. Traces already scored under the current ruleset are skipped, so a repeat call does no work. Local, no model call. Full detail: iris://capabilities.

When not to use it. To score a new execution (log_trace, then evaluate_output).

Returns. JSON: source_run, run, ruleset_hash, traces, evaluated, already_current, failed, passed, not_checked, summary, rules_changed.

Errors. IRIS_INVALID_ARGUMENT (empty run id, or the target run holds verdicts); IRIS_UNKNOWN_TRACE (source run empty); IRIS_STORAGE_ERROR. Unscorable traces are listed in failed. Failures return {"error":{code,message,recovery}} with isError; follow recovery.

Siblings. compare_runs — compare the two runs after; evaluate_output — re-score one trace; get_traces — read a run.

The rule inventory: every built-in rule with what it needs, its effective criticality and published accuracy, plus every deployed custom rule.

What it does. Read-only, no network. built_in is never narrowed by the filters; a rule whose needs are absent skips, never passes. Read critical and criticalSource before trusting passed: true. rules is the custom-rule store, filterable by eval_type and enabled_only. Full detail: iris://capabilities.

When not to use it. To add, remove or pause a rule (deploy_rule, delete_rule); built-in rules cannot be.

Returns. JSON: untrusted, rules, total, enabled_count, built_in, quarantined, plugins.

Errors. IRIS_INTERNAL_ERROR if the store cannot be read; a missing store is an empty list. Failures return {"error":{code,message,recovery}} with isError; follow recovery.

Siblings. deploy_rule — add a custom rule; delete_rule — remove, disable or re-enable one; evaluate_output — run the rules.

Deploy a custom rule that fires on every future evaluate_output call of its bundle — persisted, active immediately, audited.

What it does. Stores the rule and loads it into the running engine. eval_type says WHEN it fires (that bundle, and eval_type="all"); severity says what a failure does: low and medium lower the score, high and critical fail the verdict. definition.type picks the check, definition.config its keys. A taken name is refused unless replace is true. Full detail: iris://capabilities.

When not to use it. For a one-off check (custom_rules on evaluate_output). To pause a rule (delete_rule with enabled: false).

Returns. JSON: rule, replaced, warning.

Errors. IRIS_DUPLICATE_RULE; IRIS_INVALID_RULE_CONFIG, naming the field; IRIS_STORAGE_ERROR. Failures return {"error":{code,message,recovery}} with isError; follow recovery.

Siblings. list_rules — see what is deployed; delete_rule — remove or disable; evaluate_output — where it fires.

Remove a deployed custom rule — or, with enabled, disable or re-enable it without removing it — effective on the next evaluate_output call.

What it does. Without enabled: deletes the rule and writes an audit entry; deleted is false when no rule has that id. With enabled: false stops it firing and keeps it stored, true re-enables it. Past evaluations are untouched. Full detail: iris://capabilities.

When not to use it. On built-in rules (they are not in the store). To replace a rule (deploy_rule with replace: true).

Returns. JSON: deleted, rule_id, toggled, enabled, rule.

Errors. IRIS_STORAGE_ERROR. A malformed rule_id is refused. Failures return {"error":{code,message,recovery}} with isError; follow recovery.

Siblings. deploy_rule — add or replace a rule; list_rules — find the id; evaluate_output — where the rule fires.

Remove one stored trace by id; its spans go with it, and every evaluation linked to it keeps its verdict and loses its text.

What it does. Linked evaluations keep their verdicts and scores; their text is erased and erased_at is stamped. deleted is false when no trace has that id. Every deletion is audited at iris://audit. Full detail: iris://capabilities.

When not to use it. To expire data in bulk (retention.days). To change a trace: traces are immutable.

Returns. JSON: deleted, trace_id.

Errors. IRIS_STORAGE_ERROR. A malformed trace_id is refused. Failures return {"error":{code,message,recovery}} with isError; follow recovery.

Siblings. log_trace — store a trace; get_traces — find the trace to delete; delete_rule — the equivalent for custom rules.

Score an output with an LLM judge on your own key: a 0..1 score, a rationale, the spend.

What it does. Calls Anthropic or OpenAI with IRIS_ANTHROPIC_API_KEY or IRIS_OPENAI_API_KEY; Iris never proxies. correctness needs expected, relevance input, faithfulness source_material. Worst-case cost is checked against max_cost_usd first. Full detail: iris://capabilities.

When not to use it. For length, keyword, PII or cost checks (evaluate_output, free).

Returns. JSON: id, reference_trace_id, score, passed, verdict, pass_threshold, self_reported_pass, disagreement, rationale, dimensions, model, provider, template, input_tokens, output_tokens, cost_usd, latency_ms, raw_response_id, warnings.

Errors. IRIS_JUDGE_NOT_ENABLED (recovery has the steps); IRIS_JUDGE_UNKNOWN_MODEL; IRIS_UNKNOWN_TRACE; IRIS_BUDGET_EXCEEDED; IRIS_PROVIDER_ERROR. Failures return {"error":{code,message,recovery}} with isError; follow recovery.

Siblings. evaluate_output — the free deterministic path; verify_citations — citation grounding.

Is each citation in an output supported by its source? Extracts, fetches (opt-in, SSRF-guarded) and judges on your key.

What it does. Fetches only if the operator set IRIS_CITATION_ALLOW_FETCH=1, blocking private addresses. One judge call per resolved citation, within max_cost_usd_total. overall_score: supported / judged; null if none judged. Full detail: iris://capabilities.

When not to use it. With no citations (evaluate_output). Without IRIS_ANTHROPIC_API_KEY or IRIS_OPENAI_API_KEY.

Returns. JSON: id, reference_trace_id, overall_score, passed, verdict, total_unsupported, total_citations_found, total_resolved, total_judged, total_supported, total_cost_usd, warnings, citations.

Errors. IRIS_JUDGE_NOT_ENABLED, IRIS_JUDGE_UNKNOWN_MODEL, IRIS_UNKNOWN_TRACE (before any spend); IRIS_JUDGE_FAILED when every judge call failed. Failures return {"error":{code,message,recovery}} with isError; follow recovery.

Siblings. evaluate_with_llm_judge — semantic scoring; evaluate_output — the free deterministic path.