Iris is a local-first MCP server for logging, evaluating, comparing, and governing AI agent runs with auditable deterministic rules and optional LLM judging.
Log traces (log_trace): store agent input/output, spans, tool calls, tools, token usage, cost, session/run/case keys, and metadata in local SQLite.
Evaluate outputs (evaluate_output): score with 26 deterministic rules across completeness, relevance, safety, and cost, returning a ship/no-ship verdict with evidence and coverage.
Query and search traces (get_traces): filter by agent, framework, session, time range, or score; paginate, sort, and full-text search stored text.
Compare runs (compare_runs): statistically compare two runs of evaluations (paired McNemar or unpaired Newcombe) to detect regressions, improvements, or equivalence.
Compare traces (compare_traces): compute per-case pass rates, Wilson intervals, and flaky cases across repeated questions or sessions.
Re-score runs (evaluate_runs): re-evaluate every trace in a run under current rules into a new run without altering the source.
Manage custom rules (list_rules, deploy_rule, delete_rule): inspect built-ins, deploy persisted custom rules with severity, and disable or remove them.
Delete traces (delete_trace): remove a trace and erase linked evaluation text while keeping verdicts and scores.
LLM-as-judge (evaluate_with_llm_judge): optional semantic scoring via Anthropic or OpenAI using your own key, with hard cost caps.
Verify citations (verify_citations): extract citations, optionally fetch sources through an SSRF-guarded resolver, and judge whether each supports the cited claim.
Local-first and private: no account, no telemetry, MIT licensed; outbound network only on explicit opt-in.
Send Iris an agent run and ask for a verdict on its quality, safety, and cost — on your machine, with no SDK and no account. Most agent projects check quality by running a few remembered prompts and eyeballing the output. Iris replaces that with numbers you can audit: your agent's runs land in a SQLite database on your disk, 26 built-in rules score them deterministically — PII, prompt injection, hallucination markers, cost thresholds, and the agent's own tool calls — free, with no LLM calls, and an optional LLM judge with a hard per-eval cost cap handles the semantic questions. Every rule is inspectable and editable, because a judge you can't audit is just vibes with a number on it. MIT licensed, no telemetry. Nothing leaves your machine unless you turn one of these on: an OpenTelemetry endpoint (IRIS_OTEL_ENDPOINT), which exports traces to the collector you name; the LLM judge with your own key, which sends the text it judges to that provider, and whose citation check fetches the pages an output cites; or a webhook, which posts ids, the verdict and rule names, never the text, to the address you set.
Requires Node.js 22.13 or later. Check with node --version.
The demo database, recorded by scripts/demo-media.mts; the source is demo.mp4. A still: dashboard-overview.png.
No agent wiring, no config — one command:
npx @iris-eval/mcp-server --demoThis seeds a demo database — five small agents, two weeks of runs, every verdict the engine's own — and serves the dashboard against it at http://localhost:6920 (your browser opens automatically on first run). The dashboard lands on Failures: what failed, worst and newest first, each card naming the rule and its evidence. Worth clicking into — a PII leak caught by the safety rules, a hidden directive in a forum post that the summarizer complied with, a number the source document never said, two runs on the same twelve questions compared with an interval (Runs), a deployed custom rule and a paused one with their audit rows, and a failed LLM-judge score with its rationale.
Demo data lives in its own database (demo.db in your Iris home directory — ~/.iris on macOS/Linux, %USERPROFILE%\.iris on Windows) and never mixes with your real traces. Remove all of it with one command:
npx @iris-eval/mcp-server --demo-clearRelated MCP server: runmeter
First, prove the install works on this machine — it runs offline and opens nothing of yours:
npx @iris-eval/mcp-server --self-test # exit 0 = healthyThen add Iris to your MCP client. One command writes the client's own config file, keeps every other server in it, and pins the version you ran:
npx -y @iris-eval/mcp-server install claude-codeThe clients: claude-code, claude-desktop, cursor, windsurf, continue, vscode, cline, zed, codex, gemini. install --list shows the ones found on this machine, the Iris each one runs and the file it reads; install <client> --uninstall takes Iris out again. Every client shares one database, so after an upgrade move them all at once with install --upgrade (Updating). Restart the client to load it.
Claude Desktop: one click. Each release from 0.20.0 on attaches iris-eval.mcpb, an MCP Bundle: download the latest, open it, and Claude Desktop shows an install dialog. Nothing on it is required — an Anthropic or OpenAI key for the LLM judge is optional, and the dashboard is a switch that starts off. The bundle holds the npm package and its dependencies, so nothing else needs installing: Claude Desktop runs it under the Node it ships when that Node is 22.13 or newer (Claude Desktop 1.1.6679 ships 24.13), and Iris stores traces with Node's built-in SQLite, in the same ~/.iris every other install uses. The release notes show how to verify its signature and build attestation.
It runs in any MCP client, and every client it names has a row saying how it is checked. Verified on every CI run: Claude Code, Gemini CLI — the real client starts Iris from the config the installer wrote and reports it connected (claude mcp list, gemini mcp list), on Linux, macOS and Windows; Claude Code's capture plugin hooks are driven through the real scripts too. Configured from each client's own MCP documentation — the installer writes the file each client documents, at its documented path and in its own format (an mcpServers block, VS Code's servers, Zed's context_servers, Codex's TOML), keeping comments and every other server, and unit tests hold each path and every format writer to that documentation: Claude Desktop, Cursor, Devin Desktop (Windsurf), Continue, VS Code, Cline, Zed, OpenAI Codex CLI. Every row with its source and the date it was read: https://iris-eval.com/clients. By hand instead, one block, dashboard included:
{
"mcpServers": {
"iris-eval": {
"command": "npx",
"args": ["-y", "@iris-eval/mcp-server", "--dashboard"]
}
}
}Your client lists Iris's twelve tools on connect, and the dashboard serves at http://localhost:6920. Now paste this to your agent:
Log that last task to Iris and evaluate the output.
The trace lands on the dashboard with its scores. Prefer the MCP server headless? Drop --dashboard from the args — you can open the same dashboard any time with npx @iris-eval/mcp-server --dashboard.
One thing worth knowing up front: MCP tools are called when the model decides to call them. Iris doesn't intercept your agent, so traces are logged when your agent asks it to log them — either because you told it to, or because your code calls the tools directly. Ask your agent to "log this to Iris and evaluate it" and it will. If you want capture that doesn't depend on the model choosing, POST /api/v1/traces does exactly that — your code sends the trace over plain HTTP, no model in the loop (see docs/http-ingest.md). The CLI and host hooks on the roadmap will be thin clients over the same endpoint.
The ingest endpoint lives on the dashboard port — 6920 by default, not the MCP transport port — and it exists only while the dashboard is running. Pass --dashboard (or set IRIS_DASHBOARD=true); --transport http on its own does not start it, and a request to the transport port returns 404. With the dashboard up, anything that can send an HTTP request can log a trace — and optionally run the deterministic evals in the same request. GET /api/v1/capabilities on the same port says what this server can judge, what each rule needs, the judge state with the steps that enable it, and the limits — the same object the MCP resource iris://capabilities serves — so an HTTP caller has the frame an MCP client gets at initialize:
curl -s -X POST "http://127.0.0.1:6920/api/v1/traces" \
-H "Content-Type: application/json" \
-d '{
"agent_name": "support-bot",
"input": "What is the refund policy?",
"output": "Refunds are available within 30 days of purchase.",
"evaluate": true,
"eval_type": "safety"
}'Returns 201 with the stored trace_id and the evaluation result (in --demo mode the endpoint refuses writes with 403, so demo data never mixes with yours). The endpoint accepts the same body as the log_trace tool and sits behind the same middleware stack as the rest of the dashboard: loopback bind and the DNS-rebinding guard by default, plus Bearer auth when you set one. Two plain facts about it: it accepts unauthenticated writes unless Iris was started with --api-key (or IRIS_API_KEY) — the loopback bind is what keeps it to your machine by default, so set a key before binding beyond loopback; and what it stores is verbatim — input and output land in iris.db exactly as sent, including any text no_pii goes on to flag. Full contract, field reference, and error semantics: docs/http-ingest.md.
/plugin marketplace add iris-eval/mcp-server
/plugin install iris-eval-capture@iris-evalA second, separately installed plugin: its hooks record each turn's prompt, tool calls (the failed ones included, with the error the agent received) and final answer, keyed on the prompt, so an interrupted turn's calls and a background sub-agent's are sent too, and hand them to iris-eval ingest, detached, with critical spans redacted in the stored evaluation text — capture that does not depend on the model deciding to call a tool. A turn that ended in an API error is stored without being judged; when the model logged the turn too, both traces are kept and the plugin's names the model's. It never prints, never blocks, never sends anything anywhere. Installing iris-eval alone changes nothing about your turn loop. Limits and removal: claude-plugin-capture/README.md.
pip install iris-evalfrom iris_eval import IrisClient
iris = IrisClient() # IRIS_URL, or the running dashboard's runtime.json
iris.evaluate_output("…", input="…", agent_name="support-bot")["verdict"] # {"state": "pass", "basis": "clean", "by": []}A thin client over the HTTP API of server 0.16.0 and later, versioned on its own — iris_eval.__version__ and the PyPI page carry its number, which is not the server's: log_trace(), evaluate_output(), get_traces(), get_trace(), health(), capabilities(), sync and async, typed answers, the server's own sentence on a refusal — and a pytest plugin: an iris fixture and assert_iris(output, expect="pass") that asserts on the verdict's state. packages/python/README.md.
from iris_eval import wrap_openai
client = wrap_openai(OpenAI(), agent_name="support-bot") # every call: a GenAI span to POST /v1/traces, scoredimport { wrapOpenAI } from '@iris-eval/sdk';
const openai = wrapOpenAI(new OpenAI(), { agentName: 'support-bot' });Wrap the provider client once and each model call becomes one OpenTelemetry GenAI span sent to the OTLP door, stored with its input, output, token usage and tool calls, and scored: capture that does not depend on the model calling a tool. wrap_openai / wrap_anthropic in the Python client; wrapOpenAI, wrapAnthropic and irisMiddleware for the Vercel AI SDK in @iris-eval/sdk. Both are not yet published (the next iris-eval release on PyPI; @iris-eval/sdk is built from source until its first npm release). Streams, the SDKs' stream helpers and tool calls are covered, the original client is not changed, and Iris being down never breaks a call — packages/sdk/README.md, packages/python/README.md.
from iris_eval.langchain import IrisCallbackHandler
graph.invoke(inputs, config={"callbacks": [IrisCallbackHandler(agent_name="support-bot")]})Each top-level run becomes one trace (the run, its model calls, its tool calls and its graph nodes as GenAI spans) with its input, output, tool calls, token usage and a verdict. Python in the client (pip install iris-eval, 0.2.0 and later), JavaScript as @iris-eval/langchain (in this repository, not yet published to npm). Both are proven in CI against a real LangGraph app with a scripted model; LangSmith's own OpenTelemetry export is proven the same way — docs/otel-recipes.md.
npx -y @iris-eval/mcp-server ingest --file traces.ndjson --evaluate --fail-on anyany fails the job on every verdict that is not a pass: a failure, and a verdict that was not checked. Name one basis instead (--fail-on detector_veto) to fail only on that one.
Or the GitHub Action (0.16.0), which fails the job on the verdicts you name, writes the receipt to the job summary and posts it as one pull-request comment updated in place: uses: iris-eval/mcp-server/.github/actions/gate@v0.21.0 with traces: traces.ndjson — docs/ci-gate.md.
A fourth door (0.15.0): POST /v1/traces on the dashboard port takes the OTLP/HTTP JSON or protobuf your OpenTelemetry instrumentation already emits (the Python SDK's exporter speaks protobuf only, so this is the Python door too), and each OTLP trace becomes an Iris trace with its spans — docs/otel-integration.md; one recipe per framework (Pydantic AI, Google ADK, LangGraph via LangSmith, CrewAI, the OpenAI Agents SDK in Python and JavaScript, LlamaIndex, AutoGen, Microsoft Agent Framework, Semantic Kernel, the Vercel AI SDK and Mastra), each proved by a fixture, in docs/otel-recipes.md. ingest reads one JSON trace (or NDJSON, one per line) from stdin or a file, stores it, evaluates it under exactly the rules evaluate_output runs, prints one JSON line per trace with the verdict and its basis, and exits 1 when a verdict matches --fail-on. --dataset <id|label> restricts that gate to the case keys in a dataset (POST /api/v1/datasets promotes a run's case keys into one), so a job fails only on the cases you chose. The full recipe, the exit codes and the eight bases are in docs/ci-gate.md.
eval.plugins in config.json loads rules you wrote — an ES module whose default export is { name, kind, mechanism, version, needs, evaluate(ctx) } — pinned by the sha256 of the file, so a file that changed since you pinned it refuses startup rather than running. A loaded plugin fires like a built-in and shows on list_rules under plugins. The contract, the hash recipe and what a plugin may return: docs/plugins.md.
The evaluation engine is importable — no server, no database, no model:
import { EvalEngine, defaultConfig } from '@iris-eval/mcp-server/engine';
const engine = new EvalEngine(defaultConfig.eval.defaultThreshold, defaultConfig.eval.ruleThresholds, defaultConfig.eval);
const result = await engine.evaluateAll({ output: answer, input: prompt, toolCalls, costUsd });
result.verdict.state; // 'pass' | 'fail' | 'unknown', with result.verdict.basis and result.interpretationsThe same engine, the same rules and the same composer the server runs; builtInRules(), createCustomRule(), compose() and the published-accuracy readers are exported beside it.
import { createClient } from '@iris-eval/mcp-server/client';
const iris = createClient({ baseUrl: 'http://127.0.0.1:6920', apiKey: process.env.IRIS_API_KEY });
const { trace_id, evaluation } = await iris.logTrace({ agent_name: 'support-bot', input, output, tool_calls, evaluate: true });
evaluation?.verdict?.state; // the same object evaluate_output returnsOne body on every door: it is what log_trace and iris-eval ingest accept. A refusal throws IrisClientError with the server's own sentence and status. Both subpaths are checked from a packed tarball on every build.
npx @iris-eval/mcp-server --self-test # offline diagnostic; exit 0 = healthy, 1 = a check failed
npx @iris-eval/mcp-server --version # prints the bare version, e.g. 1.2.3--self-test first creates your Iris home if it is missing and checks that it is writable (exit 1, naming the path, if it is not), reports where your database's search index is (whole, how many traces a background build has indexed so far, or no FTS5 on this SQLite), reads your database's schema (exit 1, with the fix, when this version or an MCP client pinned to an older release cannot open it), then runs its checks — storage round-trip, a planted SSN and a planted injection caught by the safety rules, dashboard boot, the DNS-rebinding guard — inside an isolated temp home. Your real database is only read, never changed. Everything Iris writes lives under one directory, your Iris home: ~/.iris by default (%USERPROFILE%\.iris on Windows), or wherever IRIS_HOME points. That is where iris.db, config.json, custom-rules.json, audit.log, preferences.json and the demo files live; point IRIS_HOME at a scratch directory to try Iris without touching your real data.
Client | Status | What that means | Read |
Claude Code | verified | a test drives the real client on every CI run | |
Claude Desktop | claimed | configured from the client's own MCP documentation; the installer's path and format for it are unit-tested | |
Cursor | claimed | configured from the client's own MCP documentation; the installer's path and format for it are unit-tested | |
Devin Desktop (Windsurf) | claimed | configured from the client's own MCP documentation; the installer's path and format for it are unit-tested | |
Continue | claimed | configured from the client's own MCP documentation; the installer's path and format for it are unit-tested | |
VS Code | claimed | configured from the client's own MCP documentation; the installer's path and format for it are unit-tested | |
Cline | claimed | configured from the client's own MCP documentation; the installer's path and format for it are unit-tested | |
Zed | claimed | configured from the client's own MCP documentation; the installer's path and format for it are unit-tested | |
OpenAI Codex CLI | claimed | configured from the client's own MCP documentation; the installer's path and format for it are unit-tested | |
Gemini CLI | verified | a test drives the real client on every CI run |
Every row with what was checked: iris-eval.com/clients. No client is called supported without a row.
npx -y @iris-eval/mcp-server install <client> writes each of these for you. By hand, per client:
Edit your MCP config file:
macOS: ~/Library/Application Support/Claude/claude_desktop_config.json
Windows: %APPDATA%\Claude\claude_desktop_config.json
Add the JSON config above, then restart Claude Desktop.
claude mcp add --transport stdio iris-eval -- npx -y @iris-eval/mcp-serverThen restart the session (/clear or relaunch) for tools to load.
Windows note: Do not use
cmd /cwrapper — it causes path parsing issues. Thenpxcommand works directly.
Add the JSON config above to ~/.cursor/mcp.json (every project) or .cursor/mcp.json in a workspace, with "type": "stdio" in the iris-eval entry — Cursor's docs mark it required.
Add the JSON config above to mcp_config.json: ~/.config/devin/mcp_config.json on macOS and Linux, %APPDATA%\devin\mcp_config.json on Windows.
Save the JSON config above as its own file in Continue's mcpServers folder: ~/.continue/mcpServers/iris-eval.json (every workspace) or .continue/mcpServers/iris-eval.json in one.
Add to .vscode/mcp.json in your workspace (note: VS Code uses servers, not mcpServers):
{
"servers": {
"iris-eval": {
"command": "npx",
"args": ["-y", "@iris-eval/mcp-server"]
}
}
}Open Cline's MCP Servers panel → Configure MCP Servers, and add the mcpServers JSON config above to cline_mcp_settings.json (~/.cline/data/settings/cline_mcp_settings.json, shared by Cline in VS Code, JetBrains and the CLI).
Add to Zed settings.json:
{
"context_servers": {
"iris-eval": {
"command": "npx",
"args": ["-y", "@iris-eval/mcp-server"],
"env": {}
}
}
}Add to ~/.codex/config.toml:
[mcp_servers.iris-eval]
command = "npx"
args = ["-y", "@iris-eval/mcp-server"]Add the mcpServers JSON config above to ~/.gemini/settings.json. Gemini CLI connects to MCP servers only in folders it trusts: if gemini mcp list shows iris-eval as Disabled, run /permissions in that folder.
Iris is a standard stdio MCP server — one npx @iris-eval/mcp-server command, no SDK, no code changes. If your client supports MCP, it supports Iris. Client config formats change; when in doubt, check your client's MCP docs and point it at that command.
# Global install (recommended for persistent data and faster startup)
npm install -g @iris-eval/mcp-server
iris-eval --dashboard
# Docker — two servers, two ports: 3000 = MCP HTTP transport,
# 6920 = dashboard (which also serves the POST /api/v1/traces ingest endpoint).
# The image binds 0.0.0.0 inside the container, so a key is required (see Production).
# The volume is the Iris home: the database, deployed rules and audit log persist in it.
docker run -p 3000:3000 -p 6920:6920 -v iris-data:/data \
-e IRIS_API_KEY="$(openssl rand -hex 32)" ghcr.io/iris-eval/mcp-serverTip: Global install (
npm install -g) stores traces persistently at~/.iris/iris.db. Withnpx, traces persist in the same location, but startup is slower due to package resolution.
The container image, and the MCP directories that list Iris:
Trace Logging | Hierarchical span trees with per-tool-call latency, token usage, and cost in USD. Stored in SQLite, queryable instantly. |
Output Evaluation | 26 built-in rules across 4 categories: completeness, relevance, safety, cost. PII detection (21 patterns: SSN, credit card, phone, email, IBAN, DOB, MRN, IP, API key, passport, plus AWS/Slack/SendGrid/GitHub/Google/npm/DigitalOcean tokens, credentials inside URLs, secret-named assignments, PEM private-key blocks and seed phrases; date of birth, medical record number, passport and seed phrase fire only beside their label, by design), prompt injection (38 patterns, phrase + structural), stub-output detection, hallucination detection (25 context-grounded fabrication/contradiction signals — pass |
LLM-as-Judge | Optional semantic scoring via Anthropic or OpenAI — bring your own API key. Seven templates. With |
Cost Visibility | Aggregate cost across all agents over any time window. Set budget thresholds. Get flagged when agents overspend. A trace that sends token counts and a model but no cost (most OpenTelemetry and framework traces) is priced at the model's list price and marked estimated everywhere it shows; |
Web Dashboard | Real-time dark-mode UI that lands on the failures, worst and newest first — trace visualization with full-text search over every trace's text, eval results, cost breakdowns, and a command palette (⌘K) that searches your own rules, traces, and evals. |
Local-first | Everything lives in SQLite on your disk. The self-hosted server needs no account or sign-up and sends no telemetry. Outbound HTTP happens only where you opt in: your own LLM-judge key, citation fetching, an OTel exporter you configure, or a webhook you set. |
Where this is going next: the capability map — every question Iris can be asked about every subject, with what it has and what it lacks — and the three tracks.
Every built-in rule has a published precision, recall and F1 with 95% confidence intervals, measured on a labelled corpus that lives in this repository (proof/corpus/) and regenerates with one command — npm run proof — offline, with no key and no model in the loop. Those numbers are two different kinds, and the page never adds them together: some rules are measured against labels a model gave by reading the failure itself, which measures detection; the rest are checked against their own documented definition, applied independently, which shows the code implements its formula and says nothing about whether the formula catches the failure. proof/RESULTS.md and the proof page mark each rule. CI re-runs the measurement on every pull request and fails if the committed numbers differ from what the code produces, so a rule cannot change without its numbers changing with it. The numbers are on iris-eval.com/proof and in proof/RESULTS.md; how the corpus was made, what it is not, and how to read an interval are in docs/proof.md. The corpus is synthetic and model-labelled — a human blind label is pending, and the page says so; node proof/blind-sample.mjs draws the reproducible sample that will settle it.
Iris registers twelve tools that any MCP-compatible agent can invoke — trace and rule lifecycle, comparison across runs, LLM-as-judge and semantic citation verification:
log_trace — Log an agent execution with spans, tool calls, token usage, and cost; pass evaluate: true to score it in the same call
evaluate_output — Score output quality against completeness, relevance, safety, and cost rules (heuristic, deterministic, free)
get_traces — Query stored traces with filtering, pagination, and time-range support, and find the run where the agent said something with q: full-text search over input, output, tool-call values and metadata, ranked, with the matched words marked
list_rules — Enumerate deployed custom eval rules (read-only)
deploy_rule — Register a new custom eval rule so it fires on every evaluate_output of that category
delete_rule — Remove a deployed custom rule (destructive, idempotent)
delete_trace — Remove a single stored trace by ID (destructive, tenant-scoped)
evaluate_with_llm_judge — Semantic eval via LLM (Anthropic or OpenAI). Seven templates: accuracy, helpfulness, safety, correctness, faithfulness, task_completed, relevance. Cost-capped, per-eval pricing disclosed. Bring your own API key (IRIS_ANTHROPIC_API_KEY or IRIS_OPENAI_API_KEY) — Iris doesn't proxy or relay LLM calls.
verify_citations — Extract citations from output (numbered, author-year, URLs, DOIs), fetch sources behind an SSRF-guarded + domain-allowlisted resolver, and use an LLM judge to check whether each source actually supports the cited claim. Opt-in outbound HTTP. Same BYOK requirement as evaluate_with_llm_judge.
compare_runs — Did a change make the agent worse? Compares two runs of stored evaluations: a paired exact test when the runs share case keys, an interval on the difference otherwise, an honest "cannot tell" with the number of cases it would take, or "equivalent within a margin". Every rule carries its own one-sided test, corrected together (Benjamini–Hochberg) so twenty rules cannot manufacture a regression
compare_traces — How reliably does the agent answer the same question? Per-case pass rates with intervals, flaky cases first, and an overall rate that respects repeats
evaluate_runs — Re-score every trace in a run under today's rules into a new run, so a rules change is never read as an agent change
Enable the LLM judge (optional; the deterministic rules never need it)
Get an API key from Anthropic or OpenAI.
Put it in the environment of the process that runs Iris, not only your shell. Claude Code, Claude Desktop, Cursor and most MCP clients: the "env" block of the iris-eval entry in your MCP config — "iris-eval": { "command": "npx", "args": ["-y", "@iris-eval/mcp-server"], "env": { "IRIS_ANTHROPIC_API_KEY": "sk-ant-..." } } (IRIS_OPENAI_API_KEY for an OpenAI key). Docker: -e IRIS_ANTHROPIC_API_KEY=... on the run command. HTTP or CI: export it before starting iris-eval.
Restart the MCP session. A running process never sees a variable set after it started.
Confirm from inside your client: read iris://capabilities — judge.enabled must be true there. A key exported in your shell is not passed to the process your client spawns unless its config lists it. On a machine, npx @iris-eval/mcp-server --self-test prints the judge line for that shell, and GET /api/v1/health reports judge.enabled on a running dashboard.
Spend guard: each call is capped by IRIS_LLM_JUDGE_MAX_COST_USD_PER_EVAL (default 0.25 USD) and refused before any spend if the worst case would exceed it. Every judge call Iris makes on your key draws on one daily budget, IRIS_LLM_JUDGE_DAILY_BUDGET_USD (default 1 USD per UTC day); past it, no call is made until midnight UTC. Iris calls the provider directly with your key and never proxies it.
Optional: set IRIS_RELEVANCE_JUDGE_MODEL to a priced model id (claude-haiku-4-5, for example) to have answers_the_ask ask the judge whether each answer addresses its ask, and fail an off-topic one. That is one judge call per evaluation that carries an input, on your key and under the cap above; the key alone never turns it on. Each call sends that input and output to the model's provider, with the personal data and credentials no_pii flags replaced first (IRIS_RELEVANCE_JUDGE_REDACT=off sends them as they are). It draws on that daily budget and makes at most IRIS_RELEVANCE_JUDGE_MAX_CALLS_PER_REQUEST calls per request (default 20); past either, answers_the_ask reads the ask lexically and says why.
When IRIS_OTEL_ENDPOINT is configured, log_trace calls also emit a best-effort OTLP/HTTP JSON export to any OpenTelemetry collector (Jaeger, Grafana Tempo, Datadog OTLP, Honeycomb, etc). See docs/otel-integration.md.
passed is decidedevaluate_output returns both a score and a passed flag — they answer different questions:
score (0..1) is the weighted average across the rules that ran — a quality gradient.
passed is the ship/no-ship verdict, and the score is never consulted for it. A composer reads each rule by the kind of claim it makes: a policy you configured gates; a critical detector vetoes; a critical check that was asked and could not answer makes the verdict unknown (passed: false) rather than clean; every remaining detector combines into one probability that the output is bad, weighed against the loss ratio you state in eval.falsePassCost (default 1, so the cut is 0.5). verdict.basis names the layer that decided and verdict.by the rules, and verdict.also lists every later layer that would have decided it too; interpretations[] says why a rule that failed did not decide and which setting would change that, and names any question that was not judged and the input that would let it be.
Genuine safety violations hard-fail. By default no_pii, no_injection_patterns, and no_blocklist_words are critical rules: if one fails, the eval reports passed: false no matter how well the other rules scored, and the response names the culprits in critical_failures. A leaked SSN can't be averaged away. Which built-in rules are critical is a deployment setting (eval.criticalRules / eval.nonCriticalRules); every rule result carries the effective critical flag and criticalSource, and list_rules reports the roster this server applies. Custom rules deployed with severity: "high" or "critical" hard-fail the same way; low/medium severities only affect the score. One boundary to know, stated the same way on every surface: a critical rule that skipped (missing context, a broken definition, or a regex killed at the sandbox budget) has not judged the output and does not veto — every such rule is named in critical_skipped. A gate that must fail closed treats a non-empty critical_skipped as unknown, not clean, and may treat any budgetExceeded skip in rule_results the same way.
For CI gates: if you omit eval_type, every bundle runs — completeness, relevance, safety, cost and any custom rules — and the response says eval_type: "all" with a note that the default ran, plus a per-bundle categories map. Each row carries state: the verdict, read for the rules that bundle holds, so a row never reads passed over the rules that failed the evaluation. A bundle with nothing to judge (cost without cost_usd, relevance without input) reports state: "unknown" and passed: null there — not evaluated, not failing — and never counts toward the verdict. The response always echoes the eval_type that ran, so your gate can verify coverage; key on passed for the verdict and name a bundle only when you want a narrower run.
Two ways to add a rule. Inline rules ride along on one evaluate_output call (custom_rules, up to 10 per call); they fire alongside whatever eval_type bundle you chose, or alone with eval_type: "custom". Deployed rules are registered once with deploy_rule, persist in custom-rules.json under your Iris home, and fire on every future evaluate_output of their evalType. The definition is the same shape either way:
Field | Required | What it is |
| yes | 1–80 characters; appears as |
| yes | one of |
| yes | the keys for that type: |
| no | weight in the score; default |
deploy_rule wraps the definition with name, an optional description, evalType (completeness · relevance · safety · cost · custom) and severity. Severity says what a failure means: low/medium only lower the score; high/critical hard-fail the evaluation — passed: false, the rule named in critical_failures — whatever the weighted score says. A rule that skips (a cost_threshold rule with no cost_usd, or a regex killed at the 100 ms sandbox budget) has not judged the output and is listed in critical_skipped instead. Deploy a critical rule that forbids internal hostnames in anything the agent says:
{
"name": "no_internal_hostnames",
"description": "Output must not mention internal hostnames.",
"evalType": "safety",
"severity": "critical",
"definition": {
"name": "no_internal_hostnames",
"type": "regex_no_match",
"config": { "pattern": "\\b[a-z0-9-]+\\.internal\\.example\\b", "flags": "i" }
}
}The response is the persisted rule — keep the id for delete_rule:
{ "rule": { "id": "rule-588823d0", "name": "no_internal_hostnames", "evalType": "safety", "severity": "critical", "enabled": true, "version": 1, "definition": { "…": "…" } } }From the very next evaluate_output with eval_type: "safety", an output that mentions db-primary.internal.example comes back passed: false with critical_failures: ["no_internal_hostnames"] — even though all five built-in safety rules passed and the weighted score is 0.895. Regex patterns must pass a ReDoS check at deploy time and always run in a sandbox worker under a hard 100 ms deadline. list_rules shows what is deployed; the dashboard's rule composer builds the same shape from a failure you clicked on. Full reference, scoring per type, and worked examples: docs/custom-rules.md.
Full tool schemas and configuration: iris-eval.com
Iris runs on your machine or your own infrastructure. The open-source core is free and MIT licensed, with no usage limits and no account. Hosted storage, shared team history and alerting are under consideration, not under construction; if they would help you, tell us at hello@iris-eval.com.
Two commitments hold: nothing that is free today will move behind a paywall, and no compliance certification will be claimed before it is held.
Claude Desktop setup — MCP config for stdio and HTTP modes
TypeScript — MCP SDK client — connect and invoke tools
HTTP transport (TS + Python) — full client code for REST-style integration
A LangGraph agent, scored run by run (Python) — IrisCallbackHandler in the graph's callbacks; CI runs the same graph with a scripted model
A CrewAI crew over OpenTelemetry (Python) — the OpenInference instrumentor straight to Iris's OTLP door, the CrewAI recipe as a script
An OpenAI Agents SDK agent over OpenTelemetry (Python) and (JavaScript), and a LlamaIndex agent — each recipe as the script CI runs against a real Iris server
GitHub Issues — Bug reports and feature requests
GitHub Discussions — Questions and ideas
Getting help — Where to ask, report a bug, or report a vulnerability
Contributing Guide — How to contribute; AGENTS.md is the same guide for coding agents
Governance — Who decides, how a change lands, and how a release is made and verified
HTTP Ingest — Deterministic trace capture via POST /api/v1/traces
Capability map — Every question Iris can be asked, and what it lacks
Versioning policy — What each version number promises, and what has to be true before 1.0
Flag | Default | Description |
|
| Transport type: |
|
| HTTP transport port |
|
| SQLite database path |
|
| Config file path |
| — | API key for HTTP authentication (transport and dashboard, including |
|
| Enable web dashboard. Also the only way the |
|
| Dashboard port |
|
| Dashboard bind address. Loopback by default — the dashboard is unauthenticated unless |
|
| Seed a demo database (separate from your real traces) and serve the dashboard against it |
|
| Delete the demo database and exit |
|
| Run the offline install diagnostic in an isolated temp home, then exit (0 = healthy, 1 = a check failed). It also reads the configured database, read-only, and fails when this version or a pinned MCP client cannot open it |
|
| Delete every stored trace, span and evaluation from the configured database, compact the file and truncate the write-ahead log so the deleted text does not linger on disk, then exit. Deployed rules, the audit log and preferences are kept. Not reversible. Stop any running Iris server first — the file is compacted in place. Refuses to combine with |
| — | Print the bare version (e.g. |
Three commands take their own arguments and exit: iris-eval ingest loads traces from a file or stdin (A CI gate, no server needed), iris-eval export traces|evaluations --format csv|jsonl writes what is stored, filtered like the dashboard's lists, to stdout or --out (docs/api-reference.md), and iris-eval install <client> writes Iris into an MCP client's config — --uninstall takes it out, --list shows the clients found on this machine and the Iris each runs, --upgrade moves every client that runs Iris to this version (Hook up your own agent, Updating). None starts a server.
config.json is validated when Iris starts. A key Iris does not read — a typo such as eval.critcalRules, a key from another tool — or a value of the wrong type refuses startup with one sentence naming the full key, the key it most likely meant, or the type it wanted. Nothing in the file is silently ignored.
Every variable --help documents. CLI flags take precedence over environment variables when both are set.
Variable | Description |
| Transport type ( |
| HTTP transport bind address (default |
| HTTP transport port (1-65535, default |
| Directory for all per-user files: |
| SQLite database path (overrides |
| Which SQLite driver holds the database: |
| How long one trace search ( |
|
|
| Log level: |
|
|
| Dashboard port (1-65535, default |
| The receiver of the webhook that fires on a moment — merged over |
| The webhook's signing key (any string, or |
| Dashboard bind address (default |
| API key for HTTP authentication. Required to bind the HTTP transport or the dashboard beyond loopback ( |
| Path to a file whose trimmed contents are the API key — the secret-file pattern Docker and Kubernetes mount, so the key never sits in an environment block. Set this or |
| Set to |
| Comma-separated origin allowlist. Dashboard: CORS headers (supports globs, e.g. |
| The most array elements and object keys one input may hold before it is validated (a whole number, default |
| The name of |
| Set to |
| Required by |
| Required by |
| Hard cost cap per LLM judge call (default |
| A priced judge model id (e.g. |
| What every judge call together may spend per UTC day, per tenant (default |
| The name of |
| Relevance judge calls one request may make (default |
|
|
| Set to |
| Comma-separated hostname allowlist for |
| The most one |
| Enable best-effort OTLP/HTTP JSON trace export to this collector URL |
|
|
| Comma-separated |
| Per-export timeout (default |
| Website only — salts the address hash the iris-eval.com playground rate limit keeps; the server never reads it |
When using HTTP transport, Iris includes:
API key authentication with timing-safe comparison (Bearer for API clients; browser sign-in to the dashboard via ?key=)
CORS restricted to localhost by default
Rate limiting per client address and minute: 600 requests to the dashboard API (security.rateLimit.api) and 20 to the MCP endpoint (security.rateLimit.mcp), both set in config.json; an MCP request over the limit gets a JSON-RPC error that names the key
Helmet security headers
Zod input validation on all routes
ReDoS-safe regex for custom eval rules
One 1MB request size limit on every transport (security.requestSizeLimit): HTTP answers 413, stdio answers a JSON-RPC error and keeps the session open
One limit of 100,000 array elements and object keys in an input, counted before it is validated (security.maxInputElements): a tool call's arguments on every transport get an error result over it, and a dashboard or OTLP request body a 413
Validation errors return at most 20 issues and the total (issueCount), so an answer stays small however wrong the request
# Production deployment
iris-eval --transport http --port 3000 --api-key "$(openssl rand -hex 32)" --dashboardWith a key set, API clients — MCP clients, capture SDKs, POST /api/v1/traces — send Authorization: Bearer <key>. To open the dashboard in a browser, append the key once to any dashboard URL, http://localhost:6920/?key=<api key>: Iris exchanges it for an HttpOnly, SameSite=Lax session cookie and redirects to the same page with the key removed from the address bar. A page opened without a session shows a sign-in form that does the same exchange. The key is never stored in the browser, and sessions live only in the server process (at most 256 live at a time; a sign-in that finds them all live is refused rather than evicting one).
Several keys, and rotation without a gap. security.apiKeys in config.json holds any number of further keys, each with an id and exactly one of keyFile (a file whose trimmed contents are the key) or keyHash (the sha256 hex of the key, so the config file holds no secret — printf %s "$KEY" | openssl dgst -sha256), and an optional expiresAt (ISO 8601) after which it stops matching at that instant. To rotate: add the new key, move your clients, remove the old key. Keys in config.json and in key files take effect without a restart (0.20.0): on each request the server checks whether config.json or a key file it names has changed, and if so re-reads the keys before answering. Removing a key from security.apiKeys, or deleting its key file, revokes it on the next request: that request is refused, and every browser session opened with it is signed out. A config.json that cannot be read (for example, half-written) fails closed, and until it is fixed only a key from IRIS_API_KEY or --api-key is accepted. The key in IRIS_API_KEY or --api-key itself, and whether authentication is on at all, still change only on a restart. Every key authenticates until it is removed or expires, on the Bearer path and on the browser sign-in alike; the startup log names the ids. security.rateLimit.mcpKeyBy: "apiKey" counts the MCP endpoint's per-minute budget per key instead of per client address, so several agents behind one address each get their own minute.
Several clients. --transport http serves any number of MCP clients at one address: each initialize opens a session of its own over the one engine and store, up to 256 at a time. Over stdio each client starts its own server process, and those processes share the Iris home: a rule deployed, switched off or deleted through one applies in every other within 20 ms, a label written through one is read by the others as soon, and two of them changing rules at the same moment lose neither change.
Iris refuses to start when the HTTP transport or the dashboard is bound beyond loopback — 0.0.0.0, a LAN address, a container — with no API key, and says so in one sentence naming IRIS_API_KEY. That includes a bare docker run of the image, which binds 0.0.0.0 inside the container because loopback is unreachable through a published port. Loopback without a key keeps working (with a warning on the HTTP transport): the machine boundary is the exposure control there.
# The image: pass a key
docker run -p 3000:3000 -p 6920:6920 -v iris-data:/data \
-e IRIS_API_KEY="$(openssl rand -hex 32)" ghcr.io/iris-eval/mcp-server
# Compose: the file requires the variable and refuses before the container starts
IRIS_API_KEY="$(openssl rand -hex 32)" docker compose up
# A network you have already fenced some other way: run open, on purpose
IRIS_ALLOW_UNAUTHENTICATED=1 iris-eval --transport http --dashboardOpen by design, on a keyed server: GET /health on the transport and GET /api/v1/health on the dashboard answer without a key and outside every rate limit, in one shape: status, version, uptime, the SQLite driver, checks for storage, the deployed-rules file and the migrations (applied against known), the search index's state (search: ready, or how far a build has got as a share of the traces), and whether a judge key is present — never the key, never a trace, never a count of them. status is ok only when every check is; otherwise it is degraded with HTTP 503, which the Docker image's own HEALTHCHECK reads. Everything else needs Authorization: Bearer <key> or a browser session. Retention runs on every server: traces and evaluations older than retention.days (default 30) are deleted at startup, once the server is answering, and every retention.sweepIntervalHours, in short steps that never make a request wait long; --self-test prints this install's policy, and iris://capabilities / GET /api/v1/capabilities carry it as retention.
A webhook fires on a moment (0.16.0): notify.webhook in config.json (or IRIS_WEBHOOK_URL and IRIS_WEBHOOK_SECRET) names a receiver, and Iris posts one signed message when a verdict fails, a critical detection vetoes, a cost is an outlier, a rule's fail rate shifts, or a case is answered both ways for the first time — ids, the verdict, the rules and the numbers, never the agent's text. Signed the Standard Webhooks way and the GitHub way at once, retried with backoff, cooled down per agent and rule, never in the way of the evaluation; Slack and Discord bodies built in. docs/webhooks.md.
Everything Iris stores lives under your Iris home (~/.iris, or IRIS_HOME). iris.db keeps every trace's input and output verbatim — including any text no_pii goes on to flag; detection does not redact unless you ask it to: storage.redact: "critical_spans" in config.json stores each evaluation's output with the spans a critical detector flagged replaced by [REDACTED:<pattern>] (off by default; the evidence offsets still index the text the caller saw). storage.synchronous sets when a write reaches the disk: normal (the default) syncs the write-ahead log at each checkpoint, so a crash of Iris loses nothing and the file cannot be corrupted, but a power cut or an operating-system crash can undo the writes since the last sync; full syncs every commit and keeps them through both, at about 1.5 ms more per write. At startup, and every retention.sweepIntervalHours (default 24, 0 disables the timer) after that, traces and evaluations older than retention.days (default 30, 0 disables, set in config.json) are deleted and the write-ahead log is checkpointed. One thing is kept past the window: the pinned baseline run, with its evaluations, because it is what every later run is compared against. Each sweep that keeps some of it says so in the log with the run and the counts; unpin the run and the next sweep deletes it. Dataset cases and labels are not traces or evaluations and are not swept. Deleting a trace — by delete_trace or by the sweep — erases the text of every evaluation linked to it (the output, the expected text, and the rule messages) and stamps erased_at; the verdict, the scores and the evidence offsets stay. Each delete checkpoints the write-ahead log before it returns, so the deleted text is not left readable in iris.db or iris.db-wal (if a search is reading the file at that moment, or another process is reading or writing it, the delete returns without waiting and the text leaves the file as soon as it finishes). To remove everything now, stop the server and run --purge: it deletes every stored trace, span and evaluation, compacts the database and truncates the write-ahead log so the text is gone from disk, and keeps your deployed rules, audit log and preferences. Before a release applies a migration to an existing iris.db, it copies the file next to it (iris.db.<from>-to-<to>.<time>.bak, owner-only, the newest three kept; Downgrading): the copy holds the traces as they were, so the retention sweep deletes one older than retention.days and --purge deletes them all. The server does the copy and the migrations after it has answered its client, on a thread of its own: tool calls, resource reads and HTTP requests that arrive meanwhile wait for them, at most 30 s each, and are then refused with a sentence saying what the server is doing (IRIS_STORAGE_ERROR, retryable; HTTP 503 with Retry-After). Health answers throughout and says what the upgrade is doing. From 0.19.0 at 100,000 traces that are each an agent loop, the copy and the migrations took about 6 s. iris-eval ingest, --purge and --self-test still upgrade before they do anything else.
Iris does not encrypt its data at rest. iris.db and its write-ahead-log files are created owner-only (mode 600), and the Iris home directory is created mode 700 (on Windows, file ACLs govern instead). The database stores no LLM provider keys: IRIS_ANTHROPIC_API_KEY and IRIS_OPENAI_API_KEY are read from the environment and never written to disk. It does store trace inputs and outputs verbatim, so put the Iris home on an encrypted disk or volume (FileVault, BitLocker, LUKS, or an encrypted cloud volume for the Docker image's /data mount).
An export — the Export button on the dashboard's Traces and Evaluations pages, GET /api/v1/traces/export and /api/v1/evaluations/export, or iris-eval export — carries this stored text as it is, the same as the dashboard shows it: trace input and output verbatim, evaluation output with storage.redact applied. Treat an exported file like the database it came from.
npx @iris-eval/mcp-server --self-testIt checks storage, the deterministic evals, and the dashboard in an isolated temp home and prints a per-step verdict — the failure output names the broken step. Exit code 0 means the install is healthy.
ERR_MODULE_NOT_FOUNDYou may have a cached older version. Clear the npx cache and retry:
npx --yes @iris-eval/mcp-server@latestOr install globally to avoid cache issues entirely:
npm install -g @iris-eval/mcp-server@latestnpm install --ignore-scripts broke the SQLite bindingIris stores traces with better-sqlite3, a native module that fetches or compiles its binding in an install script. If that script was skipped — --ignore-scripts on the command line, ignore-scripts=true in an .npmrc (common on corporate machines), or a registry mirror that strips postinstall — startup fails with a long "Could not locate the bindings file" dump listing a dozen paths it tried. Rebuild that one module:
npm rebuild better-sqlite3
# for a global install:
npm rebuild -g better-sqlite3MCP tools only load at session start. After adding iris-eval, restart the session with /clear or relaunch the terminal.
npx @iris-eval/mcp-server --versionThe first startup log line also carries it (Starting Iris MCP server vX.Y.Z), and --self-test prints it in its summary. For a global install, npm ls -g @iris-eval/mcp-server shows the installed version.
Every MCP client on a machine shares one database, ~/.iris/iris.db, and install pins each client to the release that wrote its config. When a release changes the database's schema, the first process of that release to open the file upgrades it, and from then on a client still pinned to an older release refuses to start. So move every client in one step, before or right after you upgrade:
npx -y @iris-eval/mcp-server@latest install --upgradeIt finds every client config on this machine that runs Iris, moves each pin to that release (keeping anything you added to the entry, such as --dashboard or an env block), leaves alone a pin to a newer release and an entry that runs something other than the npm package, and lists what it did. Restart the clients it names. install --list shows which Iris each client runs.
Two installs live outside those files: the Claude Desktop extension (iris-eval.mcpb) moves when you open a newer bundle, and the Claude Code plugins with claude plugin marketplace update iris-eval and then claude plugin update iris-eval@iris-eval (and claude plugin update iris-eval-capture@iris-eval for the capture plugin).
Upgrading from 0.19.x to 0.20.0. 0.20.0 adds the search index and other additions to the database (migrations 015 and later). Once any 0.20.0 process has opened ~/.iris/iris.db (the Claude Desktop extension, npx iris-eval, or npx @iris-eval/mcp-server with no version), a client pinned to 0.19.x stops with This database was migrated by a newer Iris (…) — migration(s) 015-trace-search, … are unknown to v0.19.0. Upgrade Iris, …. That message comes from 0.19.x and cannot change; the fix is the command above. Before the upgrade, 0.20.0 copies the file next to it, so going back is possible too (below).
A start that upgrades the database prints what it did on stderr: the copy it took, which older releases can no longer open the file, and any client on this machine pinned to one of them, with the command. --self-test reads the database without changing it and says the same before you start anything.
For a global install, npm update -g @iris-eval/mcp-server, then iris-eval install --upgrade.
A release that upgraded the database copies it first, next to it: iris.db.<from>-to-<to>.<time>.bak in your Iris home (<from> is the release that last changed the file's schema, <to> the one that upgraded it; the startup line printed the exact path). To go back:
Stop every MCP client and any other Iris process that uses the database.
Keep the upgraded file, in case you come back: rename iris.db to iris.db.upgraded, and delete iris.db-wal and iris.db-shm if they are there.
Copy the backup to iris.db: cp ~/.iris/iris.db.0.19.0-to-0.20.0.<time>.bak ~/.iris/iris.db.
Pin every client back to the older release: npx -y @iris-eval/mcp-server@0.19.0 install <client> for each one (install --upgrade never moves a client back).
Traces stored after the upgrade are in iris.db.upgraded, not in the backup. If no copy was taken (the startup line says why, for example a full disk), the older release cannot open the upgraded file, and the way forward is install --upgrade.
On a platform with no prebuilt better-sqlite3, the install still succeeds. better-sqlite3 is an optional dependency: when npm can neither download a prebuilt binary for your Node and platform nor compile one (compiling needs Python and a C++ toolchain — Visual Studio's C++ build tools on Windows), npm prints the build error, skips the module, and finishes the install. Iris then runs on Node's built-in SQLite, and says so: startup prints one line on stderr naming why, and --self-test shows driver node: better-sqlite3 is not installed …. To get the native driver back, install it where a prebuild or a toolchain exists (npm install better-sqlite3 in the project; for a global install, install Iris again with npm install -g @iris-eval/mcp-server once a toolchain is available). CI installs the packed server with the native build forced to fail on every change, and requires the install to finish and the self-test to store and read a trace on the built-in.
Iris keeps everything in one SQLite file, opened by better-sqlite3 — a native addon that is downloaded or compiled for your Node and platform. When that module cannot load, Iris falls back to Node's built-in SQLite (node:sqlite, Node 22.13 or later) with one warning on stderr, so a missing prebuild is a slower start rather than a dead one. It does the same, before loading it, for a better-sqlite3 compiled on your machine against Node 24.19 or later headers and run on Node 24.19 to 24.21: there such a binary aborts the whole process the first time it frees a statement (Assertion failed: (env) != nullptr, nodejs/node#65446). Node 24.22.0 carries the fix, and Iris loads the binary there. npm rebuild better-sqlite3 replaces it with the prebuilt binary, which is safe on every version. IRIS_SQLITE_DRIVER=node chooses the built-in on purpose, native forbids the fallback. The built-in is opened with extension loading off and trusted_schema off; Node prints its own ExperimentalWarning: SQLite is an experimental feature line on stderr when it loads, and Iris does not silence it. --self-test and GET /health name the driver in use; every number on the proof page was measured on the native driver, and the test suite runs on both in CI.
Iris requires Node.js 22.13 or later. Node 20 reached end of life on 2026-04-30 and is not supported; Node 18 went in April 2025.
The floor is 22.13 rather than 22.0 because 22.13.0 is the first release that ships node:sqlite. That makes it the first version on which every supported Iris install has a second storage driver: when the native better-sqlite3 addon will not load, Iris falls back to Node's built-in SQLite instead of failing to start. Below 22.13 — and on Node 20, for its whole life — there was only ever one driver, and a missing prebuild was a dead start.
node --version # Must be v22.13.0 or newercmd /c not neededClaude Code's /doctor may suggest wrapping npx with cmd /c. This is not needed and causes path parsing issues. Use npx directly:
# Correct
claude mcp add --transport stdio iris-eval -- npx -y @iris-eval/mcp-server
# Wrong (causes /c to be parsed as a path)
claude mcp add --transport stdio iris-eval -- cmd /c "npx -y @iris-eval/mcp-server"If Iris is useful to you, consider starring the repo — it helps others find it.
MIT Licensed.
Did this change make the agent worse? Compares two runs: worse, better, equivalent, or too little evidence to tell.
What it does. Pairs by case_key (McNemar exact), else Newcombe. A rule counts only where it ran; no better if the second run was judged on less. Refuses unlike runs unless force. Local, no model call. Full detail: iris://capabilities.
When not to use it. To score one output (evaluate_output). As a gate: it reports; blocking is your policy.
Returns. JSON: comparable, incomparable_because, forced, method, before, after, difference, paired, worse, better, improvement_withheld, call, smallest_detectable, equivalent_within, coverage, critical_rises, rules_tested, regressions, improvements, discordant, discordant_total, summary, dataset.
Errors. IRIS_INVALID_ARGUMENT (empty run id, no baseline, bad margin or dataset); IRIS_STORAGE_ERROR. Failures return {"error":{code,message,recovery}} with isError; follow recovery.
Siblings. log_trace — record into a run; get_traces — find a run's traces.
| Name | Required | Description | Default |
|---|---|---|---|
| after | Yes | the run id to compare against it | |
| force | No | Compare even when the runs are not strictly comparable; the response still names what changed | |
| before | No | the baseline run id (`run` on log_trace); omit to use the run pinned with PATCH /api/v1/runs/:id | |
| dataset | No | Restrict both runs to the case keys of this dataset (its id or label); every count then covers only those cases | |
| equivalence_margin | No | δ for the equivalence test, as a difference in pass rate (0.05 = five points). Absent: equivalence is not tested; how far apart two runs may be and still count as the same is yours to choose |
| Name | Required | Description |
|---|---|---|
| call | Yes | |
| after | Yes | |
| worse | Yes | |
| before | Yes | |
| better | Yes | |
| forced | Yes | |
| method | Yes | |
| paired | Yes | |
| dataset | Yes | |
| summary | Yes | |
| coverage | Yes | |
| comparable | Yes | |
| difference | Yes | |
| discordant | Yes | |
| regressions | Yes | |
| improvements | Yes | |
| rules_tested | Yes | |
| critical_rises | Yes | |
| discordant_total | Yes | |
| equivalent_within | Yes | |
| smallest_detectable | Yes | |
| improvement_withheld | Yes | |
| incomparable_because | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnly/idempotent/non-destructive; the description still adds real behavior beyond them (McNemar vs Newcombe pairing, counting only where a rule ran, withholding 'better' on thinner coverage, refusing unlike runs without force, local with no model call). That is substantive, but the error codes and recovery guidance are procedural boilerplate the agent could get from failure responses, so it sits mid-range rather than exceptional.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loaded with the decision question and organized under labeled sections (What it does, When not to use, Returns, Errors, Siblings), which makes it scannable. The ~22-item return field list is long and largely redundant with the output schema, costing some concision.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a five-parameter comparison tool with an output schema, the description still covers statistical method, refusal conditions, error surfaces, and sibling routing. Nothing an agent needs in order to choose and invoke it correctly is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the baseline is 3; the description earns above that by explaining the consequence of force on comparability and that dataset scoping restricts every count to those case keys. It leaves equivalence_margin's semantics almost entirely to the schema's own description.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Opens with the decision question ('Did this change make the agent worse?') and names the specific operation and its outcome space: worse, better, equivalent, or too little evidence. The agent can distinguish it from compare_traces and evaluate_runs without opening a schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Contains an explicit 'When not to use it' block naming evaluate_output for single-output scoring and warning against using it as a gate, plus the condition that unlike runs are refused unless force is set. Alternatives and exclusions are stated, not implied.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
How reliably does the agent answer the same question? Per-case pass rates, flaky cases, and an interval that respects repeats.
What it does. Groups evaluations by case_key (or session) and reports each case's pass rate with a Wilson interval, the flaky cases least reliable first, and an overall rate over cases, with an exact interval, rather than over pooled attempts. Deterministic, local, no model call. Full detail: iris://capabilities.
When not to use it. To compare two runs (compare_runs). To score an output (evaluate_output).
Returns. JSON: case_key, group_by, question, cases, attempts, overall, pooled, flaky_cases, by_case, summary.
Errors. IRIS_STORAGE_ERROR. No matching evaluations is cases 0, not an error. Failures return {"error":{code,message,recovery}} with isError; follow recovery.
Siblings. compare_runs — compare two runs against each other; log_trace — record an execution with a case_key; get_traces — read the traces themselves.
| Name | Required | Description | Default |
|---|---|---|---|
| run | No | narrow to one run; omit to read every evaluation that carries a case key | |
| session | No | narrow to one session (the turns logged with that session_id) | |
| case_key | No | narrow to a single case — the fastest way to ask "is this one question flaky?" | |
| group_by | No | case_key (default): repeats of one question; session: the turns of one conversation, so a session answered both ways reads as flaky and by_case rows carry session ids | |
| question | No | read the rate for ONE question: only evaluations that judged it count, and an attempt passes when every rule answering it passed — not the composed verdict | |
| min_attempts | No | ignore cases asked fewer than this many times (default 1). A case asked once cannot be shown to be flaky |
| Name | Required | Description |
|---|---|---|
| cases | Yes | |
| pooled | Yes | |
| by_case | Yes | |
| overall | Yes | |
| summary | Yes | |
| attempts | Yes | |
| case_key | Yes | |
| group_by | Yes | |
| question | Yes | |
| flaky_cases | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnly/idempotent/non-destructive/closed-world, but the description adds genuinely new behavior: deterministic and local with no model call, IRIS_STORAGE_ERROR as the failure mode, the non-obvious fact that zero matching evaluations returns cases=0 rather than an error, and the {error:{code,message,recovery}} envelope with isError. That is exactly the operational context annotations cannot express.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Well front-loaded with the core question, then labeled sections (What it does / When not to use it / Returns / Errors / Siblings) that make it skimmable. Slightly redundant: the Siblings line largely repeats the 'When not to use it' exclusions, and the Returns field list duplicates the output schema.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a six-parameter analytics tool with an output schema, the definition covers purpose, exclusions, failure shape, the empty-result edge case, and a pointer to iris://capabilities for full detail. An agent has everything needed to select and call it correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100% and the enum parameters are documented in-schema, so the baseline is 3. The description restates the group_by dichotomy (case_key vs session) and 'over cases rather than pooled attempts,' which mirrors what the schema already says and adds no syntax or edge-case guidance for run, session, min_attempts, or question.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Opens with the concrete question the tool answers (per-case pass rates, flaky cases, interval respecting repeats) and then states the mechanism: grouping evaluations by case_key or session and reporting Wilson intervals per case plus an exact interval over cases. This is a specific verb+resource+method, clearly distinct from compare_runs (two runs against each other) and evaluate_output (score one output).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
An explicit 'When not to use it' section routes the agent to compare_runs for run-vs-run comparison and evaluate_output for scoring a single output, and the Siblings section reinforces the boundary with log_trace and get_traces. The condition that selects this tool over its alternatives is unambiguous.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Remove a deployed custom rule — or, with enabled, disable or re-enable it without removing it — effective on the next evaluate_output call.
What it does. Without enabled: deletes the rule and writes an audit entry; deleted is false when no rule has that id. With enabled: false stops it firing and keeps it stored, true re-enables it. Past evaluations are untouched. Full detail: iris://capabilities.
When not to use it. On built-in rules (they are not in the store). To replace a rule (deploy_rule with replace: true).
Returns. JSON: deleted, rule_id, toggled, enabled, rule.
Errors. IRIS_STORAGE_ERROR. A malformed rule_id is refused. Failures return {"error":{code,message,recovery}} with isError; follow recovery.
Siblings. deploy_rule — add or replace a rule; list_rules — find the id; evaluate_output — where the rule fires.
| Name | Required | Description | Default |
|---|---|---|---|
| enabled | No | When present the rule is kept: false disables it (stops firing), true re-enables it. Omit to delete | |
| rule_id | Yes | Rule id to delete or toggle (format: rule-<hex>); obtained from list_rules or deploy_rule response |
| Name | Required | Description |
|---|---|---|
| rule | No | |
| deleted | Yes | |
| enabled | No | |
| rule_id | Yes | |
| toggled | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Goes well beyond the destructive/not-idempotent annotations by disclosing that an audit entry is written, that 'deleted' is false when the id is absent, that past evaluations are untouched, and that the change applies on the next evaluate_output. Error codes and a recovery-following convention are also spelled out. The only shortfall is that reversibility of the delete itself (no restore path) is not explicitly stated.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loaded summary sentence captures the delete-or-toggle duality before any detail, and the labelled sections (What it does / When not to use it / Returns / Errors / Siblings) make it scannable. Slight redundancy in enumerating return fields that an output schema already covers.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a destructive two-parameter mutation tool, the description covers preconditions (custom rules only), side effects (audit entry, next-evaluation timing), failure modes and recovery. Return-value enumeration is redundant given an output schema, but that is harmless rather than a gap.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so both parameters are already documented, and the description largely restates them (omit 'enabled' to delete, false disables, true re-enables). It adds the small point that a malformed rule_id is refused, but supplies no format detail beyond the schema pattern.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource ('Remove a deployed custom rule') plus the alternate mode gated on 'enabled', so the agent immediately knows this tool both deletes and toggles. The sibling section names deploy_rule, list_rules and evaluate_output, making it distinguishable without opening their schemas.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
An explicit 'When not to use it' block excludes built-in rules and redirects replacement to deploy_rule with replace: true. The Siblings line further maps each alternative to its role (find the id, add/replace, where the rule fires), leaving nothing to inference.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Remove one stored trace by id; its spans go with it, and every evaluation linked to it keeps its verdict and loses its text.
What it does. Linked evaluations keep their verdicts and scores; their text is erased and erased_at is stamped. deleted is false when no trace has that id. Every deletion is audited at iris://audit. Full detail: iris://capabilities.
When not to use it. To expire data in bulk (retention.days). To change a trace: traces are immutable.
Returns. JSON: deleted, trace_id.
Errors. IRIS_STORAGE_ERROR. A malformed trace_id is refused. Failures return {"error":{code,message,recovery}} with isError; follow recovery.
Siblings. log_trace — store a trace; get_traces — find the trace to delete; delete_rule — the equivalent for custom rules.
| Name | Required | Description | Default |
|---|---|---|---|
| trace_id | Yes | Trace id to delete (32-hex lowercase; obtained from log_trace response or get_traces) |
| Name | Required | Description |
|---|---|---|
| deleted | Yes | |
| trace_id | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already flag destructiveHint=true, but the description supplies far more than the boolean: linked evaluations keep verdicts/scores while text is erased and erased_at stamped, spans are removed, `deleted: false` for unknown ids, auditing at iris://audit, and named error codes with a recovery protocol. This is precisely the cascade/permanence context an agent needs before an irreversible write.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Labeled sections (What it does / When not to use / Returns / Errors / Siblings) make it scannable and the destructive consequence is front-loaded in the opening line. Minor waste: a 'Returns' line restates what the existing output schema already declares, and the capabilities URL is filler for a single-parameter tool.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a destructive, non-idempotent single-param tool with an output schema, the description covers cascade effects, partial erasure semantics, audit trail, error taxonomy, and recovery — leaving no material gap an agent would need to fill before calling it.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Only one parameter and schema coverage is 100% — the schema already documents the 32-hex pattern and where to obtain the value. The description adds only the behavioral note that a malformed trace_id is refused, which is mildly useful but not additional semantic detail about the parameter itself. Baseline 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
First sentence gives a specific verb and resource ('Remove one stored trace by id') plus the non-obvious cascade ('its spans go with it') and the partial effect on linked evaluations. It is immediately distinguishable from log_trace and get_traces, which are named later.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Has an explicit 'When not to use it' section that redirects bulk expiry to retention.days and trace modification to the immutability constraint, and the Siblings block routes the agent to get_traces to find the id before deleting. Both when-to-use and when-not are covered.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Deploy a custom rule that fires on every future evaluate_output call of its bundle — persisted, active immediately, audited.
What it does. Stores the rule and loads it into the running engine. eval_type says WHEN it fires (that bundle, and eval_type="all"); severity says what a failure does: low and medium lower the score, high and critical fail the verdict. definition.type picks the check, definition.config its keys. A taken name is refused unless replace is true. Full detail: iris://capabilities.
When not to use it. For a one-off check (custom_rules on evaluate_output). To pause a rule (delete_rule with enabled: false).
Returns. JSON: rule, replaced, warning.
Errors. IRIS_DUPLICATE_RULE; IRIS_INVALID_RULE_CONFIG, naming the field; IRIS_STORAGE_ERROR. Failures return {"error":{code,message,recovery}} with isError; follow recovery.
Siblings. list_rules — see what is deployed; delete_rule — remove or disable; evaluate_output — where it fires.
| Name | Required | Description | Default |
|---|---|---|---|
| name | Yes | Rule name, 1-80 letters, digits, dots, dashes and underscores (used in eval results). Must be unique among deployed rules unless replace=true | |
| replace | No | true replaces a deployed rule of the same name (fresh id, audit kept); false (default) refuses | |
| evalType | No | camelCase alias of eval_type, accepted for compatibility — prefer eval_type (snake_case is canonical across the tools) | |
| severity | No | What a FAILURE of this rule means. low/medium: informational — contributes to the weighted score only (plus dashboard sort + audit alerts). high/critical: hard-fail — a failing evaluation of this rule forces the overall passed=false regardless of the weighted score | medium |
| eval_type | No | The bundle this rule fires in (and on eval_type="all"); pass one of eval_type / evalType | |
| definition | Yes | The check: type, config and optional weight; an unknown key is rejected | |
| description | No | What this rule checks for and why it matters | |
| sourceMomentId | No | camelCase alias of source_moment_id, accepted for compatibility — prefer source_moment_id | |
| source_moment_id | No | Optional id of the Decision Moment this rule came from; pass one of source_moment_id / sourceMomentId |
| Name | Required | Description |
|---|---|---|
| rule | Yes | |
| warning | No | |
| replaced | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations declare readOnlyHint=false, destructiveHint=false, and idempotentHint=false, but the description adds substantial value beyond them: the rule is persisted, active immediately, and audited; a taken name is refused unless replace is true; and the exact error codes (IRIS_DUPLICATE_RULE, IRIS_INVALID_RULE_CONFIG, IRIS_STORAGE_ERROR) plus the {error:{code,message,recovery}} envelope are disclosed. This is unusually complete behavioral disclosure for a mutation tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is front-loaded with the core behavior in the opening sentence, then segmented into 'What it does', 'When not to use it', 'Returns', 'Errors', and 'Siblings'. Despite its length, every section conveys actionable information with no filler, and an agent can skim the labeled sections quickly.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Even though an output schema exists, the 'Returns' section names the top-level keys (rule, replaced, warning), and the 'Errors' section covers failure recovery. Combined with sibling routing and detailed mutation semantics, the description provides everything an agent needs to invoke deploy_rule correctly and handle outcomes.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so baseline would be 3, and the description pushes past it by explaining the causal semantics of eval_type (WHEN it fires) and severity (low/medium lower score, high/critical hard-fail) in prose rather than merely restating enums. It stops short of documenting specific config keys per type, but the schema already does that richly.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The first sentence names a specific verb (deploy), resource (custom rule), and effect (fires on every future evaluate_output call of its bundle, persisted, active immediately, audited). A 'What it does' section further pins down eval_type, severity, and definition.type semantics, so an agent can distinguish it from siblings without opening schemas.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
There is an explicit 'When not to use it' section with alternatives named: use custom_rules on evaluate_output for a one-off check, and delete_rule with enabled: false to pause. The 'Siblings' section routes list_rules, delete_rule, and evaluate_output to their purposes, leaving nothing to inference.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Score an output with deterministic rules: a ship verdict with its basis, per-rule evidence, what was unjudged.
What it does. eval_type picks a bundle or all (the default). A rule missing its input SKIPS, never passes: input is REQUIRED when eval_type="relevance"; tool_calls, tools, cost_usd, expected feed the rest. trace_id alone scores the stored trace. Full detail: iris://capabilities.
When not to use it. For semantic judgment (evaluate_with_llm_judge). As an input firewall: the rules read the output.
Returns. JSON: id, trace_id, reference_trace_id, verdict, coverage, provenance, run_id, interpretations, erased_at, eval_type, score, passed, rule_results, rules_evaluated, rules_skipped, insufficient_data, critical_failures, critical_skipped, categories, note, rules_changed.
Errors. IRIS_UNKNOWN_TRACE (nothing written); IRIS_STORAGE_ERROR. Failures return {"error":{code,message,recovery}} with isError; follow recovery.
Siblings. log_trace — record the run first; list_rules — what each rule needs.
| Name | Required | Description | Default |
|---|---|---|---|
| input | No | The ask and any source material given — REQUIRED when eval_type="relevance"; also grounds the hallucination signals | |
| tools | No | Your MCP tools/list result, verbatim; lets the rules check call arguments. Loaded from trace_id when stored | |
| output | No | The output text to evaluate (the agent's response that gets scored against rules). Omit it with trace_id to score the trace's stored output | |
| cost_usd | No | Cost in USD, for the cost bundle and any cost_threshold rule; omitted, the stored cost of trace_id (reported or estimated), else they skip rather than pass | |
| expected | No | Expected output for comparison — consulted only by the completeness bundle's expected_coverage rule; NOT used by relevance (the relevance rules compare the output against `input`) | |
| trace_id | No | A stored trace (from log_trace / get_traces). Alone, it scores the trace as stored and the result is the trace's verdict. With output, input, tool_calls, tools, cost_usd or token_usage that differ from the record, or one eval_type, the result is kept beside the trace (reference_trace_id) and never replaces its verdict. An unknown id is rejected before anything is evaluated | |
| eval_type | No | Rule bundle: completeness | relevance | safety | cost | custom | all (the default, noted in the response) | |
| tool_calls | No | The tool calls the agent made, in order, as log_trace records them; omitted, the trajectory rules skip. Loaded from trace_id when stored | |
| token_usage | No | Token usage breakdown — only consulted by the cost bundle (eval_type="cost" or "all"; used for token-budget rules) | |
| custom_rules | No | Up to 10 one-off rules; they fire whatever eval_type is (eval_type="custom" runs only these) | |
| expected_trajectory | No | What the agent was expected to DO: tool_calls with mode and args (tool_sequence); step_budget and tolerance (step_budget) |
| Name | Required | Description |
|---|---|---|
| id | Yes | |
| note | No | |
| score | Yes | |
| passed | Yes | |
| run_id | No | |
| verdict | No | |
| coverage | No | |
| trace_id | No | |
| erased_at | No | |
| eval_type | Yes | |
| categories | No | |
| provenance | No | |
| rule_results | Yes | |
| rules_changed | No | |
| rules_skipped | Yes | |
| interpretations | No | |
| rules_evaluated | Yes | |
| critical_skipped | No | |
| critical_failures | No | |
| insufficient_data | Yes | |
| reference_trace_id | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Beyond annotations, it discloses important behavior: missing rule inputs skip rather than pass, input is required for relevance, trace_id alone scores the stored trace, and alternate inputs create a side result rather than replacing the trace verdict. It also names error codes and says failures return isError with a recovery path.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is well structured and front-loads core behavior, alternatives, returns, errors, and siblings. The long return-field list is somewhat redundant given the output schema, but it remains compact and scannable.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For an 11-parameter tool with nested objects, the description provides enough context on purpose, alternatives, required conditions, error handling, and outcome shape. The schema and output schema handle detailed field structure, while the description fills in operational semantics.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is already 100%, so the schema carries most parameter documentation. The description still adds meaningful cross-parameter semantics, especially the relevance input requirement, default eval_type behavior, skip behavior for omitted tool_calls or cost_usd, and how trace_id changes result persistence.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb and resource: score an output with deterministic rules, producing a ship verdict, per-rule evidence, and unjudged items. It clearly distinguishes deterministic rule evaluation from semantic judgment by naming evaluate_with_llm_judge as the alternative.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It gives explicit when-not-to-use guidance for semantic judgment and input-firewall use, and explains the eval_type bundle behavior, the relevance input requirement, and trace_id scoring semantics. It also routes the agent to log_trace and list_rules with concrete conditions.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Re-score every trace in a run under the current rules into a new run, so a rules change is compared, not overwritten.
What it does. Writes the verdicts into a NEW run; the source run is never modified. Traces already scored under the current ruleset are skipped, so a repeat call does no work. Local, no model call. Full detail: iris://capabilities.
When not to use it. To score a new execution (log_trace, then evaluate_output).
Returns. JSON: source_run, run, ruleset_hash, traces, evaluated, already_current, failed, passed, not_checked, summary, rules_changed.
Errors. IRIS_INVALID_ARGUMENT (empty run id, or the target run holds verdicts); IRIS_UNKNOWN_TRACE (source run empty); IRIS_STORAGE_ERROR. Unscorable traces are listed in failed. Failures return {"error":{code,message,recovery}} with isError; follow recovery.
Siblings. compare_runs — compare the two runs after; evaluate_output — re-score one trace; get_traces — read a run.
| Name | Required | Description | Default |
|---|---|---|---|
| run | Yes | the run to re-score — whatever you passed as `run` on log_trace | |
| into | No | Run id for the new verdicts; defaults to <run>+reeval-<ruleset hash>, stable across repeats | |
| label | No | a name for the new run, shown in listings |
| Name | Required | Description |
|---|---|---|
| run | Yes | |
| failed | Yes | |
| passed | Yes | |
| traces | Yes | |
| summary | Yes | |
| evaluated | Yes | |
| source_run | Yes | |
| not_checked | Yes | |
| ruleset_hash | Yes | |
| rules_changed | No | |
| already_current | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations declare write, idempotent, non-destructive; the description adds the important context beyond that: verdicts go to a NEW run, the source run is never modified, already-current traces are skipped so repeats are no-ops, and it is local with no model call. It also enumerates error codes and recovery behavior, which annotations cannot convey.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loaded with the core behavior, then sectioned into What it does / When not to use / Returns / Errors / Siblings. Despite its length, every line carries actionable information and is scannable rather than rambling.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Covers scope, safety, idempotency, error modes, and sibling routing. An output schema exists so the Returns list is a bonus rather than a requirement, and no behavior an agent needs to call this correctly is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% so the baseline would be 3, but the description adds real meaning: 'run' is whatever was passed to log_trace, and 'into' defaults to a stable <run>+reeval-<ruleset hash>. It stops short of explaining 'label' explicitly, so it is not a full 5.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource ('re-score every trace in a run under the current rules into a new run') and immediately clarifies the non-destructive semantics ('compared, not overwritten'). This distinguishes it cleanly from evaluate_output and compare_runs without opening any schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Has an explicit 'When not to use it' section pointing to the correct alternative (log_trace, then evaluate_output) and a 'Siblings' block that routes to compare_runs, evaluate_output, and get_traces. When-to-use, when-not, and alternatives are all present.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Score an output with an LLM judge on your own key: a 0..1 score, a rationale, the spend.
What it does. Calls Anthropic or OpenAI with IRIS_ANTHROPIC_API_KEY or IRIS_OPENAI_API_KEY; Iris never proxies. correctness needs expected, relevance input, faithfulness source_material. Worst-case cost is checked against max_cost_usd first. Full detail: iris://capabilities.
When not to use it. For length, keyword, PII or cost checks (evaluate_output, free).
Returns. JSON: id, reference_trace_id, score, passed, verdict, pass_threshold, self_reported_pass, disagreement, rationale, dimensions, model, provider, template, input_tokens, output_tokens, cost_usd, latency_ms, raw_response_id, warnings.
Errors. IRIS_JUDGE_NOT_ENABLED (recovery has the steps); IRIS_JUDGE_UNKNOWN_MODEL; IRIS_UNKNOWN_TRACE; IRIS_BUDGET_EXCEEDED; IRIS_PROVIDER_ERROR. Failures return {"error":{code,message,recovery}} with isError; follow recovery.
Siblings. evaluate_output — the free deterministic path; verify_citations — citation grounding.
| Name | Required | Description | Default |
|---|---|---|---|
| input | No | User question / prompt that produced the output (required for relevance; improves helpfulness and safety) | |
| model | Yes | Model ID. Supported: anthropic = claude-fable-5-1 | claude-opus-5-5 | claude-sonnet-5 (and the priced 4.x line); openai = gpt-5 | gpt-5-mini | gpt-4.1-mini (and the priced 4o, o3/o4 and retired o1 ids); an unknown id is refused with the full priced list. Required — cost varies a hundredfold across models | |
| output | Yes | The agent output text to evaluate | |
| expected | No | Reference answer (required for correctness template) | |
| provider | No | Auto-detected from model when omitted | |
| template | Yes | The question the judge answers; correctness needs expected, relevance needs input, faithfulness and task_completed read source_material | |
| trace_id | No | Keep this evaluation beside a stored trace (id from log_trace / get_traces). It is listed with the trace and never replaces its verdict; an unknown id is rejected BEFORE the judge is called | |
| timeout_ms | No | Per-request timeout; default 60_000 | |
| agent_model | No | The model that produced the output, for the same-family warning, when no linked trace records it | |
| temperature | No | Sampling temperature; default 0 (deterministic) | |
| max_cost_usd | No | Cost cap in USD for this call, at most the operator's IRIS_LLM_JUDGE_MAX_COST_USD_PER_EVAL (default 0.25). The worst case (two attempts, full max_output_tokens) is computed before the call and refused if it exceeds the cap | |
| source_material | No | Provided RAG sources (required for faithfulness template) | |
| max_output_tokens | No | Judge output token cap; default 512 |
| Name | Required | Description |
|---|---|---|
| id | Yes | |
| model | Yes | |
| score | Yes | |
| passed | Yes | |
| verdict | No | |
| cost_usd | Yes | |
| provider | Yes | |
| template | Yes | |
| warnings | No | |
| rationale | Yes | |
| dimensions | Yes | |
| latency_ms | Yes | |
| disagreement | No | |
| input_tokens | Yes | |
| output_tokens | Yes | |
| pass_threshold | Yes | |
| raw_response_id | No | |
| reference_trace_id | No | |
| self_reported_pass | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Goes well past the annotations (readOnly=false, openWorld=true, non-idempotent): BYO-key with no Iris proxy, the worst-case cost pre-check including 'two attempts, full max_output_tokens', enumerated error codes with a recovery contract, and confirmation that failures return isError. This is exactly the extra context the annotations cannot express.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loaded one-line summary followed by labeled sections (What it does / When not to use it / Returns / Errors / Siblings). Dense but every line carries decision-relevant content, including the pointer to iris://capabilities instead of expanding detail inline.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 13-parameter, open-world, cost-incurring tool the description covers prerequisites (API keys), safety rails (max_cost_usd worst-case), failure modes with recovery, and sibling routing. The Returns list is partly redundant with the output schema, but the Errors contract is not, and nothing an agent needs to call it safely is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the schema already carries parameter meaning; the description adds the cross-parameter couplings in one place (correctness needs expected, relevance needs input, faithfulness needs source_material) and flags that cost 'varies a hundredfold across models'. It does not add syntax or format detail beyond the schema, so it lands just above the baseline.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb+resource+scope: 'Score an output with an LLM judge on your own key', plus the exact outputs produced (0..1 score, rationale, spend). The 'Siblings' section explicitly separates it from evaluate_output (free deterministic) and verify_citations (citation grounding), so an agent can route without opening a schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
'When not to use it' names the exact alternative (evaluate_output) and the check types it covers (length, keyword, PII, cost). It also names the sibling relationships for verify_citations, giving both when-not and alternatives explicitly.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Query stored traces with filters, pagination and sorting; optionally with the dashboard summary.
What it does. Read-only, local. Exact-match agent_name and framework, inclusive since and until, min_score and max_score on each trace's latest evaluation, and q: full-text search, ranked, with a snippet. limit is 1..1000 (default 50). A crossed range is refused, never returned as an empty page. Full detail: iris://capabilities.
When not to use it. To score a trace (evaluate_output) or create one (log_trace). As a live stream: poll with backoff.
Returns. JSON: untrusted, traces, total, limit, offset, summary, search.
Errors. IRIS_STORAGE_ERROR. Out-of-range bounds are refused, naming the values; no match is total 0, not an error. Failures return {"error":{code,message,recovery}} with isError; follow recovery.
Siblings. log_trace — record an execution; evaluate_output — score one output; delete_trace — remove one trace.
| Name | Required | Description | Default |
|---|---|---|---|
| q | No | Full-text search over input, output, tool-call values, metadata values and span attribute and event values. Every word must appear; "quoted phrase"; word* for a prefix of 3+ characters (1 for CJK). Case and accents ignored. At most 16 terms, 4 of them prefixes | |
| limit | No | Results per page (default 50, max 1000 — values above are rejected) | |
| since | No | Inclusive lower bound, an ISO 8601 timestamp or date; anything else is rejected | |
| until | No | ISO 8601 timestamp (or date) upper bound — return traces with timestamp <= this; must not be earlier than `since` | |
| offset | No | Zero-based pagination offset — skip first N results (non-negative integer) | |
| session | No | Filter by session id — the turns of one conversation, as logged with session_id | |
| sort_by | No | Sort by timestamp | latency_ms | cost_usd | relevance (default relevance with q, else timestamp) | |
| framework | No | Filter by agent framework — exact match (e.g., langchain, autogen) | |
| max_score | No | Maximum eval score filter (0..1; values outside are rejected) — applied to LATEST eval per trace | |
| min_score | No | Minimum score (0..1) of each trace's latest evaluation; at most max_score | |
| agent_name | No | Filter by agent name — exact match (no wildcards) | |
| sort_order | No | Sort order: asc | desc (default desc — most recent / highest first) | desc |
| include_text | No | Return each stored text whole. By default each is cut to 500 characters, and the trace's cut lists what was cut with its full length; iris://traces/{trace_id} reads one trace whole | |
| include_summary | No | Include dashboard summary stats in same response — saves a round-trip when ingesting for dashboards |
| Name | Required | Description |
|---|---|---|
| limit | Yes | |
| total | Yes | |
| offset | Yes | |
| search | No | |
| traces | Yes | |
| summary | No | |
| untrusted | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already cover readOnly/idempotent/non-destructive, and the description still adds substantial context the annotations cannot: results are untrusted JSON, crossed ranges are refused rather than returned as an empty page, out-of-range bounds are refused by naming values, no-match is total 0 rather than an error, and errors carry {code,message,recovery}. Default text truncation at 500 chars with a pointer to iris://traces/{trace_id} is disclosed up front.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loaded one-line purpose followed by labeled sections (What it does / When not to use / Returns / Errors / Siblings), which makes scanning fast. Slight redundancy: the 'Siblings' section largely re-states the alternatives already given in 'When not to use it'.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 14-parameter query tool with an output schema, the description still covers the non-structural essentials: return envelope shape, error contract with recovery guidance, truncation behavior, and a pointer to iris://capabilities for full detail. Nothing an agent needs to invoke it correctly is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the schema already explains nearly every parameter in depth; the validation baseline is therefore 3. The description adds some meaning beyond it – ranked full-text search with a snippet, min/max_score scoped to each trace's latest evaluation, and include_summary framed as a round-trip saver – but most of its parameter statements restate the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Opens with a specific verb+resource+scope ('Query stored traces with filters, pagination and sorting; optionally with the dashboard summary') and the 'What it does' section enumerates exact filter semantics. It is unmistakably a filtered list/query tool and cannot be confused with log_trace or evaluate_output.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Has an explicit 'When not to use it' section naming the alternatives (evaluate_output to score, log_trace to create) and a directive not to treat it as a live stream ('poll with backoff'). The 'Siblings' section reinforces routing to log_trace, evaluate_output and delete_trace.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
The rule inventory: every built-in rule with what it needs, its effective criticality and published accuracy, plus every deployed custom rule.
What it does. Read-only, no network. built_in is never narrowed by the filters; a rule whose needs are absent skips, never passes. Read critical and criticalSource before trusting passed: true. rules is the custom-rule store, filterable by eval_type and enabled_only. Full detail: iris://capabilities.
When not to use it. To add, remove or pause a rule (deploy_rule, delete_rule); built-in rules cannot be.
Returns. JSON: untrusted, rules, total, enabled_count, built_in, quarantined, plugins.
Errors. IRIS_INTERNAL_ERROR if the store cannot be read; a missing store is an empty list. Failures return {"error":{code,message,recovery}} with isError; follow recovery.
Siblings. deploy_rule — add a custom rule; delete_rule — remove, disable or re-enable one; evaluate_output — run the rules.
| Name | Required | Description | Default |
|---|---|---|---|
| eval_type | No | Filter the custom rules to one eval category (exact match); built_in is never filtered | |
| enabled_only | No | Return only enabled custom rules (a rule disabled with delete_rule stays in the store and does not fire) |
| Name | Required | Description |
|---|---|---|
| rules | Yes | |
| total | Yes | |
| plugins | Yes | |
| built_in | Yes | |
| untrusted | No | |
| quarantined | Yes | |
| enabled_count | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnly/idempotent/destructive=false, and the description still adds genuinely non-obvious behavior: built_in is never narrowed by filters, rules whose needs are absent skip rather than pass, and agents must read critical/criticalSource before trusting passed:true. It also documents the error envelope and recovery path, which annotations cannot convey.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loads the inventory summary, then sections by What/When-not/Returns/Errors/Siblings, so an agent can scan to the part it needs. Slightly header-heavy and the Siblings list partially duplicates the tool roster, but no sentence is filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Despite an output schema existing, the description still names the top-level return keys and flags the list as untrusted, plus covers error codes and recovery. For a read-only dual-store listing tool, nothing an agent needs to call it or interpret results is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the baseline is 3, but the description adds real meaning: it clarifies that filters apply only to custom rules and never to built_in, and that a rule disabled via delete_rule persists in the store without firing. It stops short of restating the enum values, which is appropriate for a 100%-covered schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource ("The rule inventory") and enumerates exactly what is returned: built-in rules with needs/criticality/accuracy plus deployed custom rules. An agent can distinguish it from deploy_rule, delete_rule and evaluate_output without opening any schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Has an explicit "When not to use it" section routing to deploy_rule/delete_rule and noting built-ins cannot be modified, plus a Siblings line that spells out each alternative's role. The when/when-not/alternative triad is fully covered.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Store one agent execution (input, output, tool calls, spans, cost, tokens) and get the trace_id later calls key on.
What it does. Writes one immutable trace to local storage; nothing is deduplicated. Only agent_name is required; store tool_calls, tools, cost and input/output so the rules can judge them. evaluate: true (with output) also scores it in this call, exactly as evaluate_output would. Full detail: iris://capabilities.
When not to use it. To score a stored trace (evaluate_output with its trace_id). To change one (delete_trace, then log again).
Returns. JSON: trace_id, status, cost_usd, cost_source, cost_estimate, evaluation.
Errors. IRIS_STORAGE_ERROR; IRIS_INVALID_ARGUMENT when evaluate is true without output. Unknown or malformed arguments are refused, naming the valid keys. Failures return {"error":{code,message,recovery}} with isError; follow recovery.
Siblings. evaluate_output — score the stored output; get_traces — query what was logged; delete_trace — remove one trace.
| Name | Required | Description | Default |
|---|---|---|---|
| run | No | The batch this execution belongs to (a CI job id, a sweep); compare_runs compares two runs | |
| input | No | Agent input text — the user prompt or upstream input that produced this output | |
| spans | No | Detailed execution spans (hierarchical span tree with timings, attributes, events); a span without start_time takes the trace timestamp | |
| tools | No | Your MCP tools/list result, verbatim; lets the rules check call arguments later | |
| output | No | Agent output text — what the agent produced (pass to evaluate_output for scoring) | |
| case_key | No | What makes this the same question across runs (a fixture or test id); pairs traces. Derived from input when omitted | |
| cost_usd | No | Total cost in USD, stored as reported. Omitted: estimated from the token counts and the model (metadata.model or the spans) at list price, and marked estimated | |
| evaluate | No | Also score the stored trace in this call, exactly as evaluate_output would; requires output | |
| metadata | No | Opaque key-value tags (e.g. {requestId, userId, env}) — queryable in dashboard, not via get_traces filters | |
| eval_type | No | With evaluate: true, the bundle to run (default: every bundle, noted in the response) | |
| framework | No | Agent framework identifier (e.g., langchain, autogen, custom) | |
| timestamp | No | Trace timestamp (ISO 8601); defaults to now() when omitted | |
| agent_name | Yes | Agent name — used for filtering in get_traces (e.g., "customer-support-bot") | |
| latency_ms | No | Total execution time in milliseconds (end-to-end agent latency) | |
| session_id | No | The conversation this turn belongs to; groups its turns. Read from SEP-414 baggage when omitted | |
| tool_calls | No | Tool calls made, in order, each { tool_name, input?, output?, latency_ms?, error? }; the trajectory rules judge them | |
| token_usage | No | Token usage (prompt/completion/total, and cache_read_tokens / cache_creation_tokens: the cached part of prompt_tokens) — prices the trace when cost_usd is omitted |
| Name | Required | Description |
|---|---|---|
| status | Yes | |
| cost_usd | Yes | |
| trace_id | Yes | |
| evaluation | No | |
| cost_source | No | |
| cost_estimate | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The write semantics are clearly disclosed as an immutable local-storage write with no deduplication, consistent with the non-readOnly, non-idempotent annotations. It adds error codes, failure shape, recovery guidance, and the fact that evaluate=true triggers scoring in the same call, which is valuable context beyond the annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is front-loaded with the core action and trace_id return, then uses short labeled sections for behavior, exclusions, returns, errors, and siblings. For a 17-parameter tool it is compact and every section provides actionable information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
It covers the essential behavioral traits for a complex write tool: required minimal input, immutability, no deduplication, inline evaluation behavior, error handling, and alternatives. With an output schema and full schema descriptions also present, the description is complete enough for correct invocation.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the schema already documents the 17 parameters thoroughly. The description adds useful high-level guidance that only agent_name is required and that tool_calls, tools, input/output, and cost should be stored for rule evaluation, but it does not add detailed semantics beyond the schema for most parameters.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a precise verb and resource: store one agent execution trace and return a trace_id. It also explicitly distinguishes this from evaluate_output, get_traces, and delete_trace, so an agent can tell what this tool is for without opening the schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It gives explicit when-not-to-use guidance: use evaluate_output to score a stored trace, and delete_trace plus relog to change one. It also explains the inline evaluate=true alternative and names the sibling query tool, leaving little to inference.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Is each citation in an output supported by its source? Extracts, fetches (opt-in, SSRF-guarded) and judges on your key.
What it does. Fetches only if the operator set IRIS_CITATION_ALLOW_FETCH=1, blocking private addresses. One judge call per resolved citation, within max_cost_usd_total. overall_score: supported / judged; null if none judged. Full detail: iris://capabilities.
When not to use it. With no citations (evaluate_output). Without IRIS_ANTHROPIC_API_KEY or IRIS_OPENAI_API_KEY.
Returns. JSON: id, reference_trace_id, overall_score, passed, verdict, total_unsupported, total_citations_found, total_resolved, total_judged, total_supported, total_cost_usd, warnings, citations.
Errors. IRIS_JUDGE_NOT_ENABLED, IRIS_JUDGE_UNKNOWN_MODEL, IRIS_UNKNOWN_TRACE (before any spend); IRIS_JUDGE_FAILED when every judge call failed. Failures return {"error":{code,message,recovery}} with isError; follow recovery.
Siblings. evaluate_with_llm_judge — semantic scoring; evaluate_output — the free deterministic path.
| Name | Required | Description | Default |
|---|---|---|---|
| model | Yes | Judge model for per-citation verification. Supported: anthropic = claude-fable-5-1 | claude-opus-5-5 | claude-sonnet-5 (and the priced 4.x line); openai = gpt-5 | gpt-5-mini | gpt-4.1-mini (and the priced 4o, o3/o4 and retired o1 ids); an unknown id is refused with the full priced list. | |
| output | Yes | The agent output containing citations to verify | |
| provider | No | Auto-detected from model when omitted | |
| trace_id | No | Keep the verification result beside a stored trace (id from log_trace / get_traces). It is listed with the trace and never replaces its verdict; an unknown id is rejected before any fetch or judge call | |
| allow_fetch | No | false skips fetching for this call. Fetching happens only when the operator sets IRIS_CITATION_ALLOW_FETCH=1; true cannot turn it on. SSRF-guarded regardless. | |
| max_citations | No | Max citations to verify (extras skipped, not errored); default 20, at most 50 | |
| domain_allowlist | No | Fetch only from these hostnames (suffix match). Narrows IRIS_CITATION_DOMAINS when the operator set it, never adds to it. | |
| max_cost_usd_total | No | Cap TOTAL judge cost across all citations in this call, at most the operator's IRIS_CITATION_MAX_COST_USD_TOTAL (default 1.00 USD); the pipeline stops when the next call would exceed it | |
| per_source_max_bytes | No | Per-URL body cap; default 5MB | |
| per_source_timeout_ms | No | Per-URL fetch timeout; default 10_000 |
| Name | Required | Description |
|---|---|---|
| id | Yes | |
| passed | Yes | |
| verdict | No | |
| warnings | No | |
| citations | Yes | |
| total_judged | Yes | |
| overall_score | Yes | |
| total_cost_usd | Yes | |
| total_resolved | Yes | |
| total_supported | Yes | |
| total_unsupported | Yes | |
| reference_trace_id | No | |
| total_citations_found | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Beyond the annotations (openWorldHint, readOnlyHint=false), it discloses that fetching is gated by IRIS_CITATION_ALLOW_FETCH and SSRF-guarded against private addresses, that cost is bounded per call, how overall_score is computed (supported/judged, null when nothing judged), and a full error-code taxonomy with 'rejected before any spend' semantics. It also clarifies that trace_id never overwrites a stored verdict.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loaded core question, then labeled sections (What it does / When not to use it / Returns / Errors / Siblings). Dense but every line carries operational information; the pointer to iris://capabilities keeps detail out of the prompt without losing it.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 10-parameter, networked, cost-incurring tool, the description covers gating preconditions, cost bounds, error handling with recovery, and result interpretation. An output schema exists, and the brief field list is a bonus rather than a substitute for documentation.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so baseline is 3; the description adds cross-parameter behavior not obvious from individual fields — one judge call per resolved citation within max_cost_usd_total, and fetch opt-in that a per-call flag cannot override. The remaining parameters (timeouts, byte caps, allowlist narrowing) are left to schema text rather than elaborated further.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Opens with a precise question ('Is each citation in an output supported by its source?') followed by the verb+resource+mechanism: extract, fetch, judge per citation. It names the two closest siblings and states how they differ (semantic scoring vs. the free deterministic path), so an agent can disambiguate without opening any schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
A dedicated 'When not to use it' section gives explicit exclusions with routing: no citations -> evaluate_output; no provider API key -> do not call. Combined with the 'Siblings' line, both when-to-use and when-not are covered.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
v0.10.1compare_runscompare_tracesdelete_rule6 fields changedInput schema / properties / enabled / descriptionPrevious value: -"When present the rule is NOT deleted: false DISABLES it (kept in the store, stops firing immediately, history and provenance preserved); true RE-ENABLES a disabled rule. Omit to delete"New value: +"When present the rule is kept: false disables it (stops firing), true re-enables it. Omit to delete"Output schema / properties / deleted / descriptionRemoved value: -"true when a rule was removed; always false on a toggle"Output schema / properties / enabled / descriptionRemoved value: -"toggle only: the rule's state after the call"Output schema / properties / rule / descriptionRemoved value: -"toggle only: the rule as stored"Output schema / properties / rule_id / descriptionRemoved value: -"the id that was asked for"Output schema / properties / toggled / descriptionRemoved value: -"toggle only: true when the rule exists (also when it was already in the requested state)"delete_trace2 fields changedOutput schema / properties / deleted / descriptionRemoved value: -"true when a trace row was removed; false when no trace with that id existed for this tenant"Output schema / properties / trace_id / descriptionRemoved value: -"the id that was asked for"deploy_rule19 fields changedInput schema / properties / definition / descriptionPrevious value: -"Check definition (regex, length, keyword, cost, or schema). Accepts exactly type, config, weight and an optional name — an unknown key is rejected"New value: +"The check: type, config and optional weight; an unknown key is rejected"Input schema / properties / definition / properties / config / descriptionPrevious value: -"Check configuration; required keys depend on type (regex_match: pattern; min_length: min_length; max_length: max_length; contains_keywords/excludes_keywords: keywords; cost_threshold: max_cost; json_schema: none)"New value: +"Check configuration; required keys depend on type (regex_match: pattern; min_length: min_length; max_length: max_length; contains_keywords/excludes_keywords: keywords; cost_threshold: max_cost; json_schema: schema, optional; action_policy: allow and/or deny)"Input schema / properties / definition / properties / name / descriptionPrevious value: -"Optional and IGNORED if given — the server overwrites it with the top-level `name` so the rule reports under one name everywhere"New value: +"Ignored: the server overwrites it with the top-level `name`"Input schema / properties / definition / properties / type / enumPrevious value: -[
- "regex_match",
- "regex_no_match",
- "min_length",
- "max_length",
- "contains_keywords",
- "excludes_keywords",
- "json_schema",
- "cost_threshold"
-]New value: +[
+ "regex_match",
+ "regex_no_match",
+ "min_length",
+ "max_length",
+ "contains_keywords",
+ "excludes_keywords",
+ "json_schema",
+ "cost_threshold",
+ "action_policy"
+]Input schema / properties / eval_type / descriptionPrevious value: -"Eval category this rule belongs to; the rule fires on evaluate_output calls whose eval_type equals it (and on eval_type=\"all\"). Canonical snake_case spelling — pass exactly one of eval_type / evalType"New value: +"The bundle this rule fires in (and on eval_type=\"all\"); pass one of eval_type / evalType"Input schema / properties / name / descriptionPrevious value: -"Human-readable rule name (1-80 chars; used in eval results). Must be unique among deployed rules unless replace=true"New value: +"Rule name, 1-80 letters, digits, dots, dashes and underscores (used in eval results). Must be unique among deployed rules unless replace=true"Input schema / properties / name / patternAdded value: +"^[a-z0-9._-]+$"Input schema / properties / replace / descriptionPrevious value: -"When a rule with this name is already deployed: false (default) rejects the call; true deletes the existing same-named rule(s) and deploys this one in their place (fresh id; audit rows preserved)"New value: +"true replaces a deployed rule of the same name (fresh id, audit kept); false (default) refuses"Input schema / properties / source_moment_id / descriptionPrevious value: -"Optional Decision Moment ID the rule was derived from (preserves workflow-inversion provenance). Canonical snake_case — pass exactly one of source_moment_id / sourceMomentId"New value: +"Optional id of the Decision Moment this rule came from; pass one of source_moment_id / sourceMomentId"Output schema / properties / replaced / descriptionRemoved value: -"with replace: true, the earlier rule(s) of the same name that were retired"Output schema / properties / replaced / items / additionalPropertiesRemoved value: -{}Output schema / properties / replaced / items / propertiesRemoved value: -{
- "id": {
- "type": "string"
- }
-}Output schema / properties / replaced / items / requiredRemoved value: -[
- "id"
-]Output schema / properties / replaced / items / typeRemoved value: -"object"Output schema / properties / rule / descriptionRemoved value: -"the rule as persisted: id (rule-<hex>, keep it for delete_rule), name, description, evalType, severity, definition, enabled, createdAt, updatedAt, version, sourceMomentId"Output schema / properties / rule / properties / idRemoved value: -{
- "type": "string"
-}Output schema / properties / rule / properties / nameRemoved value: -{
- "type": "string"
-}Output schema / properties / rule / requiredRemoved value: -[
- "id",
- "name"
-]Output schema / properties / warning / descriptionRemoved value: -"with replace: true, one sentence naming what was retired"evaluate_output74 fields changedInput schema / properties / cost_usd / descriptionPrevious value: -"Cost in USD — consulted by the cost bundle (eval_type=\"cost\" or \"all\") AND by any cost_threshold custom rule regardless of eval_type; omit it and such a rule skips rather than passes (a critical one is listed in critical_skipped)"New value: +"Cost in USD, for the cost bundle and any cost_threshold rule; omitted, the stored cost of trace_id (reported or estimated), else they skip rather than pass"Input schema / properties / custom_rules / descriptionPrevious value: -"Custom evaluation rules, max 10 per call (deploy persistent rule sets via deploy_rule instead) — fires REGARDLESS of eval_type; pass eval_type=\"custom\" if you want ONLY these. Each entry accepts exactly name, type, config, weight — an unknown key (e.g. a misspelled weight) is rejected"New value: +"Up to 10 one-off rules; they fire whatever eval_type is (eval_type=\"custom\" runs only these)"Input schema / properties / custom_rules / items / properties / severityAdded value: +{
+ "description": "high or critical: a failure fails the verdict; otherwise it advises",
+ "enum": [
+ "low",
+ "medium",
+ "high",
+ "critical"
+ ],
+ "type": "string"
+}Input schema / properties / custom_rules / items / properties / type / enumPrevious value: -[
- "regex_match",
- "regex_no_match",
- "min_length",
- "max_length",
- "contains_keywords",
- "excludes_keywords",
- "json_schema",
- "cost_threshold"
-]New value: +[
+ "regex_match",
+ "regex_no_match",
+ "min_length",
+ "max_length",
+ "contains_keywords",
+ "excludes_keywords",
+ "json_schema",
+ "cost_threshold",
+ "action_policy"
+]Input schema / properties / eval_type / descriptionPrevious value: -"Rule bundle to apply: completeness | relevance | safety | cost | custom | all — picks which built-in rules fire. \"all\" runs every bundle in one call and adds a per-category breakdown. Defaults to \"all\" when omitted — every bundle runs, safety included, and the response carries a note saying the default ran"New value: +"Rule bundle: completeness | relevance | safety | cost | custom | all (the default, noted in the response)"Input schema / properties / expected_trajectoryAdded value: +{
+ "additionalProperties": false,
+ "description": "What the agent was expected to DO: tool_calls with mode and args (tool_sequence); step_budget and tolerance (step_budget)",
+ "properties": {
+ "args": {
+ "enum": [
+ "exact",
+ "subset"
+ ],
+ "type": "string"
+ },
+ "mode": {
+ "enum": [
+ "strict",
+ "unordered",
+ "subset",
+ "superset",
+ "ordered_subset"
+ ],
+ "type": "string"
+ },
+ "step_budget": {
+ "maximum": 9007199254740991,
+ "minimum": 1,
+ "type": "integer"
+ },
+ "tolerance": {
+ "minimum": 1,
+ "type": "number"
+ },
+ "tool_calls": {
+ "items": {
+ "additionalProperties": false,
+ "properties": {
+ "input": {},
+ "tool_name": {
+ "type": "string"
+ }
+ },
+ "required": [
+ "tool_name"
+ ],
+ "type": "object"
+ },
+ "maxItems": 500,
+ "type": "array"
+ }
+ },
+ "type": "object"
+}Input schema / properties / input / descriptionPrevious value: -"Original input for context (the ask + any source material the agent was given) — REQUIRED when eval_type=\"relevance\" (keyword_overlap and topic_consistency compare the output against it and skip without it); also grounds the safety bundle's hallucination signals"New value: +"The ask and any source material given — REQUIRED when eval_type=\"relevance\"; also grounds the hallucination signals"Input schema / properties / output / descriptionPrevious value: -"The output text to evaluate (the agent's response that gets scored against rules)"New value: +"The output text to evaluate (the agent's response that gets scored against rules). Omit it with trace_id to score the trace's stored output"Input schema / properties / tool_calls / descriptionPrevious value: -"What the agent DID — the tool calls it made, in order, each { tool_name, input?, output?, latency_ms?, error? } exactly as log_trace records them. Read by the trajectory rules — the rules that judge what the agent DID rather than what it wrote. Omit it and those rules SKIP rather than pass — an evaluation with no trajectory data reports \"not judged\", never \"clean\". When trace_id names a stored trace and this argument is omitted, the tool_calls stored on that trace are loaded and used, so a caller who already logged them need not resend them"New value: +"The tool calls the agent made, in order, as log_trace records them; omitted, the trajectory rules skip. Loaded from trace_id when stored"Input schema / properties / tool_calls / items / properties / call_idAdded value: +{
+ "type": "string"
+}Input schema / properties / tool_calls / items / properties / cost_usdAdded value: +{
+ "type": "number"
+}Input schema / properties / tool_calls / items / properties / token_usageAdded value: +{
+ "properties": {
+ "completion_tokens": {
+ "type": "number"
+ },
+ "prompt_tokens": {
+ "type": "number"
+ },
+ "total_tokens": {
+ "type": "number"
+ }
+ },
+ "type": "object"
+}Input schema / properties / tool_calls / items / properties / truncatedAdded value: +{
+ "type": "boolean"
+}Input schema / properties / toolsAdded value: +{
+ "description": "Your MCP tools/list result, verbatim; lets the rules check call arguments. Loaded from trace_id when stored",
+ "items": {
+ "additionalProperties": false,
+ "properties": {
+ "_meta": {
+ "additionalProperties": {},
+ "propertyNames": {
+ "type": "string"
+ },
+ "type": "object"
+ },
+ "annotations": {
+ "additionalProperties": {},
+ "properties": {
+ "destructiveHint": {
+ "type": "boolean"
+ },
+ "idempotentHint": {
+ "type": "boolean"
+ },
+ "openWorldHint": {
+ "type": "boolean"
+ },
+ "readOnlyHint": {
+ "type": "boolean"
+ },
+ "title": {
+ "type": "string"
+ }
+ },
+ "type": "object"
+ },
+ "description": {
+ "maxLength": 4096,
+ "type": "string"
+ },
+ "execution": {
+ "additionalProperties": {},
+ "propertyNames": {
+ "type": "string"
+ },
+ "type": "object"
+ },
+ "inputSchema": {
+ "additionalProperties": {},
+ "propertyNames": {
+ "type": "string"
+ },
+ "type": "object"
+ },
+ "name": {
+ "maxLength": 128,
+ "minLength": 1,
+ "type": "string"
+ },
+ "outputSchema": {
+ "additionalProperties": {},
+ "propertyNames": {
+ "type": "string"
+ },
+ "type": "object"
+ },
+ "title": {
+ "maxLength": 512,
+ "type": "string"
+ }
+ },
+ "required": [
+ "name"
+ ],
+ "type": "object"
+ },
+ "maxItems": 200,
+ "type": "array"
+}Input schema / properties / trace_id / descriptionPrevious value: -"Link evaluation to a trace — surfaces this eval in the dashboard's trace drill-through and lets the tool reuse the trace's stored tool_calls. Must be the id of a stored trace (from log_trace / get_traces); an unknown id is rejected before anything is evaluated"New value: +"A stored trace (from log_trace / get_traces). Alone, it scores the trace as stored and the result is the trace's verdict. With output, input, tool_calls, tools, cost_usd or token_usage that differ from the record, or one eval_type, the result is kept beside the trace (reference_trace_id) and never replaces its verdict. An unknown id is rejected before anything is evaluated"Input schema / requiredRemoved value: -[
- "output"
-]Output schema / descriptionAdded value: +"Nested evaluation fields: https://iris-eval.com/response-schema-v1.json"Output schema / properties / categories / additionalProperties / additionalPropertiesRemoved value: -{}Output schema / properties / categories / additionalProperties / propertiesRemoved value: -{
- "critical_failures": {
- "items": {
- "type": "string"
- },
- "type": "array"
- },
- "critical_skipped": {
- "items": {
- "type": "string"
- },
- "type": "array"
- },
- "insufficient_data": {
- "type": "boolean"
- },
- "passed": {
- "type": [
- "boolean",
- "null"
- ]
- },
- "rules_evaluated": {
- "maximum": 9007199254740991,
- "minimum": -9007199254740991,
- "type": "integer"
- },
- "rules_skipped": {
- "maximum": 9007199254740991,
- "minimum": -9007199254740991,
- "type": "integer"
- },
- "score": {
- "type": [
- "number",
- "null"
- ]
- }
-}Output schema / properties / categories / additionalProperties / requiredRemoved value: -[
- "score",
- "passed",
- "rules_evaluated",
- "rules_skipped",
- "insufficient_data"
-]Output schema / properties / categories / additionalProperties / typeRemoved value: -"object"Output schema / properties / categories / descriptionRemoved value: -"per-bundle verdicts for eval_type all"Output schema / properties / coverage / descriptionRemoved value: -"per question: judged, unjudged and why, or not_applicable; plus the inputs carried"Output schema / properties / coverage / properties / dormantRemoved value: -{
- "items": {
- "additionalProperties": {},
- "properties": {
- "name": {
- "type": "string"
- },
- "reason": {
- "type": "string"
- },
- "ruleId": {
- "type": "string"
- }
- },
- "required": [
- "ruleId",
- "name",
- "reason"
- ],
- "type": "object"
- },
- "type": "array"
-}Output schema / properties / coverage / properties / inputsRemoved value: -{
- "additionalProperties": {
- "type": "boolean"
- },
- "propertyNames": {
- "type": "string"
- },
- "type": "object"
-}Output schema / properties / coverage / properties / questionsRemoved value: -{
- "items": {
- "additionalProperties": {},
- "properties": {
- "id": {
- "enum": [
- "safe_output",
- "grounded",
- "complete",
- "relevant",
- "task_completed",
- "tool_use_correct",
- "within_budget"
- ],
- "type": "string"
- },
- "status": {
- "enum": [
- "judged",
- "unjudged",
- "not_applicable"
- ],
- "type": "string"
- },
- "why": {
- "type": "string"
- }
- },
- "required": [
- "id",
- "status"
- ],
- "type": "object"
- },
- "type": "array"
-}Output schema / properties / coverage / requiredRemoved value: -[
- "inputs",
- "questions"
-]Output schema / properties / critical_failures / descriptionRemoved value: -"critical rules that failed and vetoed passed"Output schema / properties / critical_failures / items / typeRemoved value: -"string"Output schema / properties / critical_skipped / descriptionRemoved value: -"critical rules that could not judge; treat as unknown"Output schema / properties / critical_skipped / items / typeRemoved value: -"string"Output schema / properties / erased_at / descriptionRemoved value: -"set once the linked trace was deleted"Output schema / properties / eval_type / descriptionRemoved value: -"the bundle that ran"Output schema / properties / id / descriptionRemoved value: -"the evaluation id, readable at iris://evaluations/{id}"Output schema / properties / insufficient_data / descriptionRemoved value: -"true when no rule could judge"Output schema / properties / interpretationsAdded value: +{
+ "items": {},
+ "type": "array"
+}Output schema / properties / note / descriptionRemoved value: -"present when eval_type was omitted"Output schema / properties / passed / descriptionRemoved value: -"the ship verdict; false when nothing was judged"Output schema / properties / provenance / descriptionRemoved value: -"Iris version, ruleset and config hashes, thresholds, corpus version, time"Output schema / properties / provenance / properties / configHashRemoved value: -{
- "type": "string"
-}Output schema / properties / provenance / properties / corpusVersionRemoved value: -{
- "type": "string"
-}Output schema / properties / provenance / properties / irisVersionRemoved value: -{
- "type": "string"
-}Output schema / properties / provenance / properties / judgedAtRemoved value: -{
- "type": "string"
-}Output schema / properties / provenance / properties / rulesetHashRemoved value: -{
- "type": "string"
-}Output schema / properties / provenance / properties / thresholdsRemoved value: -{
- "additionalProperties": {},
- "properties": {
- "default": {
- "type": "number"
- },
- "perRule": {
- "additionalProperties": {},
- "propertyNames": {
- "type": "string"
- },
- "type": "object"
- }
- },
- "required": [
- "default"
- ],
- "type": "object"
-}Output schema / properties / provenance / requiredRemoved value: -[
- "irisVersion",
- "rulesetHash",
- "configHash",
- "thresholds",
- "corpusVersion",
- "judgedAt"
-]Output schema / properties / reference_trace_idAdded value: +{
+ "type": "string"
+}Output schema / properties / rule_results / descriptionRemoved value: -"per rule: verdict, message, kind, role, question, saw, evidence, uncertainty"Output schema / properties / rule_results / items / additionalPropertiesRemoved value: -{}Output schema / properties / rule_results / items / propertiesRemoved value: -{
- "budgetExceeded": {
- "type": "boolean"
- },
- "category": {
- "enum": [
- "completeness",
- "relevance",
- "safety",
- "cost",
- "custom"
- ],
- "type": "string"
- },
- "classes": {
- "items": {
- "type": "string"
- },
- "type": "array"
- },
- "configInvalid": {
- "type": "boolean"
- },
- "critical": {
- "type": "boolean"
- },
- "criticalSource": {
- "enum": [
- "default",
- "config"
- ],
- "type": "string"
- },
- "evidence": {
- "items": {
- "oneOf": [
- {
- "additionalProperties": {},
- "properties": {
- "end": {
- "maximum": 9007199254740991,
- "minimum": 0,
- "type": "integer"
- },
- "label": {
- "type": "string"
- },
- "source": {
- "type": "string"
- },
- "start": {
- "maximum": 9007199254740991,
- "minimum": 0,
- "type": "integer"
- },
- "type": {
- "const": "span",
- "type": "string"
- }
- },
- "required": [
- "type",
- "source",
- "start",
- "end",
- "label"
- ],
- "type": "object"
- },
- {
- "additionalProperties": {},
- "properties": {
- "count": {
- "maximum": 9007199254740991,
- "minimum": 0,
- "type": "integer"
- },
- "name": {
- "type": "string"
- },
- "type": {
- "const": "pattern",
- "type": "string"
- }
- },
- "required": [
- "type",
- "name",
- "count"
- ],
- "type": "object"
- },
- {
- "additionalProperties": {},
- "properties": {
- "index": {
- "maximum": 9007199254740991,
- "minimum": 0,
- "type": "integer"
- },
- "label": {
- "type": "string"
- },
- "toolName": {
- "type": "string"
- },
- "type": {
- "const": "toolCall",
- "type": "string"
- }
- },
- "required": [
- "type",
- "index",
- "toolName",
- "label"
- ],
- "type": "object"
- },
- {
- "additionalProperties": {},
- "properties": {
- "status": {
- "enum": [
- "resolved",
- "dead",
- "unverifiable",
- "supported",
- "unsupported"
- ],
- "type": "string"
- },
- "type": {
- "const": "citation",
- "type": "string"
- },
- "url": {
- "type": "string"
- }
- },
- "required": [
- "type",
- "url",
- "status"
- ],
- "type": "object"
- },
- {
- "additionalProperties": {},
- "properties": {
- "stat": {
- "type": "string"
- },
- "threshold": {
- "type": "number"
- },
- "thresholdSource": {
- "enum": [
- "default",
- "config",
- "call",
- "rule"
- ],
- "type": "string"
- },
- "type": {
- "const": "count",
- "type": "string"
- },
- "unit": {
- "type": "string"
- },
- "value": {
- "type": "number"
- }
- },
- "required": [
- "type",
- "stat",
- "unit",
- "value"
- ],
- "type": "object"
- }
- ]
- },
- "type": "array"
- },
- "kind": {
- "enum": [
- "measurement",
- "detection",
- "inference",
- "judgment",
- "policy",
- "verification"
- ],
- "type": "string"
- },
- "message": {
- "type": "string"
- },
- "passed": {
- "type": "boolean"
- },
- "question": {
- "enum": [
- "safe_output",
- "grounded",
- "complete",
- "relevant",
- "task_completed",
- "tool_use_correct",
- "within_budget"
- ],
- "type": "string"
- },
- "role": {
- "enum": [
- "gate",
- "veto",
- "risk",
- "advisory",
- "term"
- ],
- "type": "string"
- },
- "ruleId": {
- "type": "string"
- },
- "ruleName": {
- "type": "string"
- },
- "ruleVersion": {
- "maximum": 9007199254740991,
- "minimum": -9007199254740991,
- "type": "integer"
- },
- "saw": {
- "items": {
- "enum": [
- "output",
- "input",
- "expected",
- "tool_calls",
- "tool_outputs",
- "tools_catalogue",
- "cost",
- "tokens",
- "citations"
- ],
- "type": "string"
- },
- "type": "array"
- },
- "score": {
- "type": "number"
- },
- "skipClass": {
- "enum": [
- "not_applicable",
- "defeated",
- "config_invalid"
- ],
- "type": "string"
- },
- "skipReason": {
- "type": "string"
- },
- "skipped": {
- "type": "boolean"
- },
- "uncertainty": {
- "oneOf": [
- {
- "additionalProperties": {},
- "properties": {
- "basis": {
- "const": "published_accuracy",
- "type": "string"
- },
- "corpus": {
- "additionalProperties": {},
- "properties": {
- "fn": {
- "maximum": 9007199254740991,
- "minimum": -9007199254740991,
- "type": "integer"
- },
- "fp": {
- "maximum": 9007199254740991,
- "minimum": -9007199254740991,
- "type": "integer"
- },
- "labelling": {
- "enum": [
- "same-model",
- "human-verified"
- ],
- "type": "string"
- },
- "n": {
- "maximum": 9007199254740991,
- "minimum": -9007199254740991,
- "type": "integer"
- },
- "release": {
- "type": "string"
- },
- "tn": {
- "maximum": 9007199254740991,
- "minimum": -9007199254740991,
- "type": "integer"
- },
- "tp": {
- "maximum": 9007199254740991,
- "minimum": -9007199254740991,
- "type": "integer"
- },
- "version": {
- "type": "string"
- }
- },
- "required": [
- "n",
- "tp",
- "fp",
- "fn",
- "tn",
- "version",
- "release",
- "labelling"
- ],
- "type": "object"
- },
- "fired": {
- "type": "boolean"
- },
- "missRate": {
- "additionalProperties": {},
- "properties": {
- "hi": {
- "type": "number"
- },
- "lo": {
- "type": "number"
- },
- "point": {
- "type": "number"
- }
- },
- "required": [
- "point",
- "lo",
- "hi"
- ],
- "type": "object"
- },
- "ppv": {
- "additionalProperties": {},
- "properties": {
- "hi": {
- "type": "number"
- },
- "lo": {
- "type": "number"
- },
- "point": {
- "type": "number"
- }
- },
- "required": [
- "point",
- "lo",
- "hi"
- ],
- "type": "object"
- },
- "prior": {
- "additionalProperties": {},
- "properties": {
- "pi": {
- "type": "number"
- },
- "source": {
- "enum": [
- "default",
- "config",
- "estimated"
- ],
- "type": "string"
- }
- },
- "required": [
- "pi",
- "source"
- ],
- "type": "object"
- }
- },
- "required": [
- "basis",
- "fired",
- "prior",
- "corpus"
- ],
- "type": "object"
- },
- {
- "additionalProperties": {},
- "properties": {
- "basis": {
- "const": "definition",
- "type": "string"
- },
- "conformance": {
- "additionalProperties": {},
- "properties": {
- "matched": {
- "maximum": 9007199254740991,
- "minimum": -9007199254740991,
- "type": "integer"
- },
- "n": {
- "maximum": 9007199254740991,
- "minimum": -9007199254740991,
- "type": "integer"
- }
- },
- "required": [
- "n",
- "matched"
- ],
- "type": "object"
- }
- },
- "required": [
- "basis",
- "conformance"
- ],
- "type": "object"
- },
- {
- "additionalProperties": {},
- "properties": {
- "basis": {
- "const": "self_consistency",
- "type": "string"
- },
- "samples": {
- "maximum": 9007199254740991,
- "minimum": -9007199254740991,
- "type": "integer"
- },
- "scoreSd": {
- "type": "number"
- },
- "voteFraction": {
- "type": "number"
- }
- },
- "required": [
- "basis",
- "samples",
- "voteFraction",
- "scoreSd"
- ],
- "type": "object"
- },
- {
- "additionalProperties": {},
- "properties": {
- "basis": {
- "const": "local_labels",
- "type": "string"
- },
- "n": {
- "maximum": 9007199254740991,
- "minimum": -9007199254740991,
- "type": "integer"
- },
- "precision": {
- "additionalProperties": {},
- "properties": {
- "hi": {
- "type": "number"
- },
- "lo": {
- "type": "number"
- },
- "point": {
- "type": "number"
- }
- },
- "required": [
- "point",
- "lo",
- "hi"
- ],
- "type": "object"
- }
- },
- "required": [
- "basis",
- "precision",
- "n"
- ],
- "type": "object"
- },
- {
- "additionalProperties": {},
- "properties": {
- "basis": {
- "const": "policy",
- "type": "string"
- }
- },
- "required": [
- "basis"
- ],
- "type": "object"
- },
- {
- "additionalProperties": {},
- "properties": {
- "basis": {
- "const": "unmeasured",
- "type": "string"
- },
- "why": {
- "type": "string"
- }
- },
- "required": [
- "basis",
- "why"
- ],
- "type": "object"
- }
- ]
- },
- "value": {
- "additionalProperties": {},
- "properties": {
- "stat": {
- "type": "string"
- },
- "unit": {
- "type": "string"
- },
- "value": {
- "type": "number"
- }
- },
- "required": [
- "stat",
- "unit",
- "value"
- ],
- "type": "object"
- }
-}Output schema / properties / rule_results / items / requiredRemoved value: -[
- "ruleName",
- "passed",
- "score",
- "message"
-]Output schema / properties / rule_results / items / typeRemoved value: -"object"Output schema / properties / rules_changedAdded value: +{
+ "additionalProperties": {},
+ "properties": {},
+ "type": "object"
+}Output schema / properties / rules_evaluated / descriptionRemoved value: -"rules that judged"Output schema / properties / rules_evaluated / maximumRemoved value: -9007199254740991Output schema / properties / rules_evaluated / minimumRemoved value: --9007199254740991Output schema / properties / rules_evaluated / typePrevious value: -"integer"New value: +"number"Output schema / properties / rules_skipped / descriptionRemoved value: -"rules that skipped"Output schema / properties / rules_skipped / maximumRemoved value: -9007199254740991Output schema / properties / rules_skipped / minimumRemoved value: --9007199254740991Output schema / properties / rules_skipped / typePrevious value: -"integer"New value: +"number"Output schema / properties / run_idAdded value: +{
+ "type": "string"
+}Output schema / properties / score / descriptionRemoved value: -"0..1 weighted quality over the rules that ran"Output schema / properties / suggestionsRemoved value: -{
- "description": "what to change",
- "items": {
- "type": "string"
- },
- "type": "array"
-}Output schema / properties / trace_id / descriptionRemoved value: -"the linked trace, when named"Output schema / properties / verdict / descriptionRemoved value: -"state, passed, basis (which layer decided), by (the rules), risk"Output schema / properties / verdict / properties / basisRemoved value: -{
- "enum": [
- "policy_gate",
- "detector_veto",
- "critical_unknown",
- "required_evidence_missing",
- "risk_over_loss",
- "score_below_threshold",
- "clean",
- "no_rules"
- ],
- "type": "string"
-}Output schema / properties / verdict / properties / byRemoved value: -{
- "items": {
- "type": "string"
- },
- "type": "array"
-}Output schema / properties / verdict / properties / confidenceRemoved value: -{
- "enum": [
- "decisive",
- "marginal"
- ],
- "type": "string"
-}Output schema / properties / verdict / properties / passedRemoved value: -{
- "type": "boolean"
-}Output schema / properties / verdict / properties / riskRemoved value: -{
- "anyOf": [
- {
- "additionalProperties": {},
- "properties": {
- "hi": {
- "type": "number"
- },
- "lo": {
- "type": "number"
- },
- "pBad": {
- "type": "number"
- }
- },
- "required": [
- "pBad",
- "lo",
- "hi"
- ],
- "type": "object"
- },
- {
- "type": "null"
- }
- ]
-}Output schema / properties / verdict / properties / stateRemoved value: -{
- "enum": [
- "pass",
- "fail",
- "unknown"
- ],
- "type": "string"
-}Output schema / properties / verdict / requiredRemoved value: -[
- "state",
- "passed",
- "basis",
- "by",
- "risk"
-]Output schema / requiredPrevious value: -[
- "id",
- "eval_type",
- "score",
- "passed",
- "rule_results",
- "suggestions",
- "rules_evaluated",
- "rules_skipped",
- "insufficient_data"
-]New value: +[
+ "id",
+ "eval_type",
+ "score",
+ "passed",
+ "rule_results",
+ "rules_evaluated",
+ "rules_skipped",
+ "insufficient_data"
+]evaluate_runsevaluate_with_llm_judge27 fields changedInput schema / properties / agent_modelAdded value: +{
+ "description": "The model that produced the output, for the same-family warning, when no linked trace records it",
+ "minLength": 1,
+ "type": "string"
+}Input schema / properties / input / descriptionPrevious value: -"User question / prompt that produced the output (improves accuracy for helpfulness/safety)"New value: +"User question / prompt that produced the output (required for relevance; improves helpfulness and safety)"Input schema / properties / max_cost_usd / descriptionPrevious value: -"Cost cap in USD for this call; defaults to IRIS_LLM_JUDGE_MAX_COST_USD_PER_EVAL or 0.25. The worst case (two attempts, full max_output_tokens) is computed before the call and refused if it exceeds the cap"New value: +"Cost cap in USD for this call, at most the operator's IRIS_LLM_JUDGE_MAX_COST_USD_PER_EVAL (default 0.25). The worst case (two attempts, full max_output_tokens) is computed before the call and refused if it exceeds the cap"Input schema / properties / model / descriptionPrevious value: -"Model ID. Supported: anthropic = claude-opus-4-7 | claude-sonnet-4-6 | claude-haiku-4-5 | claude-haiku-4-5-20251001; openai = gpt-4o | gpt-4o-mini | o1-mini. Required — cost varies a hundredfold across models"New value: +"Model ID. Supported: anthropic = claude-fable-5-1 | claude-opus-5-5 | claude-sonnet-5 (and the priced 4.x line); openai = gpt-5 | gpt-5-mini | gpt-4.1-mini (and the priced 4o, o3/o4 and retired o1 ids); an unknown id is refused with the full priced list. Required — cost varies a hundredfold across models"Input schema / properties / template / descriptionPrevious value: -"Judge dimension: accuracy (factual correctness), helpfulness (does it address the ask), safety (harm potential), correctness (vs reference answer — requires `expected`), faithfulness (RAG grounding — requires `source_material`)."New value: +"The question the judge answers; correctness needs expected, relevance needs input, faithfulness and task_completed read source_material"Input schema / properties / template / enumPrevious value: -[
- "accuracy",
- "helpfulness",
- "safety",
- "correctness",
- "faithfulness"
-]New value: +[
+ "accuracy",
+ "helpfulness",
+ "safety",
+ "correctness",
+ "faithfulness",
+ "task_completed",
+ "relevance"
+]Input schema / properties / trace_id / descriptionPrevious value: -"Link this evaluation to a stored trace (id from log_trace / get_traces); an unknown id is rejected BEFORE the judge is called"New value: +"Keep this evaluation beside a stored trace (id from log_trace / get_traces). It is listed with the trace and never replaces its verdict; an unknown id is rejected BEFORE the judge is called"Output schema / properties / cost_usd / descriptionRemoved value: -"the exact spend from the pricing table"Output schema / properties / dimensions / descriptionRemoved value: -"per-dimension sub-scores for the template"Output schema / properties / disagreement / descriptionRemoved value: -"true when the model's own boolean disagrees with the threshold verdict — its rubric and its judgement have come apart on this output"Output schema / properties / id / descriptionRemoved value: -"the evaluation id; read it back at iris://evaluations/{id}"Output schema / properties / input_tokens / descriptionRemoved value: -"tokens sent, across both attempts when a retry ran"Output schema / properties / latency_ms / descriptionRemoved value: -"wall time of the provider call(s)"Output schema / properties / model / descriptionRemoved value: -"the model that judged"Output schema / properties / output_tokens / descriptionRemoved value: -"tokens received, across both attempts when a retry ran"Output schema / properties / pass_threshold / descriptionRemoved value: -"the threshold the score was read against, so you can check the arithmetic"Output schema / properties / passed / descriptionRemoved value: -"the verdict: the score against the template's threshold, which is pass_threshold below. Not the model's own boolean — that is self_reported_pass"Output schema / properties / provider / descriptionRemoved value: -"the provider called"Output schema / properties / rationale / descriptionRemoved value: -"the judge's reasoning, in its words"Output schema / properties / raw_response_id / descriptionRemoved value: -"the provider's response id, for your own audit"Output schema / properties / reference_trace_idAdded value: +{
+ "type": "string"
+}Output schema / properties / score / descriptionRemoved value: -"0..1 from the judge"Output schema / properties / self_reported_pass / descriptionRemoved value: -"what the model said about passing, when it said anything. Recorded, never obeyed"Output schema / properties / template / descriptionRemoved value: -"the template used"Output schema / properties / trace_idRemoved value: -{
- "description": "the linked trace, when one was named",
- "type": "string"
-}Output schema / properties / verdictAdded value: +{
+ "additionalProperties": {},
+ "properties": {},
+ "type": "object"
+}Output schema / properties / warningsAdded value: +{
+ "items": {},
+ "type": "array"
+}get_traces28 fields changedInput schema / properties / include_textAdded value: +{
+ "default": false,
+ "description": "Return each stored text whole. By default each is cut to 500 characters, and the trace's cut lists what was cut with its full length; iris://traces/{trace_id} reads one trace whole",
+ "type": "boolean"
+}Input schema / properties / min_score / descriptionPrevious value: -"Minimum eval score filter (0..1; values outside are rejected) — applied to LATEST eval per trace, not all evals; must be <= max_score when both are set"New value: +"Minimum score (0..1) of each trace's latest evaluation; at most max_score"Input schema / properties / qAdded value: +{
+ "description": "Full-text search over input, output, tool-call values, metadata values and span attribute and event values. Every word must appear; \"quoted phrase\"; word* for a prefix of 3+ characters (1 for CJK). Case and accents ignored. At most 16 terms, 4 of them prefixes",
+ "maxLength": 500,
+ "type": "string"
+}Input schema / properties / sessionAdded value: +{
+ "description": "Filter by session id — the turns of one conversation, as logged with session_id",
+ "type": "string"
+}Input schema / properties / since / descriptionPrevious value: -"ISO 8601 timestamp (or date) lower bound — return traces with timestamp >= this; anything that is not an ISO timestamp is rejected, never treated as \"no bound\""New value: +"Inclusive lower bound, an ISO 8601 timestamp or date; anything else is rejected"Input schema / properties / sort_by / defaultRemoved value: -"timestamp"Input schema / properties / sort_by / descriptionPrevious value: -"Sort by timestamp | latency_ms | cost_usd (default timestamp)"New value: +"Sort by timestamp | latency_ms | cost_usd | relevance (default relevance with q, else timestamp)"Input schema / properties / sort_by / enumPrevious value: -[
- "timestamp",
- "latency_ms",
- "cost_usd"
-]New value: +[
+ "timestamp",
+ "latency_ms",
+ "cost_usd",
+ "relevance"
+]Output schema / properties / limit / descriptionRemoved value: -"the page size applied"Output schema / properties / limit / maximumRemoved value: -9007199254740991Output schema / properties / limit / minimumRemoved value: --9007199254740991Output schema / properties / limit / typePrevious value: -"integer"New value: +"number"Output schema / properties / offset / descriptionRemoved value: -"the offset applied"Output schema / properties / offset / maximumRemoved value: -9007199254740991Output schema / properties / offset / minimumRemoved value: --9007199254740991Output schema / properties / offset / typePrevious value: -"integer"New value: +"number"Output schema / properties / searchAdded value: +{
+ "additionalProperties": {},
+ "properties": {},
+ "type": "object"
+}Output schema / properties / summary / descriptionRemoved value: -"the dashboard aggregates for the last hour, when include_summary was true"Output schema / properties / total / descriptionRemoved value: -"how many traces match the filters, across every page"Output schema / properties / total / maximumRemoved value: -9007199254740991Output schema / properties / total / minimumRemoved value: --9007199254740991Output schema / properties / total / typePrevious value: -"integer"New value: +"number"Output schema / properties / traces / descriptionRemoved value: -"the page of traces: trace_id, agent_name, framework, input, output, tool_calls, latency_ms, token_usage, cost_usd, metadata, timestamp"Output schema / properties / traces / items / additionalPropertiesRemoved value: -{}Output schema / properties / traces / items / propertiesRemoved value: -{
- "trace_id": {
- "type": "string"
- }
-}Output schema / properties / traces / items / requiredRemoved value: -[
- "trace_id"
-]Output schema / properties / traces / items / typeRemoved value: -"object"Output schema / properties / untrustedAdded value: +{
+ "additionalProperties": {},
+ "properties": {},
+ "type": "object"
+}list_rules22 fields changedOutput schema / properties / built_in / descriptionRemoved value: -"the shipped roster, never filtered: name, category, description, weight, kind, mechanism, needs, question, classes, version, the EFFECTIVE critical flag with criticalSource, and proof (published precision, recall, intervals and ppvAt from https://iris-eval.com/proof; null where the proof is a conformance check)"Output schema / properties / built_in / items / additionalPropertiesRemoved value: -{}Output schema / properties / built_in / items / propertiesRemoved value: -{
- "name": {
- "type": "string"
- }
-}Output schema / properties / built_in / items / requiredRemoved value: -[
- "name"
-]Output schema / properties / built_in / items / typeRemoved value: -"object"Output schema / properties / enabled_count / descriptionRemoved value: -"of those, how many are enabled"Output schema / properties / enabled_count / maximumRemoved value: -9007199254740991Output schema / properties / enabled_count / minimumRemoved value: --9007199254740991Output schema / properties / enabled_count / typePrevious value: -"integer"New value: +"number"Output schema / properties / pluginsAdded value: +{
+ "items": {},
+ "type": "array"
+}Output schema / properties / quarantined / descriptionRemoved value: -"entries in the store this version could not validate; they do not fire and are never deleted by a deploy"Output schema / properties / rules / descriptionRemoved value: -"the deployed custom rules after the filters: id, name, description, evalType, severity, definition, enabled, createdAt, updatedAt, version, sourceMomentId"Output schema / properties / rules / items / additionalPropertiesRemoved value: -{}Output schema / properties / rules / items / propertiesRemoved value: -{
- "id": {
- "type": "string"
- },
- "name": {
- "type": "string"
- }
-}Output schema / properties / rules / items / requiredRemoved value: -[
- "id",
- "name"
-]Output schema / properties / rules / items / typeRemoved value: -"object"Output schema / properties / total / descriptionRemoved value: -"custom rules after the filters"Output schema / properties / total / maximumRemoved value: -9007199254740991Output schema / properties / total / minimumRemoved value: --9007199254740991Output schema / properties / total / typePrevious value: -"integer"New value: +"number"Output schema / properties / untrustedAdded value: +{
+ "additionalProperties": {},
+ "properties": {},
+ "type": "object"
+}Output schema / requiredPrevious value: -[
- "rules",
- "total",
- "enabled_count",
- "built_in",
- "quarantined"
-]New value: +[
+ "rules",
+ "total",
+ "enabled_count",
+ "built_in",
+ "quarantined",
+ "plugins"
+]log_trace25 fields changedInput schema / properties / case_keyAdded value: +{
+ "description": "What makes this the same question across runs (a fixture or test id); pairs traces. Derived from input when omitted",
+ "type": "string"
+}Input schema / properties / cost_usd / descriptionPrevious value: -"Total cost in USD — overrides per-span aggregation when provided (treated as authoritative)"New value: +"Total cost in USD, stored as reported. Omitted: estimated from the token counts and the model (metadata.model or the spans) at list price, and marked estimated"Input schema / properties / eval_typeAdded value: +{
+ "description": "With evaluate: true, the bundle to run (default: every bundle, noted in the response)",
+ "enum": [
+ "completeness",
+ "relevance",
+ "safety",
+ "cost",
+ "custom",
+ "all"
+ ],
+ "type": "string"
+}Input schema / properties / evaluateAdded value: +{
+ "default": false,
+ "description": "Also score the stored trace in this call, exactly as evaluate_output would; requires output",
+ "type": "boolean"
+}Input schema / properties / runAdded value: +{
+ "description": "The batch this execution belongs to (a CI job id, a sweep); compare_runs compares two runs",
+ "type": "string"
+}Input schema / properties / session_idAdded value: +{
+ "description": "The conversation this turn belongs to; groups its turns. Read from SEP-414 baggage when omitted",
+ "maxLength": 200,
+ "minLength": 1,
+ "type": "string"
+}Input schema / properties / token_usage / descriptionPrevious value: -"Token usage breakdown (prompt/completion/total — used for cost analysis)"New value: +"Token usage (prompt/completion/total, and cache_read_tokens / cache_creation_tokens: the cached part of prompt_tokens) — prices the trace when cost_usd is omitted"Input schema / properties / token_usage / properties / cache_creation_1h_tokensAdded value: +{
+ "minimum": 0,
+ "type": "number"
+}Input schema / properties / token_usage / properties / cache_creation_tokensAdded value: +{
+ "minimum": 0,
+ "type": "number"
+}Input schema / properties / token_usage / properties / cache_read_tokensAdded value: +{
+ "minimum": 0,
+ "type": "number"
+}Input schema / properties / token_usage / properties / prompt_tokens_detailsAdded value: +{
+ "properties": {
+ "cached_tokens": {
+ "minimum": 0,
+ "type": "number"
+ }
+ },
+ "type": "object"
+}Input schema / properties / tool_calls / descriptionPrevious value: -"Tool calls made during execution, in order, each { tool_name, input?, output?, latency_ms?, error? } — what the trajectory rules judge; evaluate_output reuses them when given this trace_id"New value: +"Tool calls made, in order, each { tool_name, input?, output?, latency_ms?, error? }; the trajectory rules judge them"Input schema / properties / tool_calls / items / properties / call_idAdded value: +{
+ "type": "string"
+}Input schema / properties / tool_calls / items / properties / cost_usdAdded value: +{
+ "type": "number"
+}Input schema / properties / tool_calls / items / properties / token_usageAdded value: +{
+ "properties": {
+ "completion_tokens": {
+ "type": "number"
+ },
+ "prompt_tokens": {
+ "type": "number"
+ },
+ "total_tokens": {
+ "type": "number"
+ }
+ },
+ "type": "object"
+}Input schema / properties / tool_calls / items / properties / truncatedAdded value: +{
+ "type": "boolean"
+}Input schema / properties / toolsAdded value: +{
+ "description": "Your MCP tools/list result, verbatim; lets the rules check call arguments later",
+ "items": {
+ "additionalProperties": false,
+ "properties": {
+ "_meta": {
+ "additionalProperties": {},
+ "propertyNames": {
+ "type": "string"
+ },
+ "type": "object"
+ },
+ "annotations": {
+ "additionalProperties": {},
+ "properties": {
+ "destructiveHint": {
+ "type": "boolean"
+ },
+ "idempotentHint": {
+ "type": "boolean"
+ },
+ "openWorldHint": {
+ "type": "boolean"
+ },
+ "readOnlyHint": {
+ "type": "boolean"
+ },
+ "title": {
+ "type": "string"
+ }
+ },
+ "type": "object"
+ },
+ "description": {
+ "maxLength": 4096,
+ "type": "string"
+ },
+ "execution": {
+ "additionalProperties": {},
+ "propertyNames": {
+ "type": "string"
+ },
+ "type": "object"
+ },
+ "inputSchema": {
+ "additionalProperties": {},
+ "propertyNames": {
+ "type": "string"
+ },
+ "type": "object"
+ },
+ "name": {
+ "maxLength": 128,
+ "minLength": 1,
+ "type": "string"
+ },
+ "outputSchema": {
+ "additionalProperties": {},
+ "propertyNames": {
+ "type": "string"
+ },
+ "type": "object"
+ },
+ "title": {
+ "maxLength": 512,
+ "type": "string"
+ }
+ },
+ "required": [
+ "name"
+ ],
+ "type": "object"
+ },
+ "maxItems": 200,
+ "type": "array"
+}Output schema / descriptionAdded value: +"Nested evaluation fields: https://iris-eval.com/response-schema-v1.json"Output schema / properties / cost_estimateAdded value: +{}Output schema / properties / cost_sourceAdded value: +{
+ "enum": [
+ "reported",
+ "estimated"
+ ],
+ "type": "string"
+}Output schema / properties / cost_usdAdded value: +{
+ "type": [
+ "number",
+ "null"
+ ]
+}Output schema / properties / evaluationAdded value: +{
+ "additionalProperties": {},
+ "properties": {},
+ "type": "object"
+}Output schema / properties / status / descriptionRemoved value: -"always \"stored\" on success"Output schema / properties / trace_id / descriptionRemoved value: -"the stored trace id, 32 hex — pass it to evaluate_output, get_traces or delete_trace"Output schema / requiredPrevious value: -[
- "trace_id",
- "status"
-]New value: +[
+ "trace_id",
+ "status",
+ "cost_usd"
+]verify_citations38 fields changedInput schema / properties / allow_fetch / descriptionPrevious value: -"Permit outbound HTTP to resolve URLs/DOIs. Defaults to IRIS_CITATION_ALLOW_FETCH=1; false otherwise. SSRF-guarded regardless."New value: +"false skips fetching for this call. Fetching happens only when the operator sets IRIS_CITATION_ALLOW_FETCH=1; true cannot turn it on. SSRF-guarded regardless."Input schema / properties / domain_allowlist / descriptionPrevious value: -"Restrict fetches to hostnames in this list (suffix match allowed). Merged with IRIS_CITATION_DOMAINS env."New value: +"Fetch only from these hostnames (suffix match). Narrows IRIS_CITATION_DOMAINS when the operator set it, never adds to it."Input schema / properties / max_cost_usd_total / descriptionPrevious value: -"Cap TOTAL judge cost across all citations in this call; default 1.00 USD — the pipeline stops when the next call would exceed it"New value: +"Cap TOTAL judge cost across all citations in this call, at most the operator's IRIS_CITATION_MAX_COST_USD_TOTAL (default 1.00 USD); the pipeline stops when the next call would exceed it"Input schema / properties / model / descriptionPrevious value: -"Judge model for per-citation verification. Supported: anthropic = claude-opus-4-7 | claude-sonnet-4-6 | claude-haiku-4-5 | claude-haiku-4-5-20251001; openai = gpt-4o | gpt-4o-mini | o1-mini."New value: +"Judge model for per-citation verification. Supported: anthropic = claude-fable-5-1 | claude-opus-5-5 | claude-sonnet-5 (and the priced 4.x line); openai = gpt-5 | gpt-5-mini | gpt-4.1-mini (and the priced 4o, o3/o4 and retired o1 ids); an unknown id is refused with the full priced list."Input schema / properties / trace_id / descriptionPrevious value: -"Link verification result to a stored trace (id from log_trace / get_traces); an unknown id is rejected before any fetch or judge call"New value: +"Keep the verification result beside a stored trace (id from log_trace / get_traces). It is listed with the trace and never replaces its verdict; an unknown id is rejected before any fetch or judge call"Output schema / properties / citations / descriptionRemoved value: -"per citation: the citation (raw, kind, identifier, offsets), resolve_status ok | skipped | error, resolve_error, source (url, status, content_type, bytes_fetched, truncated), judge (supported, confidence, rationale, cost_usd, latency_ms, tokens)"Output schema / properties / citations / items / additionalPropertiesRemoved value: -{}Output schema / properties / citations / items / propertiesRemoved value: -{
- "resolve_status": {
- "type": "string"
- }
-}Output schema / properties / citations / items / requiredRemoved value: -[
- "resolve_status"
-]Output schema / properties / citations / items / typeRemoved value: -"object"Output schema / properties / id / descriptionRemoved value: -"the evaluation id; read it back at iris://evaluations/{id}"Output schema / properties / overall_score / descriptionRemoved value: -"supported / judged; null when nothing was judged"Output schema / properties / passed / descriptionRemoved value: -"true when every judged citation was supported; false when any judged citation was not; NULL when nothing was judged — no verdict, because nothing was verified. Until 0.10.0 that last case returned true."Output schema / properties / reference_trace_idAdded value: +{
+ "type": "string"
+}Output schema / properties / total_citations_found / descriptionRemoved value: -"citations extracted from the output"Output schema / properties / total_citations_found / maximumRemoved value: -9007199254740991Output schema / properties / total_citations_found / minimumRemoved value: --9007199254740991Output schema / properties / total_citations_found / typePrevious value: -"integer"New value: +"number"Output schema / properties / total_cost_usd / descriptionRemoved value: -"the spend across every judge call"Output schema / properties / total_judged / descriptionRemoved value: -"citations the judge ruled on"Output schema / properties / total_judged / maximumRemoved value: -9007199254740991Output schema / properties / total_judged / minimumRemoved value: --9007199254740991Output schema / properties / total_judged / typePrevious value: -"integer"New value: +"number"Output schema / properties / total_resolved / descriptionRemoved value: -"citations whose source was fetched"Output schema / properties / total_resolved / maximumRemoved value: -9007199254740991Output schema / properties / total_resolved / minimumRemoved value: --9007199254740991Output schema / properties / total_resolved / typePrevious value: -"integer"New value: +"number"Output schema / properties / total_supported / descriptionRemoved value: -"citations the judge found supported"Output schema / properties / total_supported / maximumRemoved value: -9007199254740991Output schema / properties / total_supported / minimumRemoved value: --9007199254740991Output schema / properties / total_supported / typePrevious value: -"integer"New value: +"number"Output schema / properties / total_unsupported / descriptionRemoved value: -"judged citations the judge ruled unsupported — the number the verdict turns on"Output schema / properties / total_unsupported / maximumRemoved value: -9007199254740991Output schema / properties / total_unsupported / minimumRemoved value: --9007199254740991Output schema / properties / total_unsupported / typePrevious value: -"integer"New value: +"number"Output schema / properties / trace_idRemoved value: -{
- "description": "the linked trace, when one was named",
- "type": "string"
-}Output schema / properties / verdictAdded value: +{
+ "additionalProperties": {},
+ "properties": {},
+ "type": "object"
+}Output schema / properties / warningsAdded value: +{
+ "items": {},
+ "type": "array"
+}v0.10.0evaluate_with_llm_judge5 fields changedOutput schema / properties / disagreementAdded value: +{
+ "description": "true when the model's own boolean disagrees with the threshold verdict — its rubric and its judgement have come apart on this output",
+ "type": "boolean"
+}Output schema / properties / pass_thresholdAdded value: +{
+ "description": "the threshold the score was read against, so you can check the arithmetic",
+ "type": "number"
+}Output schema / properties / passed / descriptionPrevious value: -"the judge's own pass verdict for the template"New value: +"the verdict: the score against the template's threshold, which is pass_threshold below. Not the model's own boolean — that is self_reported_pass"Output schema / properties / self_reported_passAdded value: +{
+ "description": "what the model said about passing, when it said anything. Recorded, never obeyed",
+ "type": "boolean"
+}Output schema / requiredPrevious value: -[
- "id",
- "score",
- "passed",
- "rationale",
- "dimensions",
- "model",
- "provider",
- "template",
- "input_tokens",
- "output_tokens",
- "cost_usd",
- "latency_ms"
-]New value: +[
+ "id",
+ "score",
+ "passed",
+ "pass_threshold",
+ "rationale",
+ "dimensions",
+ "model",
+ "provider",
+ "template",
+ "input_tokens",
+ "output_tokens",
+ "cost_usd",
+ "latency_ms"
+]verify_citations4 fields changedOutput schema / properties / passed / descriptionPrevious value: -"true when every judged citation was supported, or nothing was judged and nothing failed"New value: +"true when every judged citation was supported; false when any judged citation was not; NULL when nothing was judged — no verdict, because nothing was verified. Until 0.10.0 that last case returned true."Output schema / properties / passed / typePrevious value: -"boolean"New value: +[
+ "boolean",
+ "null"
+]Output schema / properties / total_unsupportedAdded value: +{
+ "description": "judged citations the judge ruled unsupported — the number the verdict turns on",
+ "maximum": 9007199254740991,
+ "minimum": -9007199254740991,
+ "type": "integer"
+}Output schema / requiredPrevious value: -[
- "id",
- "overall_score",
- "passed",
- "total_citations_found",
- "total_resolved",
- "total_judged",
- "total_supported",
- "total_cost_usd",
- "citations"
-]New value: +[
+ "id",
+ "overall_score",
+ "passed",
+ "total_unsupported",
+ "total_citations_found",
+ "total_resolved",
+ "total_judged",
+ "total_supported",
+ "total_cost_usd",
+ "citations"
+]v0.9.0delete_rule1 field changedOutput schema / (root)Previous value: -nullNew value: +{
+ "$schema": "http://json-schema.org/draft-07/schema#",
+ "additionalProperties": {},
+ "properties": {
+ "deleted": {
+ "description": "true when a rule was removed; always false on a toggle",
+ "type": "boolean"
+ },
+ "enabled": {
+ "description": "toggle only: the rule's state after the call",
+ "type": "boolean"
+ },
+ "rule": {
+ "additionalProperties": {},
+ "description": "toggle only: the rule as stored",
+ "properties": {},
+ "type": "object"
+ },
+ "rule_id": {
+ "description": "the id that was asked for",
+ "type": "string"
+ },
+ "toggled": {
+ "description": "toggle only: true when the rule exists (also when it was already in the requested state)",
+ "type": "boolean"
+ }
+ },
+ "required": [
+ "deleted",
+ "rule_id"
+ ],
+ "type": "object"
+}delete_trace1 field changedOutput schema / (root)Previous value: -nullNew value: +{
+ "$schema": "http://json-schema.org/draft-07/schema#",
+ "additionalProperties": {},
+ "properties": {
+ "deleted": {
+ "description": "true when a trace row was removed; false when no trace with that id existed for this tenant",
+ "type": "boolean"
+ },
+ "trace_id": {
+ "description": "the id that was asked for",
+ "type": "string"
+ }
+ },
+ "required": [
+ "deleted",
+ "trace_id"
+ ],
+ "type": "object"
+}deploy_rule1 field changedOutput schema / (root)Previous value: -nullNew value: +{
+ "$schema": "http://json-schema.org/draft-07/schema#",
+ "additionalProperties": {},
+ "properties": {
+ "replaced": {
+ "description": "with replace: true, the earlier rule(s) of the same name that were retired",
+ "items": {
+ "additionalProperties": {},
+ "properties": {
+ "id": {
+ "type": "string"
+ }
+ },
+ "required": [
+ "id"
+ ],
+ "type": "object"
+ },
+ "type": "array"
+ },
+ "rule": {
+ "additionalProperties": {},
+ "description": "the rule as persisted: id (rule-<hex>, keep it for delete_rule), name, description, evalType, severity, definition, enabled, createdAt, updatedAt, version, sourceMomentId",
+ "properties": {
+ "id": {
+ "type": "string"
+ },
+ "name": {
+ "type": "string"
+ }
+ },
+ "required": [
+ "id",
+ "name"
+ ],
+ "type": "object"
+ },
+ "warning": {
+ "description": "with replace: true, one sentence naming what was retired",
+ "type": "string"
+ }
+ },
+ "required": [
+ "rule"
+ ],
+ "type": "object"
+}evaluate_output2 fields changedInput schema / properties / trace_id / descriptionPrevious value: -"Link evaluation to a trace — surfaces this eval in the dashboard's trace drill-through. Must be the id of a stored trace (from log_trace / get_traces); an unknown id is rejected before anything is evaluated"New value: +"Link evaluation to a trace — surfaces this eval in the dashboard's trace drill-through and lets the tool reuse the trace's stored tool_calls. Must be the id of a stored trace (from log_trace / get_traces); an unknown id is rejected before anything is evaluated"Output schema / (root)Previous value: -nullNew value: +{
+ "$schema": "http://json-schema.org/draft-07/schema#",
+ "additionalProperties": {},
+ "properties": {
+ "categories": {
+ "additionalProperties": {
+ "additionalProperties": {},
+ "properties": {
+ "critical_failures": {
+ "items": {
+ "type": "string"
+ },
+ "type": "array"
+ },
+ "critical_skipped": {
+ "items": {
+ "type": "string"
+ },
+ "type": "array"
+ },
+ "insufficient_data": {
+ "type": "boolean"
+ },
+ "passed": {
+ "type": [
+ "boolean",
+ "null"
+ ]
+ },
+ "rules_evaluated": {
+ "maximum": 9007199254740991,
+ "minimum": -9007199254740991,
+ "type": "integer"
+ },
+ "rules_skipped": {
+ "maximum": 9007199254740991,
+ "minimum": -9007199254740991,
+ "type": "integer"
+ },
+ "score": {
+ "type": [
+ "number",
+ "null"
+ ]
+ }
+ },
+ "required": [
+ "score",
+ "passed",
+ "rules_evaluated",
+ "rules_skipped",
+ "insufficient_data"
+ ],
+ "type": "object"
+ },
+ "description": "per-bundle verdicts for eval_type all",
+ "propertyNames": {
+ "type": "string"
+ },
+ "type": "object"
+ },
+ "coverage": {
+ "additionalProperties": {},
+ "description": "per question: judged, unjudged and why, or not_applicable; plus the inputs carried",
+ "properties": {
+ "dormant": {
+ "items": {
+ "additionalProperties": {},
+ "properties": {
+ "name": {
+ "type": "string"
+ },
+ "reason": {
+ "type": "string"
+ },
+ "ruleId": {
+ "type": "string"
+ }
+ },
+ "required": [
+ "ruleId",
+ "name",
+ "reason"
+ ],
+ "type": "object"
+ },
+ "type": "array"
+ },
+ "inputs": {
+ "additionalProperties": {
+ "type": "boolean"
+ },
+ "propertyNames": {
+ "type": "string"
+ },
+ "type": "object"
+ },
+ "questions": {
+ "items": {
+ "additionalProperties": {},
+ "properties": {
+ "id": {
+ "enum": [
+ "safe_output",
+ "grounded",
+ "complete",
+ "relevant",
+ "task_completed",
+ "tool_use_correct",
+ "within_budget"
+ ],
+ "type": "string"
+ },
+ "status": {
+ "enum": [
+ "judged",
+ "unjudged",
+ "not_applicable"
+ ],
+ "type": "string"
+ },
+ "why": {
+ "type": "string"
+ }
+ },
+ "required": [
+ "id",
+ "status"
+ ],
+ "type": "object"
+ },
+ "type": "array"
+ }
+ },
+ "required": [
+ "inputs",
+ "questions"
+ ],
+ "type": "object"
+ },
+ "critical_failures": {
+ "description": "critical rules that failed and vetoed passed",
+ "items": {
+ "type": "string"
+ },
+ "type": "array"
+ },
+ "critical_skipped": {
+ "description": "critical rules that could not judge; treat as unknown",
+ "items": {
+ "type": "string"
+ },
+ "type": "array"
+ },
+ "erased_at": {
+ "description": "set once the linked trace was deleted",
+ "type": "string"
+ },
+ "eval_type": {
+ "description": "the bundle that ran",
+ "enum": [
+ "completeness",
+ "relevance",
+ "safety",
+ "cost",
+ "custom",
+ "all"
+ ],
+ "type": "string"
+ },
+ "id": {
+ "description": "the evaluation id, readable at iris://evaluations/{id}",
+ "type": "string"
+ },
+ "insufficient_data": {
+ "description": "true when no rule could judge",
+ "type": "boolean"
+ },
+ "note": {
+ "description": "present when eval_type was omitted",
+ "type": "string"
+ },
+ "passed": {
+ "description": "the ship verdict; false when nothing was judged",
+ "type": "boolean"
+ },
+ "provenance": {
+ "additionalProperties": {},
+ "description": "Iris version, ruleset and config hashes, thresholds, corpus version, time",
+ "properties": {
+ "configHash": {
+ "type": "string"
+ },
+ "corpusVersion": {
+ "type": "string"
+ },
+ "irisVersion": {
+ "type": "string"
+ },
+ "judgedAt": {
+ "type": "string"
+ },
+ "rulesetHash": {
+ "type": "string"
+ },
+ "thresholds": {
+ "additionalProperties": {},
+ "properties": {
+ "default": {
+ "type": "number"
+ },
+ "perRule": {
+ "additionalProperties": {},
+ "propertyNames": {
+ "type": "string"
+ },
+ "type": "object"
+ }
+ },
+ "required": [
+ "default"
+ ],
+ "type": "object"
+ }
+ },
+ "required": [
+ "irisVersion",
+ "rulesetHash",
+ "configHash",
+ "thresholds",
+ "corpusVersion",
+ "judgedAt"
+ ],
+ "type": "object"
+ },
+ "rule_results": {
+ "description": "per rule: verdict, message, kind, role, question, saw, evidence, uncertainty",
+ "items": {
+ "additionalProperties": {},
+ "properties": {
+ "budgetExceeded": {
+ "type": "boolean"
+ },
+ "category": {
+ "enum": [
+ "completeness",
+ "relevance",
+ "safety",
+ "cost",
+ "custom"
+ ],
+ "type": "string"
+ },
+ "classes": {
+ "items": {
+ "type": "string"
+ },
+ "type": "array"
+ },
+ "configInvalid": {
+ "type": "boolean"
+ },
+ "critical": {
+ "type": "boolean"
+ },
+ "criticalSource": {
+ "enum": [
+ "default",
+ "config"
+ ],
+ "type": "string"
+ },
+ "evidence": {
+ "items": {
+ "oneOf": [
+ {
+ "additionalProperties": {},
+ "properties": {
+ "end": {
+ "maximum": 9007199254740991,
+ "minimum": 0,
+ "type": "integer"
+ },
+ "label": {
+ "type": "string"
+ },
+ "source": {
+ "type": "string"
+ },
+ "start": {
+ "maximum": 9007199254740991,
+ "minimum": 0,
+ "type": "integer"
+ },
+ "type": {
+ "const": "span",
+ "type": "string"
+ }
+ },
+ "required": [
+ "type",
+ "source",
+ "start",
+ "end",
+ "label"
+ ],
+ "type": "object"
+ },
+ {
+ "additionalProperties": {},
+ "properties": {
+ "count": {
+ "maximum": 9007199254740991,
+ "minimum": 0,
+ "type": "integer"
+ },
+ "name": {
+ "type": "string"
+ },
+ "type": {
+ "const": "pattern",
+ "type": "string"
+ }
+ },
+ "required": [
+ "type",
+ "name",
+ "count"
+ ],
+ "type": "object"
+ },
+ {
+ "additionalProperties": {},
+ "properties": {
+ "index": {
+ "maximum": 9007199254740991,
+ "minimum": 0,
+ "type": "integer"
+ },
+ "label": {
+ "type": "string"
+ },
+ "toolName": {
+ "type": "string"
+ },
+ "type": {
+ "const": "toolCall",
+ "type": "string"
+ }
+ },
+ "required": [
+ "type",
+ "index",
+ "toolName",
+ "label"
+ ],
+ "type": "object"
+ },
+ {
+ "additionalProperties": {},
+ "properties": {
+ "status": {
+ "enum": [
+ "resolved",
+ "dead",
+ "unverifiable",
+ "supported",
+ "unsupported"
+ ],
+ "type": "string"
+ },
+ "type": {
+ "const": "citation",
+ "type": "string"
+ },
+ "url": {
+ "type": "string"
+ }
+ },
+ "required": [
+ "type",
+ "url",
+ "status"
+ ],
+ "type": "object"
+ },
+ {
+ "additionalProperties": {},
+ "properties": {
+ "stat": {
+ "type": "string"
+ },
+ "threshold": {
+ "type": "number"
+ },
+ "thresholdSource": {
+ "enum": [
+ "default",
+ "config",
+ "call",
+ "rule"
+ ],
+ "type": "string"
+ },
+ "type": {
+ "const": "count",
+ "type": "string"
+ },
+ "unit": {
+ "type": "string"
+ },
+ "value": {
+ "type": "number"
+ }
+ },
+ "required": [
+ "type",
+ "stat",
+ "unit",
+ "value"
+ ],
+ "type": "object"
+ }
+ ]
+ },
+ "type": "array"
+ },
+ "kind": {
+ "enum": [
+ "measurement",
+ "detection",
+ "inference",
+ "judgment",
+ "policy",
+ "verification"
+ ],
+ "type": "string"
+ },
+ "message": {
+ "type": "string"
+ },
+ "passed": {
+ "type": "boolean"
+ },
+ "question": {
+ "enum": [
+ "safe_output",
+ "grounded",
+ "complete",
+ "relevant",
+ "task_completed",
+ "tool_use_correct",
+ "within_budget"
+ ],
+ "type": "string"
+ },
+ "role": {
+ "enum": [
+ "gate",
+ "veto",
+ "risk",
+ "advisory",
+ "term"
+ ],
+ "type": "string"
+ },
+ "ruleId": {
+ "type": "string"
+ },
+ "ruleName": {
+ "type": "string"
+ },
+ "ruleVersion": {
+ "maximum": 9007199254740991,
+ "minimum": -9007199254740991,
+ "type": "integer"
+ },
+ "saw": {
+ "items": {
+ "enum": [
+ "output",
+ "input",
+ "expected",
+ "tool_calls",
+ "tool_outputs",
+ "tools_catalogue",
+ "cost",
+ "tokens",
+ "citations"
+ ],
+ "type": "string"
+ },
+ "type": "array"
+ },
+ "score": {
+ "type": "number"
+ },
+ "skipClass": {
+ "enum": [
+ "not_applicable",
+ "defeated",
+ "config_invalid"
+ ],
+ "type": "string"
+ },
+ "skipReason": {
+ "type": "string"
+ },
+ "skipped": {
+ "type": "boolean"
+ },
+ "uncertainty": {
+ "oneOf": [
+ {
+ "additionalProperties": {},
+ "properties": {
+ "basis": {
+ "const": "published_accuracy",
+ "type": "string"
+ },
+ "corpus": {
+ "additionalProperties": {},
+ "properties": {
+ "fn": {
+ "maximum": 9007199254740991,
+ "minimum": -9007199254740991,
+ "type": "integer"
+ },
+ "fp": {
+ "maximum": 9007199254740991,
+ "minimum": -9007199254740991,
+ "type": "integer"
+ },
+ "labelling": {
+ "enum": [
+ "same-model",
+ "human-verified"
+ ],
+ "type": "string"
+ },
+ "n": {
+ "maximum": 9007199254740991,
+ "minimum": -9007199254740991,
+ "type": "integer"
+ },
+ "release": {
+ "type": "string"
+ },
+ "tn": {
+ "maximum": 9007199254740991,
+ "minimum": -9007199254740991,
+ "type": "integer"
+ },
+ "tp": {
+ "maximum": 9007199254740991,
+ "minimum": -9007199254740991,
+ "type": "integer"
+ },
+ "version": {
+ "type": "string"
+ }
+ },
+ "required": [
+ "n",
+ "tp",
+ "fp",
+ "fn",
+ "tn",
+ "version",
+ "release",
+ "labelling"
+ ],
+ "type": "object"
+ },
+ "fired": {
+ "type": "boolean"
+ },
+ "missRate": {
+ "additionalProperties": {},
+ "properties": {
+ "hi": {
+ "type": "number"
+ },
+ "lo": {
+ "type": "number"
+ },
+ "point": {
+ "type": "number"
+ }
+ },
+ "required": [
+ "point",
+ "lo",
+ "hi"
+ ],
+ "type": "object"
+ },
+ "ppv": {
+ "additionalProperties": {},
+ "properties": {
+ "hi": {
+ "type": "number"
+ },
+ "lo": {
+ "type": "number"
+ },
+ "point": {
+ "type": "number"
+ }
+ },
+ "required": [
+ "point",
+ "lo",
+ "hi"
+ ],
+ "type": "object"
+ },
+ "prior": {
+ "additionalProperties": {},
+ "properties": {
+ "pi": {
+ "type": "number"
+ },
+ "source": {
+ "enum": [
+ "default",
+ "config",
+ "estimated"
+ ],
+ "type": "string"
+ }
+ },
+ "required": [
+ "pi",
+ "source"
+ ],
+ "type": "object"
+ }
+ },
+ "required": [
+ "basis",
+ "fired",
+ "prior",
+ "corpus"
+ ],
+ "type": "object"
+ },
+ {
+ "additionalProperties": {},
+ "properties": {
+ "basis": {
+ "const": "definition",
+ "type": "string"
+ },
+ "conformance": {
+ "additionalProperties": {},
+ "properties": {
+ "matched": {
+ "maximum": 9007199254740991,
+ "minimum": -9007199254740991,
+ "type": "integer"
+ },
+ "n": {
+ "maximum": 9007199254740991,
+ "minimum": -9007199254740991,
+ "type": "integer"
+ }
+ },
+ "required": [
+ "n",
+ "matched"
+ ],
+ "type": "object"
+ }
+ },
+ "required": [
+ "basis",
+ "conformance"
+ ],
+ "type": "object"
+ },
+ {
+ "additionalProperties": {},
+ "properties": {
+ "basis": {
+ "const": "self_consistency",
+ "type": "string"
+ },
+ "samples": {
+ "maximum": 9007199254740991,
+ "minimum": -9007199254740991,
+ "type": "integer"
+ },
+ "scoreSd": {
+ "type": "number"
+ },
+ "voteFraction": {
+ "type": "number"
+ }
+ },
+ "required": [
+ "basis",
+ "samples",
+ "voteFraction",
+ "scoreSd"
+ ],
+ "type": "object"
+ },
+ {
+ "additionalProperties": {},
+ "properties": {
+ "basis": {
+ "const": "local_labels",
+ "type": "string"
+ },
+ "n": {
+ "maximum": 9007199254740991,
+ "minimum": -9007199254740991,
+ "type": "integer"
+ },
+ "precision": {
+ "additionalProperties": {},
+ "properties": {
+ "hi": {
+ "type": "number"
+ },
+ "lo": {
+ "type": "number"
+ },
+ "point": {
+ "type": "number"
+ }
+ },
+ "required": [
+ "point",
+ "lo",
+ "hi"
+ ],
+ "type": "object"
+ }
+ },
+ "required": [
+ "basis",
+ "precision",
+ "n"
+ ],
+ "type": "object"
+ },
+ {
+ "additionalProperties": {},
+ "properties": {
+ "basis": {
+ "const": "policy",
+ "type": "string"
+ }
+ },
+ "required": [
+ "basis"
+ ],
+ "type": "object"
+ },
+ {
+ "additionalProperties": {},
+ "properties": {
+ "basis": {
+ "const": "unmeasured",
+ "type": "string"
+ },
+ "why": {
+ "type": "string"
+ }
+ },
+ "required": [
+ "basis",
+ "why"
+ ],
+ "type": "object"
+ }
+ ]
+ },
+ "value": {
+ "additionalProperties": {},
+ "properties": {
+ "stat": {
+ "type": "string"
+ },
+ "unit": {
+ "type": "string"
+ },
+ "value": {
+ "type": "number"
+ }
+ },
+ "required": [
+ "stat",
+ "unit",
+ "value"
+ ],
+ "type": "object"
+ }
+ },
+ "required": [
+ "ruleName",
+ "passed",
+ "score",
+ "message"
+ ],
+ "type": "object"
+ },
+ "type": "array"
+ },
+ "rules_evaluated": {
+ "description": "rules that judged",
+ "maximum": 9007199254740991,
+ "minimum": -9007199254740991,
+ "type": "integer"
+ },
+ "rules_skipped": {
+ "description": "rules that skipped",
+ "maximum": 9007199254740991,
+ "minimum": -9007199254740991,
+ "type": "integer"
+ },
+ "score": {
+ "description": "0..1 weighted quality over the rules that ran",
+ "type": "number"
+ },
+ "suggestions": {
+ "description": "what to change",
+ "items": {
+ "type": "string"
+ },
+ "type": "array"
+ },
+ "trace_id": {
+ "description": "the linked trace, when named",
+ "type": "string"
+ },
+ "verdict": {
+ "additionalProperties": {},
+ "description": "state, passed, basis (which layer decided), by (the rules), risk",
+ "properties": {
+ "basis": {
+ "enum": [
+ "policy_gate",
+ "detector_veto",
+ "critical_unknown",
+ "required_evidence_missing",
+ "risk_over_loss",
+ "score_below_threshold",
+ "clean",
+ "no_rules"
+ ],
+ "type": "string"
+ },
+ "by": {
+ "items": {
+ "type": "string"
+ },
+ "type": "array"
+ },
+ "confidence": {
+ "enum": [
+ "decisive",
+ "marginal"
+ ],
+ "type": "string"
+ },
+ "passed": {
+ "type": "boolean"
+ },
+ "risk": {
+ "anyOf": [
+ {
+ "additionalProperties": {},
+ "properties": {
+ "hi": {
+ "type": "number"
+ },
+ "lo": {
+ "type": "number"
+ },
+ "pBad": {
+ "type": "number"
+ }
+ },
+ "required": [
+ "pBad",
+ "lo",
+ "hi"
+ ],
+ "type": "object"
+ },
+ {
+ "type": "null"
+ }
+ ]
+ },
+ "state": {
+ "enum": [
+ "pass",
+ "fail",
+ "unknown"
+ ],
+ "type": "string"
+ }
+ },
+ "required": [
+ "state",
+ "passed",
+ "basis",
+ "by",
+ "risk"
+ ],
+ "type": "object"
+ }
+ },
+ "required": [
+ "id",
+ "eval_type",
+ "score",
+ "passed",
+ "rule_results",
+ "suggestions",
+ "rules_evaluated",
+ "rules_skipped",
+ "insufficient_data"
+ ],
+ "type": "object"
+}evaluate_with_llm_judge3 fields changedInput schema / properties / max_cost_usd / descriptionPrevious value: -"Cost cap in USD; defaults to IRIS_LLM_JUDGE_MAX_COST_USD_PER_EVAL or 0.25"New value: +"Cost cap in USD for this call; defaults to IRIS_LLM_JUDGE_MAX_COST_USD_PER_EVAL or 0.25. The worst case (two attempts, full max_output_tokens) is computed before the call and refused if it exceeds the cap"Input schema / properties / model / descriptionPrevious value: -"Model ID. Supported: anthropic = claude-opus-4-7 | claude-sonnet-4-6 | claude-haiku-4-5 | claude-haiku-4-5-20251001; openai = gpt-4o | gpt-4o-mini | o1-mini."New value: +"Model ID. Supported: anthropic = claude-opus-4-7 | claude-sonnet-4-6 | claude-haiku-4-5 | claude-haiku-4-5-20251001; openai = gpt-4o | gpt-4o-mini | o1-mini. Required — cost varies a hundredfold across models"Output schema / (root)Previous value: -nullNew value: +{
+ "$schema": "http://json-schema.org/draft-07/schema#",
+ "additionalProperties": {},
+ "properties": {
+ "cost_usd": {
+ "description": "the exact spend from the pricing table",
+ "type": [
+ "number",
+ "null"
+ ]
+ },
+ "dimensions": {
+ "additionalProperties": {},
+ "description": "per-dimension sub-scores for the template",
+ "propertyNames": {
+ "type": "string"
+ },
+ "type": "object"
+ },
+ "id": {
+ "description": "the evaluation id; read it back at iris://evaluations/{id}",
+ "type": "string"
+ },
+ "input_tokens": {
+ "description": "tokens sent, across both attempts when a retry ran",
+ "type": "number"
+ },
+ "latency_ms": {
+ "description": "wall time of the provider call(s)",
+ "type": "number"
+ },
+ "model": {
+ "description": "the model that judged",
+ "type": "string"
+ },
+ "output_tokens": {
+ "description": "tokens received, across both attempts when a retry ran",
+ "type": "number"
+ },
+ "passed": {
+ "description": "the judge's own pass verdict for the template",
+ "type": "boolean"
+ },
+ "provider": {
+ "description": "the provider called",
+ "enum": [
+ "anthropic",
+ "openai"
+ ],
+ "type": "string"
+ },
+ "rationale": {
+ "description": "the judge's reasoning, in its words",
+ "type": "string"
+ },
+ "raw_response_id": {
+ "description": "the provider's response id, for your own audit",
+ "type": "string"
+ },
+ "score": {
+ "description": "0..1 from the judge",
+ "type": "number"
+ },
+ "template": {
+ "description": "the template used",
+ "type": "string"
+ },
+ "trace_id": {
+ "description": "the linked trace, when one was named",
+ "type": "string"
+ }
+ },
+ "required": [
+ "id",
+ "score",
+ "passed",
+ "rationale",
+ "dimensions",
+ "model",
+ "provider",
+ "template",
+ "input_tokens",
+ "output_tokens",
+ "cost_usd",
+ "latency_ms"
+ ],
+ "type": "object"
+}get_traces3 fields changedInput schema / properties / agent_name / descriptionPrevious value: -"Filter by agent name — exact match (no wildcards in v0.4)"New value: +"Filter by agent name — exact match (no wildcards)"Input schema / properties / limit / descriptionPrevious value: -"Results per page (default 50, max 1000 — values >1000 return 400)"New value: +"Results per page (default 50, max 1000 — values above are rejected)"Output schema / (root)Previous value: -nullNew value: +{
+ "$schema": "http://json-schema.org/draft-07/schema#",
+ "additionalProperties": {},
+ "properties": {
+ "limit": {
+ "description": "the page size applied",
+ "maximum": 9007199254740991,
+ "minimum": -9007199254740991,
+ "type": "integer"
+ },
+ "offset": {
+ "description": "the offset applied",
+ "maximum": 9007199254740991,
+ "minimum": -9007199254740991,
+ "type": "integer"
+ },
+ "summary": {
+ "additionalProperties": {},
+ "description": "the dashboard aggregates for the last hour, when include_summary was true",
+ "properties": {},
+ "type": "object"
+ },
+ "total": {
+ "description": "how many traces match the filters, across every page",
+ "maximum": 9007199254740991,
+ "minimum": -9007199254740991,
+ "type": "integer"
+ },
+ "traces": {
+ "description": "the page of traces: trace_id, agent_name, framework, input, output, tool_calls, latency_ms, token_usage, cost_usd, metadata, timestamp",
+ "items": {
+ "additionalProperties": {},
+ "properties": {
+ "trace_id": {
+ "type": "string"
+ }
+ },
+ "required": [
+ "trace_id"
+ ],
+ "type": "object"
+ },
+ "type": "array"
+ }
+ },
+ "required": [
+ "traces",
+ "total",
+ "limit",
+ "offset"
+ ],
+ "type": "object"
+}list_rules3 fields changedInput schema / properties / enabled_only / descriptionPrevious value: -"Return only enabled rules (excludes disabled ones)"New value: +"Return only enabled custom rules (a rule disabled with delete_rule stays in the store and does not fire)"Input schema / properties / eval_type / descriptionPrevious value: -"Filter to rules of a specific eval category"New value: +"Filter the custom rules to one eval category (exact match); built_in is never filtered"Output schema / (root)Previous value: -nullNew value: +{
+ "$schema": "http://json-schema.org/draft-07/schema#",
+ "additionalProperties": {},
+ "properties": {
+ "built_in": {
+ "description": "the shipped roster, never filtered: name, category, description, weight, kind, mechanism, needs, question, classes, version, the EFFECTIVE critical flag with criticalSource, and proof (published precision, recall, intervals and ppvAt from https://iris-eval.com/proof; null where the proof is a conformance check)",
+ "items": {
+ "additionalProperties": {},
+ "properties": {
+ "name": {
+ "type": "string"
+ }
+ },
+ "required": [
+ "name"
+ ],
+ "type": "object"
+ },
+ "type": "array"
+ },
+ "enabled_count": {
+ "description": "of those, how many are enabled",
+ "maximum": 9007199254740991,
+ "minimum": -9007199254740991,
+ "type": "integer"
+ },
+ "quarantined": {
+ "description": "entries in the store this version could not validate; they do not fire and are never deleted by a deploy",
+ "items": {},
+ "type": "array"
+ },
+ "rules": {
+ "description": "the deployed custom rules after the filters: id, name, description, evalType, severity, definition, enabled, createdAt, updatedAt, version, sourceMomentId",
+ "items": {
+ "additionalProperties": {},
+ "properties": {
+ "id": {
+ "type": "string"
+ },
+ "name": {
+ "type": "string"
+ }
+ },
+ "required": [
+ "id",
+ "name"
+ ],
+ "type": "object"
+ },
+ "type": "array"
+ },
+ "total": {
+ "description": "custom rules after the filters",
+ "maximum": 9007199254740991,
+ "minimum": -9007199254740991,
+ "type": "integer"
+ }
+ },
+ "required": [
+ "rules",
+ "total",
+ "enabled_count",
+ "built_in",
+ "quarantined"
+ ],
+ "type": "object"
+}log_trace3 fields changedInput schema / properties / spans / descriptionPrevious value: -"Detailed execution spans (hierarchical span tree with timings, attributes, events)"New value: +"Detailed execution spans (hierarchical span tree with timings, attributes, events); a span without start_time takes the trace timestamp"Input schema / properties / tool_calls / descriptionPrevious value: -"Tool calls made during execution (per-call latency, errors, input/output)"New value: +"Tool calls made during execution, in order, each { tool_name, input?, output?, latency_ms?, error? } — what the trajectory rules judge; evaluate_output reuses them when given this trace_id"Output schema / (root)Previous value: -nullNew value: +{
+ "$schema": "http://json-schema.org/draft-07/schema#",
+ "additionalProperties": {},
+ "properties": {
+ "status": {
+ "const": "stored",
+ "description": "always \"stored\" on success",
+ "type": "string"
+ },
+ "trace_id": {
+ "description": "the stored trace id, 32 hex — pass it to evaluate_output, get_traces or delete_trace",
+ "type": "string"
+ }
+ },
+ "required": [
+ "trace_id",
+ "status"
+ ],
+ "type": "object"
+}verify_citations4 fields changedInput schema / properties / max_citations / descriptionPrevious value: -"Max citations to verify (extras skipped); default 20"New value: +"Max citations to verify (extras skipped, not errored); default 20, at most 50"Input schema / properties / max_cost_usd_total / descriptionPrevious value: -"Cap TOTAL judge cost across all citations in this call; default $1.00"New value: +"Cap TOTAL judge cost across all citations in this call; default 1.00 USD — the pipeline stops when the next call would exceed it"Input schema / properties / model / descriptionPrevious value: -"Judge model for per-citation verification. Supported: anthropic = claude-opus-4-7 | claude-sonnet-4-6 | claude-haiku-4-5-20251001; openai = gpt-4o | gpt-4o-mini | o1-mini."New value: +"Judge model for per-citation verification. Supported: anthropic = claude-opus-4-7 | claude-sonnet-4-6 | claude-haiku-4-5 | claude-haiku-4-5-20251001; openai = gpt-4o | gpt-4o-mini | o1-mini."Output schema / (root)Previous value: -nullNew value: +{
+ "$schema": "http://json-schema.org/draft-07/schema#",
+ "additionalProperties": {},
+ "properties": {
+ "citations": {
+ "description": "per citation: the citation (raw, kind, identifier, offsets), resolve_status ok | skipped | error, resolve_error, source (url, status, content_type, bytes_fetched, truncated), judge (supported, confidence, rationale, cost_usd, latency_ms, tokens)",
+ "items": {
+ "additionalProperties": {},
+ "properties": {
+ "resolve_status": {
+ "type": "string"
+ }
+ },
+ "required": [
+ "resolve_status"
+ ],
+ "type": "object"
+ },
+ "type": "array"
+ },
+ "id": {
+ "description": "the evaluation id; read it back at iris://evaluations/{id}",
+ "type": "string"
+ },
+ "overall_score": {
+ "description": "supported / judged; null when nothing was judged",
+ "type": [
+ "number",
+ "null"
+ ]
+ },
+ "passed": {
+ "description": "true when every judged citation was supported, or nothing was judged and nothing failed",
+ "type": "boolean"
+ },
+ "total_citations_found": {
+ "description": "citations extracted from the output",
+ "maximum": 9007199254740991,
+ "minimum": -9007199254740991,
+ "type": "integer"
+ },
+ "total_cost_usd": {
+ "description": "the spend across every judge call",
+ "type": "number"
+ },
+ "total_judged": {
+ "description": "citations the judge ruled on",
+ "maximum": 9007199254740991,
+ "minimum": -9007199254740991,
+ "type": "integer"
+ },
+ "total_resolved": {
+ "description": "citations whose source was fetched",
+ "maximum": 9007199254740991,
+ "minimum": -9007199254740991,
+ "type": "integer"
+ },
+ "total_supported": {
+ "description": "citations the judge found supported",
+ "maximum": 9007199254740991,
+ "minimum": -9007199254740991,
+ "type": "integer"
+ },
+ "trace_id": {
+ "description": "the linked trace, when one was named",
+ "type": "string"
+ }
+ },
+ "required": [
+ "id",
+ "overall_score",
+ "passed",
+ "total_citations_found",
+ "total_resolved",
+ "total_judged",
+ "total_supported",
+ "total_cost_usd",
+ "citations"
+ ],
+ "type": "object"
+}v0.8.0delete_rule3 fields changedInput schema / additionalPropertiesAdded value: +falseInput schema / properties / enabledAdded value: +{
+ "description": "When present the rule is NOT deleted: false DISABLES it (kept in the store, stops firing immediately, history and provenance preserved); true RE-ENABLES a disabled rule. Omit to delete",
+ "type": "boolean"
+}Input schema / properties / rule_id / descriptionPrevious value: -"Rule id to delete (format: rule-<hex>); obtained from list_rules or deploy_rule response"New value: +"Rule id to delete or toggle (format: rule-<hex>); obtained from list_rules or deploy_rule response"delete_trace1 field changedInput schema / additionalPropertiesAdded value: +falsedeploy_rule20 fields changedInput schema / additionalPropertiesAdded value: +falseInput schema / properties / definition / additionalPropertiesAdded value: +falseInput schema / properties / definition / descriptionPrevious value: -"Check definition (regex, length, keyword, cost, or schema)"New value: +"Check definition (regex, length, keyword, cost, or schema). Accepts exactly type, config, weight and an optional name — an unknown key is rejected"Input schema / properties / definition / properties / config / descriptionAdded value: +"Check configuration; required keys depend on type (regex_match: pattern; min_length: min_length; max_length: max_length; contains_keywords/excludes_keywords: keywords; cost_threshold: max_cost; json_schema: none)"Input schema / properties / definition / properties / name / descriptionAdded value: +"Optional and IGNORED if given — the server overwrites it with the top-level `name` so the rule reports under one name everywhere"Input schema / properties / definition / properties / name / maxLengthAdded value: +80Input schema / properties / definition / properties / name / minLengthAdded value: +1Input schema / properties / definition / properties / type / descriptionAdded value: +"Check type — decides which config keys are required"Input schema / properties / definition / properties / weight / descriptionAdded value: +"Weight in the weighted score (default 1; must be > 0)"Input schema / properties / definition / properties / weight / exclusiveMinimumAdded value: +0Input schema / properties / definition / requiredPrevious value: -[
- "name",
- "type",
- "config"
-]New value: +[
+ "type",
+ "config"
+]Input schema / properties / evalType / descriptionPrevious value: -"Eval category this rule belongs to; determines when it fires"New value: +"camelCase alias of eval_type, accepted for compatibility — prefer eval_type (snake_case is canonical across the tools)"Input schema / properties / eval_typeAdded value: +{
+ "description": "Eval category this rule belongs to; the rule fires on evaluate_output calls whose eval_type equals it (and on eval_type=\"all\"). Canonical snake_case spelling — pass exactly one of eval_type / evalType",
+ "enum": [
+ "completeness",
+ "relevance",
+ "safety",
+ "cost",
+ "custom"
+ ],
+ "type": "string"
+}Input schema / properties / name / descriptionPrevious value: -"Human-readable rule name (used in eval results)"New value: +"Human-readable rule name (1-80 chars; used in eval results). Must be unique among deployed rules unless replace=true"Input schema / properties / name / maxLengthPrevious value: -120New value: +80Input schema / properties / replaceAdded value: +{
+ "default": false,
+ "description": "When a rule with this name is already deployed: false (default) rejects the call; true deletes the existing same-named rule(s) and deploys this one in their place (fresh id; audit rows preserved)",
+ "type": "boolean"
+}Input schema / properties / severity / descriptionPrevious value: -"Severity used for dashboard sort + audit alerts"New value: +"What a FAILURE of this rule means. low/medium: informational — contributes to the weighted score only (plus dashboard sort + audit alerts). high/critical: hard-fail — a failing evaluation of this rule forces the overall passed=false regardless of the weighted score"Input schema / properties / sourceMomentId / descriptionPrevious value: -"Optional Decision Moment ID the rule was derived from (preserves workflow-inversion provenance)"New value: +"camelCase alias of source_moment_id, accepted for compatibility — prefer source_moment_id"Input schema / properties / source_moment_idAdded value: +{
+ "description": "Optional Decision Moment ID the rule was derived from (preserves workflow-inversion provenance). Canonical snake_case — pass exactly one of source_moment_id / sourceMomentId",
+ "type": "string"
+}Input schema / requiredPrevious value: -[
- "name",
- "evalType",
- "definition"
-]New value: +[
+ "name",
+ "definition"
+]evaluate_output19 fields changedInput schema / additionalPropertiesAdded value: +falseInput schema / properties / cost_usd / descriptionPrevious value: -"Cost in USD — only consulted when eval_type=\"cost\" (compared against cost_threshold rules)"New value: +"Cost in USD — consulted by the cost bundle (eval_type=\"cost\" or \"all\") AND by any cost_threshold custom rule regardless of eval_type; omit it and such a rule skips rather than passes (a critical one is listed in critical_skipped)"Input schema / properties / custom_rules / descriptionPrevious value: -"Custom evaluation rules — fires REGARDLESS of eval_type; pass eval_type=\"custom\" if you want ONLY these"New value: +"Custom evaluation rules, max 10 per call (deploy persistent rule sets via deploy_rule instead) — fires REGARDLESS of eval_type; pass eval_type=\"custom\" if you want ONLY these. Each entry accepts exactly name, type, config, weight — an unknown key (e.g. a misspelled weight) is rejected"Input schema / properties / custom_rules / items / additionalPropertiesAdded value: +falseInput schema / properties / custom_rules / items / properties / config / descriptionAdded value: +"Check configuration; keys depend on type (pattern, min_length, keywords, max_cost, …)"Input schema / properties / custom_rules / items / properties / name / descriptionAdded value: +"Rule name as it will appear in rule_results"Input schema / properties / custom_rules / items / properties / name / minLengthAdded value: +1Input schema / properties / custom_rules / items / properties / type / descriptionAdded value: +"Check type — decides which config keys the rule reads"Input schema / properties / custom_rules / items / properties / weight / descriptionAdded value: +"Weight in the weighted score (default 1; must be > 0)"Input schema / properties / custom_rules / items / properties / weight / exclusiveMinimumAdded value: +0Input schema / properties / custom_rules / maxItemsAdded value: +10Input schema / properties / eval_type / defaultRemoved value: -"completeness"Input schema / properties / eval_type / descriptionPrevious value: -"Rule bundle to apply: completeness | relevance | safety | cost | custom — picks which built-in rules fire"New value: +"Rule bundle to apply: completeness | relevance | safety | cost | custom | all — picks which built-in rules fire. \"all\" runs every bundle in one call and adds a per-category breakdown. Defaults to \"all\" when omitted — every bundle runs, safety included, and the response carries a note saying the default ran"Input schema / properties / eval_type / enumPrevious value: -[
- "completeness",
- "relevance",
- "safety",
- "cost",
- "custom"
-]New value: +[
+ "completeness",
+ "relevance",
+ "safety",
+ "cost",
+ "custom",
+ "all"
+]Input schema / properties / expected / descriptionPrevious value: -"Expected output for comparison — REQUIRED when eval_type=\"relevance\" (used as keyword-overlap target)"New value: +"Expected output for comparison — consulted only by the completeness bundle's expected_coverage rule; NOT used by relevance (the relevance rules compare the output against `input`)"Input schema / properties / input / descriptionPrevious value: -"Original input for context — improves relevance scoring (keyword overlap vs input)"New value: +"Original input for context (the ask + any source material the agent was given) — REQUIRED when eval_type=\"relevance\" (keyword_overlap and topic_consistency compare the output against it and skip without it); also grounds the safety bundle's hallucination signals"Input schema / properties / token_usage / descriptionPrevious value: -"Token usage breakdown — only consulted when eval_type=\"cost\" (used for token-budget rules)"New value: +"Token usage breakdown — only consulted by the cost bundle (eval_type=\"cost\" or \"all\"; used for token-budget rules)"Input schema / properties / tool_callsAdded value: +{
+ "description": "What the agent DID — the tool calls it made, in order, each { tool_name, input?, output?, latency_ms?, error? } exactly as log_trace records them. Read by the trajectory rules — the rules that judge what the agent DID rather than what it wrote. Omit it and those rules SKIP rather than pass — an evaluation with no trajectory data reports \"not judged\", never \"clean\". When trace_id names a stored trace and this argument is omitted, the tool_calls stored on that trace are loaded and used, so a caller who already logged them need not resend them",
+ "items": {
+ "additionalProperties": false,
+ "properties": {
+ "error": {
+ "type": "string"
+ },
+ "input": {},
+ "latency_ms": {
+ "type": "number"
+ },
+ "output": {},
+ "tool_name": {
+ "type": "string"
+ }
+ },
+ "required": [
+ "tool_name"
+ ],
+ "type": "object"
+ },
+ "type": "array"
+}Input schema / properties / trace_id / descriptionPrevious value: -"Link evaluation to a trace — surfaces this eval in the dashboard's trace drill-through"New value: +"Link evaluation to a trace — surfaces this eval in the dashboard's trace drill-through. Must be the id of a stored trace (from log_trace / get_traces); an unknown id is rejected before anything is evaluated"evaluate_with_llm_judge2 fields changedInput schema / additionalPropertiesAdded value: +falseInput schema / properties / trace_id / descriptionPrevious value: -"Link this evaluation to a trace"New value: +"Link this evaluation to a stored trace (id from log_trace / get_traces); an unknown id is rejected BEFORE the judge is called"get_traces16 fields changedInput schema / additionalPropertiesAdded value: +falseInput schema / properties / limit / maximumAdded value: +1000Input schema / properties / limit / minimumAdded value: +1Input schema / properties / limit / typePrevious value: -"number"New value: +"integer"Input schema / properties / max_score / descriptionPrevious value: -"Maximum eval score filter (0..1) — applied to LATEST eval per trace"New value: +"Maximum eval score filter (0..1; values outside are rejected) — applied to LATEST eval per trace"Input schema / properties / max_score / maximumAdded value: +1Input schema / properties / max_score / minimumAdded value: +0Input schema / properties / min_score / descriptionPrevious value: -"Minimum eval score filter (0..1) — applied to LATEST eval per trace, not all evals"New value: +"Minimum eval score filter (0..1; values outside are rejected) — applied to LATEST eval per trace, not all evals; must be <= max_score when both are set"Input schema / properties / min_score / maximumAdded value: +1Input schema / properties / min_score / minimumAdded value: +0Input schema / properties / offset / descriptionPrevious value: -"Zero-based pagination offset — skip first N results"New value: +"Zero-based pagination offset — skip first N results (non-negative integer)"Input schema / properties / offset / maximumAdded value: +9007199254740991Input schema / properties / offset / minimumAdded value: +0Input schema / properties / offset / typePrevious value: -"number"New value: +"integer"Input schema / properties / since / descriptionPrevious value: -"ISO timestamp lower bound — return traces with timestamp >= this"New value: +"ISO 8601 timestamp (or date) lower bound — return traces with timestamp >= this; anything that is not an ISO timestamp is rejected, never treated as \"no bound\""Input schema / properties / until / descriptionPrevious value: -"ISO timestamp upper bound — return traces with timestamp < this"New value: +"ISO 8601 timestamp (or date) upper bound — return traces with timestamp <= this; must not be earlier than `since`"list_rules1 field changedInput schema / additionalPropertiesAdded value: +falselog_trace2 fields changedInput schema / additionalPropertiesAdded value: +falseInput schema / properties / tool_calls / items / additionalPropertiesAdded value: +falseverify_citations2 fields changedInput schema / additionalPropertiesAdded value: +falseInput schema / properties / trace_id / descriptionPrevious value: -"Link verification result to a trace"New value: +"Link verification result to a stored trace (id from log_trace / get_traces); an unknown id is rejected before any fetch or judge call"v0.4.6delete_rule1 field changedInput schema / additionalPropertiesRemoved value: -falsedelete_trace1 field changedInput schema / additionalPropertiesRemoved value: -falsedeploy_rule3 fields changedInput schema / additionalPropertiesRemoved value: -falseInput schema / properties / definition / additionalPropertiesRemoved value: -falseInput schema / properties / definition / properties / config / propertyNamesAdded value: +{
+ "type": "string"
+}evaluate_output4 fields changedInput schema / additionalPropertiesRemoved value: -falseInput schema / properties / custom_rules / items / additionalPropertiesRemoved value: -falseInput schema / properties / custom_rules / items / properties / config / propertyNamesAdded value: +{
+ "type": "string"
+}Input schema / properties / token_usage / additionalPropertiesRemoved value: -falseevaluate_with_llm_judge2 fields changedInput schema / additionalPropertiesRemoved value: -falseInput schema / properties / timeout_ms / maximumAdded value: +9007199254740991get_traces1 field changedInput schema / additionalPropertiesRemoved value: -falselist_rules1 field changedInput schema / additionalPropertiesRemoved value: -falselog_trace8 fields changedInput schema / additionalPropertiesRemoved value: -falseInput schema / properties / metadata / propertyNamesAdded value: +{
+ "type": "string"
+}Input schema / properties / spans / items / additionalPropertiesRemoved value: -falseInput schema / properties / spans / items / properties / attributes / propertyNamesAdded value: +{
+ "type": "string"
+}Input schema / properties / spans / items / properties / events / items / additionalPropertiesRemoved value: -falseInput schema / properties / spans / items / properties / events / items / properties / attributes / propertyNamesAdded value: +{
+ "type": "string"
+}Input schema / properties / token_usage / additionalPropertiesRemoved value: -falseInput schema / properties / tool_calls / items / additionalPropertiesRemoved value: -falseverify_citations3 fields changedInput schema / additionalPropertiesRemoved value: -falseInput schema / properties / per_source_max_bytes / maximumAdded value: +9007199254740991Input schema / properties / per_source_timeout_ms / maximumAdded value: +9007199254740991v0.1.10delete_ruledelete_tracedeploy_ruleevaluate_output8 fields changedInput schema / properties / cost_usd / descriptionPrevious value: -"Cost for cost evaluation"New value: +"Cost in USD — only consulted when eval_type=\"cost\" (compared against cost_threshold rules)"Input schema / properties / custom_rules / descriptionPrevious value: -"Custom evaluation rules"New value: +"Custom evaluation rules — fires REGARDLESS of eval_type; pass eval_type=\"custom\" if you want ONLY these"Input schema / properties / eval_type / descriptionPrevious value: -"Type of evaluation"New value: +"Rule bundle to apply: completeness | relevance | safety | cost | custom — picks which built-in rules fire"Input schema / properties / expected / descriptionPrevious value: -"Expected output for comparison"New value: +"Expected output for comparison — REQUIRED when eval_type=\"relevance\" (used as keyword-overlap target)"Input schema / properties / input / descriptionPrevious value: -"Original input for context"New value: +"Original input for context — improves relevance scoring (keyword overlap vs input)"Input schema / properties / output / descriptionPrevious value: -"The output text to evaluate"New value: +"The output text to evaluate (the agent's response that gets scored against rules)"Input schema / properties / token_usage / descriptionPrevious value: -"Token usage for cost evaluation"New value: +"Token usage breakdown — only consulted when eval_type=\"cost\" (used for token-budget rules)"Input schema / properties / trace_id / descriptionPrevious value: -"Link evaluation to a trace"New value: +"Link evaluation to a trace — surfaces this eval in the dashboard's trace drill-through"evaluate_with_llm_judgeget_traces11 fields changedInput schema / properties / agent_name / descriptionPrevious value: -"Filter by agent name"New value: +"Filter by agent name — exact match (no wildcards in v0.4)"Input schema / properties / framework / descriptionPrevious value: -"Filter by framework"New value: +"Filter by agent framework — exact match (e.g., langchain, autogen)"Input schema / properties / include_summary / descriptionPrevious value: -"Include dashboard summary stats"New value: +"Include dashboard summary stats in same response — saves a round-trip when ingesting for dashboards"Input schema / properties / limit / descriptionPrevious value: -"Results per page"New value: +"Results per page (default 50, max 1000 — values >1000 return 400)"Input schema / properties / max_score / descriptionPrevious value: -"Maximum eval score filter"New value: +"Maximum eval score filter (0..1) — applied to LATEST eval per trace"Input schema / properties / min_score / descriptionPrevious value: -"Minimum eval score filter"New value: +"Minimum eval score filter (0..1) — applied to LATEST eval per trace, not all evals"Input schema / properties / offset / descriptionPrevious value: -"Pagination offset"New value: +"Zero-based pagination offset — skip first N results"Input schema / properties / since / descriptionPrevious value: -"ISO timestamp lower bound"New value: +"ISO timestamp lower bound — return traces with timestamp >= this"Input schema / properties / sort_by / descriptionPrevious value: -"Sort field"New value: +"Sort by timestamp | latency_ms | cost_usd (default timestamp)"Input schema / properties / sort_order / descriptionPrevious value: -"Sort order"New value: +"Sort order: asc | desc (default desc — most recent / highest first)"Input schema / properties / until / descriptionPrevious value: -"ISO timestamp upper bound"New value: +"ISO timestamp upper bound — return traces with timestamp < this"list_ruleslog_trace11 fields changedInput schema / properties / agent_name / descriptionPrevious value: -"Name of the agent"New value: +"Agent name — used for filtering in get_traces (e.g., \"customer-support-bot\")"Input schema / properties / cost_usd / descriptionPrevious value: -"Total cost in USD"New value: +"Total cost in USD — overrides per-span aggregation when provided (treated as authoritative)"Input schema / properties / framework / descriptionPrevious value: -"Agent framework name"New value: +"Agent framework identifier (e.g., langchain, autogen, custom)"Input schema / properties / input / descriptionPrevious value: -"Agent input text"New value: +"Agent input text — the user prompt or upstream input that produced this output"Input schema / properties / latency_ms / descriptionPrevious value: -"Total execution time in milliseconds"New value: +"Total execution time in milliseconds (end-to-end agent latency)"Input schema / properties / metadata / descriptionPrevious value: -"Arbitrary metadata"New value: +"Opaque key-value tags (e.g. {requestId, userId, env}) — queryable in dashboard, not via get_traces filters"Input schema / properties / output / descriptionPrevious value: -"Agent output text"New value: +"Agent output text — what the agent produced (pass to evaluate_output for scoring)"Input schema / properties / spans / descriptionPrevious value: -"Detailed execution spans"New value: +"Detailed execution spans (hierarchical span tree with timings, attributes, events)"Input schema / properties / timestamp / descriptionPrevious value: -"Trace timestamp (ISO 8601)"New value: +"Trace timestamp (ISO 8601); defaults to now() when omitted"Input schema / properties / token_usage / descriptionPrevious value: -"Token usage breakdown"New value: +"Token usage breakdown (prompt/completion/total — used for cost analysis)"Input schema / properties / tool_calls / descriptionPrevious value: -"Tool calls made during execution"New value: +"Tool calls made during execution (per-call latency, errors, input/output)"verify_citationsv0.1.8evaluate_outputget_traceslog_tracev0.1.7evaluate_outputget_traceslog_traceevaluate_outputget_traceslog_traceScored across 12 tools
Three evaluation entry points (evaluate_output, evaluate_with_llm_judge, evaluate_runs) and two comparison tools (compare_runs, compare_traces) superficially overlap, but each description states its scope and an explicit 'when not to use it' that routes to the sibling. The only genuinely soft boundary is compare_runs vs compare_traces (cross-run regression vs per-case reliability), which a careful agent can still resolve from the descriptions.
Nearly everything follows verb_noun snake_case (log_trace, delete_trace, deploy_rule, list_rules, compare_runs, evaluate_output). Minor deviations: get_traces uses 'get' where the equivalent read for rules is 'list_rules', and evaluate_with_llm_judge is a long outlier pattern. Readable and predictable overall.
12 tools for an eval/observability server: traces (log/get/delete), rules (deploy/list/delete-toggle), evaluation (deterministic/LLM/re-run), analysis (compare runs/traces), and citation verification. Each tool occupies a distinct slot with no filler.
Trace lifecycle, rule lifecycle (create/replace/disable/enable/delete), evaluation, and regression analysis are all covered; immutability and replace-via-deploy are documented rather than gaps. Minor gaps: no list_runs or run-listing tool (run ids must be discovered through get_traces), and data retention/export is only reachable via configuration, not a tool.
MCP observability. Query live traffic, errors, duration, and alerts from your AI agent.
MCP server for building and testing AI agents with multi-model experimentation and insights.
Hosted MCP server for LLM cost estimation, model comparison, and budget-aware routing.
AgentGuard — 20-tool AI safety MCP: policy preflight, risk scoring, audit logging, rate limits.
