iris-eval/mcp-server

Verify Citations

verify_citations

Check whether each citation in an agent output is supported by its source: extracts, optionally fetches, and judges citations, returning per-citation verdicts and an overall score.

Instructions

Is each citation in an output supported by its source? Extracts, fetches (opt-in, SSRF-guarded) and judges on your key.

What it does. Fetches only if the operator set IRIS_CITATION_ALLOW_FETCH=1, blocking private addresses. One judge call per resolved citation, within max_cost_usd_total. overall_score: supported / judged; null if none judged. Full detail: iris://capabilities.

When not to use it. With no citations (evaluate_output). Without IRIS_ANTHROPIC_API_KEY or IRIS_OPENAI_API_KEY.

Returns. JSON: id, reference_trace_id, overall_score, passed, verdict, total_unsupported, total_citations_found, total_resolved, total_judged, total_supported, total_cost_usd, warnings, citations.

Errors. IRIS_JUDGE_NOT_ENABLED, IRIS_JUDGE_UNKNOWN_MODEL, IRIS_UNKNOWN_TRACE (before any spend); IRIS_JUDGE_FAILED when every judge call failed. Failures return {"error":{code,message,recovery}} with isError; follow recovery.

Siblings. evaluate_with_llm_judge — semantic scoring; evaluate_output — the free deterministic path.

Input Schema

Name	Required	Description
`model`	Yes	Judge model for per-citation verification. Supported: anthropic = claude-fable-5-1 \| claude-opus-5-5 \| claude-sonnet-5 (and the priced 4.x line); openai = gpt-5 \| gpt-5-mini \| gpt-4.1-mini (and the priced 4o, o3/o4 and retired o1 ids); an unknown id is refused with the full priced list.
`output`	Yes	The agent output containing citations to verify
`provider`	No	Auto-detected from model when omitted
`trace_id`	No	Keep the verification result beside a stored trace (id from log_trace / get_traces). It is listed with the trace and never replaces its verdict; an unknown id is rejected before any fetch or judge call
`allow_fetch`	No	false skips fetching for this call. Fetching happens only when the operator sets IRIS_CITATION_ALLOW_FETCH=1; true cannot turn it on. SSRF-guarded regardless.
`max_citations`	No	Max citations to verify (extras skipped, not errored); default 20, at most 50
`domain_allowlist`	No	Fetch only from these hostnames (suffix match). Narrows IRIS_CITATION_DOMAINS when the operator set it, never adds to it.
`max_cost_usd_total`	No	Cap TOTAL judge cost across all citations in this call, at most the operator's IRIS_CITATION_MAX_COST_USD_TOTAL (default 1.00 USD); the pipeline stops when the next call would exceed it
`per_source_max_bytes`	No	Per-URL body cap; default 5MB
`per_source_timeout_ms`	No	Per-URL fetch timeout; default 10_000

Output Schema

Name	Required	Description	Default
`id`	Yes
`passed`	Yes
`verdict`	No
`warnings`	No
`citations`	Yes
`total_judged`	Yes
`overall_score`	Yes
`total_cost_usd`	Yes
`total_resolved`	Yes
`total_supported`	Yes
`total_unsupported`	Yes
`reference_trace_id`	No
`total_citations_found`	Yes

Schema Changelog

Changes observed during successful MCP inspections.

Changed38 schema fields changedv0.10.1Oct 26, 2026

changedInput schema / properties / allow_fetch / description

Previous value: -"Permit outbound HTTP to resolve URLs/DOIs. Defaults to IRIS_CITATION_ALLOW_FETCH=1; false otherwise. SSRF-guarded regardless."New value: +"false skips fetching for this call. Fetching happens only when the operator sets IRIS_CITATION_ALLOW_FETCH=1; true cannot turn it on. SSRF-guarded regardless."

changedInput schema / properties / domain_allowlist / description

Previous value: -"Restrict fetches to hostnames in this list (suffix match allowed). Merged with IRIS_CITATION_DOMAINS env."New value: +"Fetch only from these hostnames (suffix match). Narrows IRIS_CITATION_DOMAINS when the operator set it, never adds to it."

changedInput schema / properties / max_cost_usd_total / description

Previous value: -"Cap TOTAL judge cost across all citations in this call; default 1.00 USD — the pipeline stops when the next call would exceed it"New value: +"Cap TOTAL judge cost across all citations in this call, at most the operator's IRIS_CITATION_MAX_COST_USD_TOTAL (default 1.00 USD); the pipeline stops when the next call would exceed it"

changedInput schema / properties / model / description

Previous value: -"Judge model for per-citation verification. Supported: anthropic = claude-opus-4-7 | claude-sonnet-4-6 | claude-haiku-4-5 | claude-haiku-4-5-20251001; openai = gpt-4o | gpt-4o-mini | o1-mini."New value: +"Judge model for per-citation verification. Supported: anthropic = claude-fable-5-1 | claude-opus-5-5 | claude-sonnet-5 (and the priced 4.x line); openai = gpt-5 | gpt-5-mini | gpt-4.1-mini (and the priced 4o, o3/o4 and retired o1 ids); an unknown id is refused with the full priced list."

changedInput schema / properties / trace_id / description

Previous value: -"Link verification result to a stored trace (id from log_trace / get_traces); an unknown id is rejected before any fetch or judge call"New value: +"Keep the verification result beside a stored trace (id from log_trace / get_traces). It is listed with the trace and never replaces its verdict; an unknown id is rejected before any fetch or judge call"

removedOutput schema / properties / citations / description

Removed value: -"per citation: the citation (raw, kind, identifier, offsets), resolve_status ok | skipped | error, resolve_error, source (url, status, content_type, bytes_fetched, truncated), judge (supported, confidence, rationale, cost_usd, latency_ms, tokens)"

removedOutput schema / properties / citations / items / additionalProperties
```
Removed value: -{}
```

removedOutput schema / properties / citations / items / properties

Removed value: -{
-  "resolve_status": {
-    "type": "string"
-  }
-}

removedOutput schema / properties / citations / items / required
```
Removed value: -[
-  "resolve_status"
-]
```
removedOutput schema / properties / citations / items / type
```
Removed value: -"object"
```

removedOutput schema / properties / id / description

Removed value: -"the evaluation id; read it back at iris://evaluations/{id}"

removedOutput schema / properties / overall_score / description

Removed value: -"supported / judged; null when nothing was judged"

removedOutput schema / properties / passed / description

Removed value: -"true when every judged citation was supported; false when any judged citation was not; NULL when nothing was judged — no verdict, because nothing was verified. Until 0.10.0 that last case returned true."

addedOutput schema / properties / reference_trace_id
```
Added value: +{
+  "type": "string"
+}
```
removedOutput schema / properties / total_citations_found / description
```
Removed value: -"citations extracted from the output"
```
removedOutput schema / properties / total_citations_found / maximum
```
Removed value: -9007199254740991
```
removedOutput schema / properties / total_citations_found / minimum
```
Removed value: --9007199254740991
```
changedOutput schema / properties / total_citations_found / type
```
Previous value: -"integer"New value: +"number"
```
removedOutput schema / properties / total_cost_usd / description
```
Removed value: -"the spend across every judge call"
```
removedOutput schema / properties / total_judged / description
```
Removed value: -"citations the judge ruled on"
```
removedOutput schema / properties / total_judged / maximum
```
Removed value: -9007199254740991
```
removedOutput schema / properties / total_judged / minimum
```
Removed value: --9007199254740991
```
changedOutput schema / properties / total_judged / type
```
Previous value: -"integer"New value: +"number"
```
removedOutput schema / properties / total_resolved / description
```
Removed value: -"citations whose source was fetched"
```
removedOutput schema / properties / total_resolved / maximum
```
Removed value: -9007199254740991
```
removedOutput schema / properties / total_resolved / minimum
```
Removed value: --9007199254740991
```
changedOutput schema / properties / total_resolved / type
```
Previous value: -"integer"New value: +"number"
```
removedOutput schema / properties / total_supported / description
```
Removed value: -"citations the judge found supported"
```
removedOutput schema / properties / total_supported / maximum
```
Removed value: -9007199254740991
```
removedOutput schema / properties / total_supported / minimum
```
Removed value: --9007199254740991
```
changedOutput schema / properties / total_supported / type
```
Previous value: -"integer"New value: +"number"
```

removedOutput schema / properties / total_unsupported / description

Removed value: -"judged citations the judge ruled unsupported — the number the verdict turns on"

removedOutput schema / properties / total_unsupported / maximum
```
Removed value: -9007199254740991
```
removedOutput schema / properties / total_unsupported / minimum
```
Removed value: --9007199254740991
```
changedOutput schema / properties / total_unsupported / type
```
Previous value: -"integer"New value: +"number"
```

removedOutput schema / properties / trace_id

Removed value: -{
-  "description": "the linked trace, when one was named",
-  "type": "string"
-}

addedOutput schema / properties / verdict

Added value: +{
+  "additionalProperties": {},
+  "properties": {},
+  "type": "object"
+}

addedOutput schema / properties / warnings

Added value: +{
+  "items": {},
+  "type": "array"
+}

Changed4 schema fields changedv0.10.0Sep 6, 2026

changedOutput schema / properties / passed / description

Previous value: -"true when every judged citation was supported, or nothing was judged and nothing failed"New value: +"true when every judged citation was supported; false when any judged citation was not; NULL when nothing was judged — no verdict, because nothing was verified. Until 0.10.0 that last case returned true."

changedOutput schema / properties / passed / type

Previous value: -"boolean"New value: +[
+  "boolean",
+  "null"
+]

addedOutput schema / properties / total_unsupported

Added value: +{
+  "description": "judged citations the judge ruled unsupported — the number the verdict turns on",
+  "maximum": 9007199254740991,
+  "minimum": -9007199254740991,
+  "type": "integer"
+}

changedOutput schema / required

Previous value: -[
-  "id",
-  "overall_score",
-  "passed",
-  "total_citations_found",
-  "total_resolved",
-  "total_judged",
-  "total_supported",
-  "total_cost_usd",
-  "citations"
-]New value: +[
+  "id",
+  "overall_score",
+  "passed",
+  "total_unsupported",
+  "total_citations_found",
+  "total_resolved",
+  "total_judged",
+  "total_supported",
+  "total_cost_usd",
+  "citations"
+]

Changed4 schema fields changedv0.9.0Sep 5, 2026

changedInput schema / properties / max_citations / description

Previous value: -"Max citations to verify (extras skipped); default 20"New value: +"Max citations to verify (extras skipped, not errored); default 20, at most 50"

changedInput schema / properties / max_cost_usd_total / description

Previous value: -"Cap TOTAL judge cost across all citations in this call; default $1.00"New value: +"Cap TOTAL judge cost across all citations in this call; default 1.00 USD — the pipeline stops when the next call would exceed it"

changedInput schema / properties / model / description

Previous value: -"Judge model for per-citation verification. Supported: anthropic = claude-opus-4-7 | claude-sonnet-4-6 | claude-haiku-4-5-20251001; openai = gpt-4o | gpt-4o-mini | o1-mini."New value: +"Judge model for per-citation verification. Supported: anthropic = claude-opus-4-7 | claude-sonnet-4-6 | claude-haiku-4-5 | claude-haiku-4-5-20251001; openai = gpt-4o | gpt-4o-mini | o1-mini."

changedOutput schema / (root)

Previous value: -nullNew value: +{
+  "$schema": "http://json-schema.org/draft-07/schema#",
+  "additionalProperties": {},
+  "properties": {
+    "citations": {
+      "description": "per citation: the citation (raw, kind, identifier, offsets), resolve_status ok | skipped | error, resolve_error, source (url, status, content_type, bytes_fetched, truncated), judge (supported, confidence, rationale, cost_usd, latency_ms, tokens)",
+      "items": {
+        "additionalProperties": {},
+        "properties": {
+          "resolve_status": {
+            "type": "string"
+          }
+        },
+        "required": [
+          "resolve_status"
+        ],
+        "type": "object"
+      },
+      "type": "array"
+    },
+    "id": {
+      "description": "the evaluation id; read it back at iris://evaluations/{id}",
+      "type": "string"
+    },
+    "overall_score": {
+      "description": "supported / judged; null when nothing was judged",
+      "type": [
+        "number",
+        "null"
+      ]
+    },
+    "passed": {
+      "description": "true when every judged citation was supported, or nothing was judged and nothing failed",
+      "type": "boolean"
+    },
+    "total_citations_found": {
+      "description": "citations extracted from the output",
+      "maximum": 9007199254740991,
+      "minimum": -9007199254740991,
+      "type": "integer"
+    },
+    "total_cost_usd": {
+      "description": "the spend across every judge call",
+      "type": "number"
+    },
+    "total_judged": {
+      "description": "citations the judge ruled on",
+      "maximum": 9007199254740991,
+      "minimum": -9007199254740991,
+      "type": "integer"
+    },
+    "total_resolved": {
+      "description": "citations whose source was fetched",
+      "maximum": 9007199254740991,
+      "minimum": -9007199254740991,
+      "type": "integer"
+    },
+    "total_supported": {
+      "description": "citations the judge found supported",
+      "maximum": 9007199254740991,
+      "minimum": -9007199254740991,
+      "type": "integer"
+    },
+    "trace_id": {
+      "description": "the linked trace, when one was named",
+      "type": "string"
+    }
+  },
+  "required": [
+    "id",
+    "overall_score",
+    "passed",
+    "total_citations_found",
+    "total_resolved",
+    "total_judged",
+    "total_supported",
+    "total_cost_usd",
+    "citations"
+  ],
+  "type": "object"
+}

Changed2 schema fields changedv0.8.0Sep 4, 2026

addedInput schema / additionalProperties
```
Added value: +false
```

changedInput schema / properties / trace_id / description

Previous value: -"Link verification result to a trace"New value: +"Link verification result to a stored trace (id from log_trace / get_traces); an unknown id is rejected before any fetch or judge call"

Changed3 schema fields changedv0.4.6Aug 11, 2026
- removedInput schema / additionalProperties
```
Removed value: -false
```
- addedInput schema / properties / per_source_max_bytes / maximum
```
Added value: +9007199254740991
```
- addedInput schema / properties / per_source_timeout_ms / maximum
```
Added value: +9007199254740991
```
Addedv0.1.10May 3, 2026

TDQS

A4.9/5.0

Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Beyond the annotations (openWorldHint, readOnlyHint=false), it discloses that fetching is gated by IRIS_CITATION_ALLOW_FETCH and SSRF-guarded against private addresses, that cost is bounded per call, how overall_score is computed (supported/judged, null when nothing judged), and a full error-code taxonomy with 'rejected before any spend' semantics. It also clarifies that trace_id never overwrites a stored verdict.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded core question, then labeled sections (What it does / When not to use it / Returns / Errors / Siblings). Dense but every line carries operational information; the pointer to iris://capabilities keeps detail out of the prompt without losing it.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 10-parameter, networked, cost-incurring tool, the description covers gating preconditions, cost bounds, error handling with recovery, and result interpretation. An output schema exists, and the brief field list is a bonus rather than a substitute for documentation.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so baseline is 3; the description adds cross-parameter behavior not obvious from individual fields — one judge call per resolved citation within max_cost_usd_total, and fetch opt-in that a per-call flag cannot override. The remaining parameters (timeouts, byte caps, allowlist narrowing) are left to schema text rather than elaborated further.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Opens with a precise question ('Is each citation in an output supported by its source?') followed by the verb+resource+mechanism: extract, fetch, judge per citation. It names the two closest siblings and states how they differ (semantic scoring vs. the free deterministic path), so an agent can disambiguate without opening any schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

A dedicated 'When not to use it' section gives explicit exclusions with routing: no citations -> evaluate_output; no provider API key -> do not call. Combined with the 'Siblings' line, both when-to-use and when-not are covered.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Deploy Server