iris-eval/mcp-server

Evaluate Output

evaluate_output

Idempotent

Scores agent outputs against deterministic rules, returning a ship verdict, per-rule evidence, and skipped checks so you can catch relevance, safety, cost, or completeness issues before shipping.

Instructions

Score an output with deterministic rules: a ship verdict with its basis, per-rule evidence, what was unjudged.

What it does. eval_type picks a bundle or all (the default). A rule missing its input SKIPS, never passes: input is REQUIRED when eval_type="relevance"; tool_calls, tools, cost_usd, expected feed the rest. trace_id alone scores the stored trace. Full detail: iris://capabilities.

When not to use it. For semantic judgment (evaluate_with_llm_judge). As an input firewall: the rules read the output.

Returns. JSON: id, trace_id, reference_trace_id, verdict, coverage, provenance, run_id, interpretations, erased_at, eval_type, score, passed, rule_results, rules_evaluated, rules_skipped, insufficient_data, critical_failures, critical_skipped, categories, note, rules_changed.

Errors. IRIS_UNKNOWN_TRACE (nothing written); IRIS_STORAGE_ERROR. Failures return {"error":{code,message,recovery}} with isError; follow recovery.

Siblings. log_trace — record the run first; list_rules — what each rule needs.

Input Schema

Name	Required	Description
`input`	No	The ask and any source material given — REQUIRED when eval_type="relevance"; also grounds the hallucination signals
`tools`	No	Your MCP tools/list result, verbatim; lets the rules check call arguments. Loaded from trace_id when stored
`output`	No	The output text to evaluate (the agent's response that gets scored against rules). Omit it with trace_id to score the trace's stored output
`cost_usd`	No	Cost in USD, for the cost bundle and any cost_threshold rule; omitted, the stored cost of trace_id (reported or estimated), else they skip rather than pass
`expected`	No	Expected output for comparison — consulted only by the completeness bundle's expected_coverage rule; NOT used by relevance (the relevance rules compare the output against `input`)
`trace_id`	No	A stored trace (from log_trace / get_traces). Alone, it scores the trace as stored and the result is the trace's verdict. With output, input, tool_calls, tools, cost_usd or token_usage that differ from the record, or one eval_type, the result is kept beside the trace (reference_trace_id) and never replaces its verdict. An unknown id is rejected before anything is evaluated
`eval_type`	No	Rule bundle: completeness \| relevance \| safety \| cost \| custom \| all (the default, noted in the response)
`tool_calls`	No	The tool calls the agent made, in order, as log_trace records them; omitted, the trajectory rules skip. Loaded from trace_id when stored
`token_usage`	No	Token usage breakdown — only consulted by the cost bundle (eval_type="cost" or "all"; used for token-budget rules)
`custom_rules`	No	Up to 10 one-off rules; they fire whatever eval_type is (eval_type="custom" runs only these)
`expected_trajectory`	No	What the agent was expected to DO: tool_calls with mode and args (tool_sequence); step_budget and tolerance (step_budget)

Output Schema

Name	Required	Description	Default
`id`	Yes
`note`	No
`score`	Yes
`passed`	Yes
`run_id`	No
`verdict`	No
`coverage`	No
`trace_id`	No
`erased_at`	No
`eval_type`	Yes
`categories`	No
`provenance`	No
`rule_results`	Yes
`rules_changed`	No
`rules_skipped`	Yes
`interpretations`	No
`rules_evaluated`	Yes
`critical_skipped`	No
`critical_failures`	No
`insufficient_data`	Yes
`reference_trace_id`	No

Schema Changelog

Changes observed during successful MCP inspections.

Changed74 schema fields changedv0.10.1Oct 26, 2026

changedInput schema / properties / cost_usd / description

Previous value: -"Cost in USD — consulted by the cost bundle (eval_type=\"cost\" or \"all\") AND by any cost_threshold custom rule regardless of eval_type; omit it and such a rule skips rather than passes (a critical one is listed in critical_skipped)"New value: +"Cost in USD, for the cost bundle and any cost_threshold rule; omitted, the stored cost of trace_id (reported or estimated), else they skip rather than pass"

changedInput schema / properties / custom_rules / description

Previous value: -"Custom evaluation rules, max 10 per call (deploy persistent rule sets via deploy_rule instead) — fires REGARDLESS of eval_type; pass eval_type=\"custom\" if you want ONLY these. Each entry accepts exactly name, type, config, weight — an unknown key (e.g. a misspelled weight) is rejected"New value: +"Up to 10 one-off rules; they fire whatever eval_type is (eval_type=\"custom\" runs only these)"

addedInput schema / properties / custom_rules / items / properties / severity

Added value: +{
+  "description": "high or critical: a failure fails the verdict; otherwise it advises",
+  "enum": [
+    "low",
+    "medium",
+    "high",
+    "critical"
+  ],
+  "type": "string"
+}

changedInput schema / properties / custom_rules / items / properties / type / enum

Previous value: -[
-  "regex_match",
-  "regex_no_match",
-  "min_length",
-  "max_length",
-  "contains_keywords",
-  "excludes_keywords",
-  "json_schema",
-  "cost_threshold"
-]New value: +[
+  "regex_match",
+  "regex_no_match",
+  "min_length",
+  "max_length",
+  "contains_keywords",
+  "excludes_keywords",
+  "json_schema",
+  "cost_threshold",
+  "action_policy"
+]

changedInput schema / properties / eval_type / description

Previous value: -"Rule bundle to apply: completeness | relevance | safety | cost | custom | all — picks which built-in rules fire. \"all\" runs every bundle in one call and adds a per-category breakdown. Defaults to \"all\" when omitted — every bundle runs, safety included, and the response carries a note saying the default ran"New value: +"Rule bundle: completeness | relevance | safety | cost | custom | all (the default, noted in the response)"

addedInput schema / properties / expected_trajectory

Added value: +{
+  "additionalProperties": false,
+  "description": "What the agent was expected to DO: tool_calls with mode and args (tool_sequence); step_budget and tolerance (step_budget)",
+  "properties": {
+    "args": {
+      "enum": [
+        "exact",
+        "subset"
+      ],
+      "type": "string"
+    },
+    "mode": {
+      "enum": [
+        "strict",
+        "unordered",
+        "subset",
+        "superset",
+        "ordered_subset"
+      ],
+      "type": "string"
+    },
+    "step_budget": {
+      "maximum": 9007199254740991,
+      "minimum": 1,
+      "type": "integer"
+    },
+    "tolerance": {
+      "minimum": 1,
+      "type": "number"
+    },
+    "tool_calls": {
+      "items": {
+        "additionalProperties": false,
+        "properties": {
+          "input": {},
+          "tool_name": {
+            "type": "string"
+          }
+        },
+        "required": [
+          "tool_name"
+        ],
+        "type": "object"
+      },
+      "maxItems": 500,
+      "type": "array"
+    }
+  },
+  "type": "object"
+}

changedInput schema / properties / input / description

Previous value: -"Original input for context (the ask + any source material the agent was given) — REQUIRED when eval_type=\"relevance\" (keyword_overlap and topic_consistency compare the output against it and skip without it); also grounds the safety bundle's hallucination signals"New value: +"The ask and any source material given — REQUIRED when eval_type=\"relevance\"; also grounds the hallucination signals"

changedInput schema / properties / output / description

Previous value: -"The output text to evaluate (the agent's response that gets scored against rules)"New value: +"The output text to evaluate (the agent's response that gets scored against rules). Omit it with trace_id to score the trace's stored output"

changedInput schema / properties / tool_calls / description

Previous value: -"What the agent DID — the tool calls it made, in order, each { tool_name, input?, output?, latency_ms?, error? } exactly as log_trace records them. Read by the trajectory rules — the rules that judge what the agent DID rather than what it wrote. Omit it and those rules SKIP rather than pass — an evaluation with no trajectory data reports \"not judged\", never \"clean\". When trace_id names a stored trace and this argument is omitted, the tool_calls stored on that trace are loaded and used, so a caller who already logged them need not resend them"New value: +"The tool calls the agent made, in order, as log_trace records them; omitted, the trajectory rules skip. Loaded from trace_id when stored"

addedInput schema / properties / tool_calls / items / properties / call_id
```
Added value: +{
+  "type": "string"
+}
```
addedInput schema / properties / tool_calls / items / properties / cost_usd
```
Added value: +{
+  "type": "number"
+}
```

addedInput schema / properties / tool_calls / items / properties / token_usage

Added value: +{
+  "properties": {
+    "completion_tokens": {
+      "type": "number"
+    },
+    "prompt_tokens": {
+      "type": "number"
+    },
+    "total_tokens": {
+      "type": "number"
+    }
+  },
+  "type": "object"
+}

addedInput schema / properties / tool_calls / items / properties / truncated
```
Added value: +{
+  "type": "boolean"
+}
```

addedInput schema / properties / tools

Added value: +{
+  "description": "Your MCP tools/list result, verbatim; lets the rules check call arguments. Loaded from trace_id when stored",
+  "items": {
+    "additionalProperties": false,
+    "properties": {
+      "_meta": {
+        "additionalProperties": {},
+        "propertyNames": {
+          "type": "string"
+        },
+        "type": "object"
+      },
+      "annotations": {
+        "additionalProperties": {},
+        "properties": {
+          "destructiveHint": {
+            "type": "boolean"
+          },
+          "idempotentHint": {
+            "type": "boolean"
+          },
+          "openWorldHint": {
+            "type": "boolean"
+          },
+          "readOnlyHint": {
+            "type": "boolean"
+          },
+          "title": {
+            "type": "string"
+          }
+        },
+        "type": "object"
+      },
+      "description": {
+        "maxLength": 4096,
+        "type": "string"
+      },
+      "execution": {
+        "additionalProperties": {},
+        "propertyNames": {
+          "type": "string"
+        },
+        "type": "object"
+      },
+      "inputSchema": {
+        "additionalProperties": {},
+        "propertyNames": {
+          "type": "string"
+        },
+        "type": "object"
+      },
+      "name": {
+        "maxLength": 128,
+        "minLength": 1,
+        "type": "string"
+      },
+      "outputSchema": {
+        "additionalProperties": {},
+        "propertyNames": {
+          "type": "string"
+        },
+        "type": "object"
+      },
+      "title": {
+        "maxLength": 512,
+        "type": "string"
+      }
+    },
+    "required": [
+      "name"
+    ],
+    "type": "object"
+  },
+  "maxItems": 200,
+  "type": "array"
+}

changedInput schema / properties / trace_id / description

Previous value: -"Link evaluation to a trace — surfaces this eval in the dashboard's trace drill-through and lets the tool reuse the trace's stored tool_calls. Must be the id of a stored trace (from log_trace / get_traces); an unknown id is rejected before anything is evaluated"New value: +"A stored trace (from log_trace / get_traces). Alone, it scores the trace as stored and the result is the trace's verdict. With output, input, tool_calls, tools, cost_usd or token_usage that differ from the record, or one eval_type, the result is kept beside the trace (reference_trace_id) and never replaces its verdict. An unknown id is rejected before anything is evaluated"

removedInput schema / required
```
Removed value: -[
-  "output"
-]
```

addedOutput schema / description

Added value: +"Nested evaluation fields: https://iris-eval.com/response-schema-v1.json"

removedOutput schema / properties / categories / additionalProperties / additionalProperties
```
Removed value: -{}
```

removedOutput schema / properties / categories / additionalProperties / properties

Removed value: -{
-  "critical_failures": {
-    "items": {
-      "type": "string"
-    },
-    "type": "array"
-  },
-  "critical_skipped": {
-    "items": {
-      "type": "string"
-    },
-    "type": "array"
-  },
-  "insufficient_data": {
-    "type": "boolean"
-  },
-  "passed": {
-    "type": [
-      "boolean",
-      "null"
-    ]
-  },
-  "rules_evaluated": {
-    "maximum": 9007199254740991,
-    "minimum": -9007199254740991,
-    "type": "integer"
-  },
-  "rules_skipped": {
-    "maximum": 9007199254740991,
-    "minimum": -9007199254740991,
-    "type": "integer"
-  },
-  "score": {
-    "type": [
-      "number",
-      "null"
-    ]
-  }
-}

removedOutput schema / properties / categories / additionalProperties / required

Removed value: -[
-  "score",
-  "passed",
-  "rules_evaluated",
-  "rules_skipped",
-  "insufficient_data"
-]

removedOutput schema / properties / categories / additionalProperties / type
```
Removed value: -"object"
```
removedOutput schema / properties / categories / description
```
Removed value: -"per-bundle verdicts for eval_type all"
```

removedOutput schema / properties / coverage / description

Removed value: -"per question: judged, unjudged and why, or not_applicable; plus the inputs carried"

removedOutput schema / properties / coverage / properties / dormant

Removed value: -{
-  "items": {
-    "additionalProperties": {},
-    "properties": {
-      "name": {
-        "type": "string"
-      },
-      "reason": {
-        "type": "string"
-      },
-      "ruleId": {
-        "type": "string"
-      }
-    },
-    "required": [
-      "ruleId",
-      "name",
-      "reason"
-    ],
-    "type": "object"
-  },
-  "type": "array"
-}

removedOutput schema / properties / coverage / properties / inputs

Removed value: -{
-  "additionalProperties": {
-    "type": "boolean"
-  },
-  "propertyNames": {
-    "type": "string"
-  },
-  "type": "object"
-}

removedOutput schema / properties / coverage / properties / questions

Removed value: -{
-  "items": {
-    "additionalProperties": {},
-    "properties": {
-      "id": {
-        "enum": [
-          "safe_output",
-          "grounded",
-          "complete",
-          "relevant",
-          "task_completed",
-          "tool_use_correct",
-          "within_budget"
-        ],
-        "type": "string"
-      },
-      "status": {
-        "enum": [
-          "judged",
-          "unjudged",
-          "not_applicable"
-        ],
-        "type": "string"
-      },
-      "why": {
-        "type": "string"
-      }
-    },
-    "required": [
-      "id",
-      "status"
-    ],
-    "type": "object"
-  },
-  "type": "array"
-}

removedOutput schema / properties / coverage / required
```
Removed value: -[
-  "inputs",
-  "questions"
-]
```
removedOutput schema / properties / critical_failures / description
```
Removed value: -"critical rules that failed and vetoed passed"
```
removedOutput schema / properties / critical_failures / items / type
```
Removed value: -"string"
```

removedOutput schema / properties / critical_skipped / description

Removed value: -"critical rules that could not judge; treat as unknown"

removedOutput schema / properties / critical_skipped / items / type
```
Removed value: -"string"
```
removedOutput schema / properties / erased_at / description
```
Removed value: -"set once the linked trace was deleted"
```
removedOutput schema / properties / eval_type / description
```
Removed value: -"the bundle that ran"
```

removedOutput schema / properties / id / description

Removed value: -"the evaluation id, readable at iris://evaluations/{id}"

removedOutput schema / properties / insufficient_data / description
```
Removed value: -"true when no rule could judge"
```

addedOutput schema / properties / interpretations

Added value: +{
+  "items": {},
+  "type": "array"
+}

removedOutput schema / properties / note / description

Removed value: -"present when eval_type was omitted"

removedOutput schema / properties / passed / description

Removed value: -"the ship verdict; false when nothing was judged"

removedOutput schema / properties / provenance / description

Removed value: -"Iris version, ruleset and config hashes, thresholds, corpus version, time"

removedOutput schema / properties / provenance / properties / configHash
```
Removed value: -{
-  "type": "string"
-}
```
removedOutput schema / properties / provenance / properties / corpusVersion
```
Removed value: -{
-  "type": "string"
-}
```
removedOutput schema / properties / provenance / properties / irisVersion
```
Removed value: -{
-  "type": "string"
-}
```
removedOutput schema / properties / provenance / properties / judgedAt
```
Removed value: -{
-  "type": "string"
-}
```
removedOutput schema / properties / provenance / properties / rulesetHash
```
Removed value: -{
-  "type": "string"
-}
```

removedOutput schema / properties / provenance / properties / thresholds

Removed value: -{
-  "additionalProperties": {},
-  "properties": {
-    "default": {
-      "type": "number"
-    },
-    "perRule": {
-      "additionalProperties": {},
-      "propertyNames": {
-        "type": "string"
-      },
-      "type": "object"
-    }
-  },
-  "required": [
-    "default"
-  ],
-  "type": "object"
-}

removedOutput schema / properties / provenance / required

Removed value: -[
-  "irisVersion",
-  "rulesetHash",
-  "configHash",
-  "thresholds",
-  "corpusVersion",
-  "judgedAt"
-]

addedOutput schema / properties / reference_trace_id
```
Added value: +{
+  "type": "string"
+}
```

removedOutput schema / properties / rule_results / description

Removed value: -"per rule: verdict, message, kind, role, question, saw, evidence, uncertainty"

removedOutput schema / properties / rule_results / items / additionalProperties
```
Removed value: -{}
```

removedOutput schema / properties / rule_results / items / properties

Removed value: -{
-  "budgetExceeded": {
-    "type": "boolean"
-  },
-  "category": {
-    "enum": [
-      "completeness",
-      "relevance",
-      "safety",
-      "cost",
-      "custom"
-    ],
-    "type": "string"
-  },
-  "classes": {
-    "items": {
-      "type": "string"
-    },
-    "type": "array"
-  },
-  "configInvalid": {
-    "type": "boolean"
-  },
-  "critical": {
-    "type": "boolean"
-  },
-  "criticalSource": {
-    "enum": [
-      "default",
-      "config"
-    ],
-    "type": "string"
-  },
-  "evidence": {
-    "items": {
-      "oneOf": [
-        {
-          "additionalProperties": {},
-          "properties": {
-            "end": {
-              "maximum": 9007199254740991,
-              "minimum": 0,
-              "type": "integer"
-            },
-            "label": {
-              "type": "string"
-            },
-            "source": {
-              "type": "string"
-            },
-            "start": {
-              "maximum": 9007199254740991,
-              "minimum": 0,
-              "type": "integer"
-            },
-            "type": {
-              "const": "span",
-              "type": "string"
-            }
-          },
-          "required": [
-            "type",
-            "source",
-            "start",
-            "end",
-            "label"
-          ],
-          "type": "object"
-        },
-        {
-          "additionalProperties": {},
-          "properties": {
-            "count": {
-              "maximum": 9007199254740991,
-              "minimum": 0,
-              "type": "integer"
-            },
-            "name": {
-              "type": "string"
-            },
-            "type": {
-              "const": "pattern",
-              "type": "string"
-            }
-          },
-          "required": [
-            "type",
-            "name",
-            "count"
-          ],
-          "type": "object"
-        },
-        {
-          "additionalProperties": {},
-          "properties": {
-            "index": {
-              "maximum": 9007199254740991,
-              "minimum": 0,
-              "type": "integer"
-            },
-            "label": {
-              "type": "string"
-            },
-            "toolName": {
-              "type": "string"
-            },
-            "type": {
-              "const": "toolCall",
-              "type": "string"
-            }
-          },
-          "required": [
-            "type",
-            "index",
-            "toolName",
-            "label"
-          ],
-          "type": "object"
-        },
-        {
-          "additionalProperties": {},
-          "properties": {
-            "status": {
-              "enum": [
-                "resolved",
-                "dead",
-                "unverifiable",
-                "supported",
-                "unsupported"
-              ],
-              "type": "string"
-            },
-            "type": {
-              "const": "citation",
-              "type": "string"
-            },
-            "url": {
-              "type": "string"
-            }
-          },
-          "required": [
-            "type",
-            "url",
-            "status"
-          ],
-          "type": "object"
-        },
-        {
-          "additionalProperties": {},
-          "properties": {
-            "stat": {
-              "type": "string"
-            },
-            "threshold": {
-              "type": "number"
-            },
-            "thresholdSource": {
-              "enum": [
-                "default",
-                "config",
-                "call",
-                "rule"
-              ],
-              "type": "string"
-            },
-            "type": {
-              "const": "count",
-              "type": "string"
-            },
-            "unit": {
-              "type": "string"
-            },
-            "value": {
-              "type": "number"
-            }
-          },
-          "required": [
-            "type",
-            "stat",
-            "unit",
-            "value"
-          ],
-          "type": "object"
-        }
-      ]
-    },
-    "type": "array"
-  },
-  "kind": {
-    "enum": [
-      "measurement",
-      "detection",
-      "inference",
-      "judgment",
-      "policy",
-      "verification"
-    ],
-    "type": "string"
-  },
-  "message": {
-    "type": "string"
-  },
-  "passed": {
-    "type": "boolean"
-  },
-  "question": {
-    "enum": [
-      "safe_output",
-      "grounded",
-      "complete",
-      "relevant",
-      "task_completed",
-      "tool_use_correct",
-      "within_budget"
-    ],
-    "type": "string"
-  },
-  "role": {
-    "enum": [
-      "gate",
-      "veto",
-      "risk",
-      "advisory",
-      "term"
-    ],
-    "type": "string"
-  },
-  "ruleId": {
-    "type": "string"
-  },
-  "ruleName": {
-    "type": "string"
-  },
-  "ruleVersion": {
-    "maximum": 9007199254740991,
-    "minimum": -9007199254740991,
-    "type": "integer"
-  },
-  "saw": {
-    "items": {
-      "enum": [
-        "output",
-        "input",
-        "expected",
-        "tool_calls",
-        "tool_outputs",
-        "tools_catalogue",
-        "cost",
-        "tokens",
-        "citations"
-      ],
-      "type": "string"
-    },
-    "type": "array"
-  },
-  "score": {
-    "type": "number"
-  },
-  "skipClass": {
-    "enum": [
-      "not_applicable",
-      "defeated",
-      "config_invalid"
-    ],
-    "type": "string"
-  },
-  "skipReason": {
-    "type": "string"
-  },
-  "skipped": {
-    "type": "boolean"
-  },
-  "uncertainty": {
-    "oneOf": [
-      {
-        "additionalProperties": {},
-        "properties": {
-          "basis": {
-            "const": "published_accuracy",
-            "type": "string"
-          },
-          "corpus": {
-            "additionalProperties": {},
-            "properties": {
-              "fn": {
-                "maximum": 9007199254740991,
-                "minimum": -9007199254740991,
-                "type": "integer"
-              },
-              "fp": {
-                "maximum": 9007199254740991,
-                "minimum": -9007199254740991,
-                "type": "integer"
-              },
-              "labelling": {
-                "enum": [
-                  "same-model",
-                  "human-verified"
-                ],
-                "type": "string"
-              },
-              "n": {
-                "maximum": 9007199254740991,
-                "minimum": -9007199254740991,
-                "type": "integer"
-              },
-              "release": {
-                "type": "string"
-              },
-              "tn": {
-                "maximum": 9007199254740991,
-                "minimum": -9007199254740991,
-                "type": "integer"
-              },
-              "tp": {
-                "maximum": 9007199254740991,
-                "minimum": -9007199254740991,
-                "type": "integer"
-              },
-              "version": {
-                "type": "string"
-              }
-            },
-            "required": [
-              "n",
-              "tp",
-              "fp",
-              "fn",
-              "tn",
-              "version",
-              "release",
-              "labelling"
-            ],
-            "type": "object"
-          },
-          "fired": {
-            "type": "boolean"
-          },
-          "missRate": {
-            "additionalProperties": {},
-            "properties": {
-              "hi": {
-                "type": "number"
-              },
-              "lo": {
-                "type": "number"
-              },
-              "point": {
-                "type": "number"
-              }
-            },
-            "required": [
-              "point",
-              "lo",
-              "hi"
-            ],
-            "type": "object"
-          },
-          "ppv": {
-            "additionalProperties": {},
-            "properties": {
-              "hi": {
-                "type": "number"
-              },
-              "lo": {
-                "type": "number"
-              },
-              "point": {
-                "type": "number"
-              }
-            },
-            "required": [
-              "point",
-              "lo",
-              "hi"
-            ],
-            "type": "object"
-          },
-          "prior": {
-            "additionalProperties": {},
-            "properties": {
-              "pi": {
-                "type": "number"
-              },
-              "source": {
-                "enum": [
-                  "default",
-                  "config",
-                  "estimated"
-                ],
-                "type": "string"
-              }
-            },
-            "required": [
-              "pi",
-              "source"
-            ],
-            "type": "object"
-          }
-        },
-        "required": [
-          "basis",
-          "fired",
-          "prior",
-          "corpus"
-        ],
-        "type": "object"
-      },
-      {
-        "additionalProperties": {},
-        "properties": {
-          "basis": {
-            "const": "definition",
-            "type": "string"
-          },
-          "conformance": {
-            "additionalProperties": {},
-            "properties": {
-              "matched": {
-                "maximum": 9007199254740991,
-                "minimum": -9007199254740991,
-                "type": "integer"
-              },
-              "n": {
-                "maximum": 9007199254740991,
-                "minimum": -9007199254740991,
-                "type": "integer"
-              }
-            },
-            "required": [
-              "n",
-              "matched"
-            ],
-            "type": "object"
-          }
-        },
-        "required": [
-          "basis",
-          "conformance"
-        ],
-        "type": "object"
-      },
-      {
-        "additionalProperties": {},
-        "properties": {
-          "basis": {
-            "const": "self_consistency",
-            "type": "string"
-          },
-          "samples": {
-            "maximum": 9007199254740991,
-            "minimum": -9007199254740991,
-            "type": "integer"
-          },
-          "scoreSd": {
-            "type": "number"
-          },
-          "voteFraction": {
-            "type": "number"
-          }
-        },
-        "required": [
-          "basis",
-          "samples",
-          "voteFraction",
-          "scoreSd"
-        ],
-        "type": "object"
-      },
-      {
-        "additionalProperties": {},
-        "properties": {
-          "basis": {
-            "const": "local_labels",
-            "type": "string"
-          },
-          "n": {
-            "maximum": 9007199254740991,
-            "minimum": -9007199254740991,
-            "type": "integer"
-          },
-          "precision": {
-            "additionalProperties": {},
-            "properties": {
-              "hi": {
-                "type": "number"
-              },
-              "lo": {
-                "type": "number"
-              },
-              "point": {
-                "type": "number"
-              }
-            },
-            "required": [
-              "point",
-              "lo",
-              "hi"
-            ],
-            "type": "object"
-          }
-        },
-        "required": [
-          "basis",
-          "precision",
-          "n"
-        ],
-        "type": "object"
-      },
-      {
-        "additionalProperties": {},
-        "properties": {
-          "basis": {
-            "const": "policy",
-            "type": "string"
-          }
-        },
-        "required": [
-          "basis"
-        ],
-        "type": "object"
-      },
-      {
-        "additionalProperties": {},
-        "properties": {
-          "basis": {
-            "const": "unmeasured",
-            "type": "string"
-          },
-          "why": {
-            "type": "string"
-          }
-        },
-        "required": [
-          "basis",
-          "why"
-        ],
-        "type": "object"
-      }
-    ]
-  },
-  "value": {
-    "additionalProperties": {},
-    "properties": {
-      "stat": {
-        "type": "string"
-      },
-      "unit": {
-        "type": "string"
-      },
-      "value": {
-        "type": "number"
-      }
-    },
-    "required": [
-      "stat",
-      "unit",
-      "value"
-    ],
-    "type": "object"
-  }
-}

removedOutput schema / properties / rule_results / items / required

Removed value: -[
-  "ruleName",
-  "passed",
-  "score",
-  "message"
-]

removedOutput schema / properties / rule_results / items / type
```
Removed value: -"object"
```

addedOutput schema / properties / rules_changed

Added value: +{
+  "additionalProperties": {},
+  "properties": {},
+  "type": "object"
+}

removedOutput schema / properties / rules_evaluated / description
```
Removed value: -"rules that judged"
```
removedOutput schema / properties / rules_evaluated / maximum
```
Removed value: -9007199254740991
```
removedOutput schema / properties / rules_evaluated / minimum
```
Removed value: --9007199254740991
```
changedOutput schema / properties / rules_evaluated / type
```
Previous value: -"integer"New value: +"number"
```
removedOutput schema / properties / rules_skipped / description
```
Removed value: -"rules that skipped"
```
removedOutput schema / properties / rules_skipped / maximum
```
Removed value: -9007199254740991
```
removedOutput schema / properties / rules_skipped / minimum
```
Removed value: --9007199254740991
```
changedOutput schema / properties / rules_skipped / type
```
Previous value: -"integer"New value: +"number"
```
addedOutput schema / properties / run_id
```
Added value: +{
+  "type": "string"
+}
```

removedOutput schema / properties / score / description

Removed value: -"0..1 weighted quality over the rules that ran"

removedOutput schema / properties / suggestions

Removed value: -{
-  "description": "what to change",
-  "items": {
-    "type": "string"
-  },
-  "type": "array"
-}

removedOutput schema / properties / trace_id / description
```
Removed value: -"the linked trace, when named"
```

removedOutput schema / properties / verdict / description

Removed value: -"state, passed, basis (which layer decided), by (the rules), risk"

removedOutput schema / properties / verdict / properties / basis

Removed value: -{
-  "enum": [
-    "policy_gate",
-    "detector_veto",
-    "critical_unknown",
-    "required_evidence_missing",
-    "risk_over_loss",
-    "score_below_threshold",
-    "clean",
-    "no_rules"
-  ],
-  "type": "string"
-}

removedOutput schema / properties / verdict / properties / by

Removed value: -{
-  "items": {
-    "type": "string"
-  },
-  "type": "array"
-}

removedOutput schema / properties / verdict / properties / confidence

Removed value: -{
-  "enum": [
-    "decisive",
-    "marginal"
-  ],
-  "type": "string"
-}

removedOutput schema / properties / verdict / properties / passed
```
Removed value: -{
-  "type": "boolean"
-}
```

removedOutput schema / properties / verdict / properties / risk

Removed value: -{
-  "anyOf": [
-    {
-      "additionalProperties": {},
-      "properties": {
-        "hi": {
-          "type": "number"
-        },
-        "lo": {
-          "type": "number"
-        },
-        "pBad": {
-          "type": "number"
-        }
-      },
-      "required": [
-        "pBad",
-        "lo",
-        "hi"
-      ],
-      "type": "object"
-    },
-    {
-      "type": "null"
-    }
-  ]
-}

removedOutput schema / properties / verdict / properties / state

Removed value: -{
-  "enum": [
-    "pass",
-    "fail",
-    "unknown"
-  ],
-  "type": "string"
-}

removedOutput schema / properties / verdict / required

Removed value: -[
-  "state",
-  "passed",
-  "basis",
-  "by",
-  "risk"
-]

changedOutput schema / required

Previous value: -[
-  "id",
-  "eval_type",
-  "score",
-  "passed",
-  "rule_results",
-  "suggestions",
-  "rules_evaluated",
-  "rules_skipped",
-  "insufficient_data"
-]New value: +[
+  "id",
+  "eval_type",
+  "score",
+  "passed",
+  "rule_results",
+  "rules_evaluated",
+  "rules_skipped",
+  "insufficient_data"
+]

Changed2 schema fields changedv0.9.0Sep 5, 2026

changedInput schema / properties / trace_id / description

Previous value: -"Link evaluation to a trace — surfaces this eval in the dashboard's trace drill-through. Must be the id of a stored trace (from log_trace / get_traces); an unknown id is rejected before anything is evaluated"New value: +"Link evaluation to a trace — surfaces this eval in the dashboard's trace drill-through and lets the tool reuse the trace's stored tool_calls. Must be the id of a stored trace (from log_trace / get_traces); an unknown id is rejected before anything is evaluated"

changedOutput schema / (root)

Previous value: -nullNew value: +{
+  "$schema": "http://json-schema.org/draft-07/schema#",
+  "additionalProperties": {},
+  "properties": {
+    "categories": {
+      "additionalProperties": {
+        "additionalProperties": {},
+        "properties": {
+          "critical_failures": {
+            "items": {
+              "type": "string"
+            },
+            "type": "array"
+          },
+          "critical_skipped": {
+            "items": {
+              "type": "string"
+            },
+            "type": "array"
+          },
+          "insufficient_data": {
+            "type": "boolean"
+          },
+          "passed": {
+            "type": [
+              "boolean",
+              "null"
+            ]
+          },
+          "rules_evaluated": {
+            "maximum": 9007199254740991,
+            "minimum": -9007199254740991,
+            "type": "integer"
+          },
+          "rules_skipped": {
+            "maximum": 9007199254740991,
+            "minimum": -9007199254740991,
+            "type": "integer"
+          },
+          "score": {
+            "type": [
+              "number",
+              "null"
+            ]
+          }
+        },
+        "required": [
+          "score",
+          "passed",
+          "rules_evaluated",
+          "rules_skipped",
+          "insufficient_data"
+        ],
+        "type": "object"
+      },
+      "description": "per-bundle verdicts for eval_type all",
+      "propertyNames": {
+        "type": "string"
+      },
+      "type": "object"
+    },
+    "coverage": {
+      "additionalProperties": {},
+      "description": "per question: judged, unjudged and why, or not_applicable; plus the inputs carried",
+      "properties": {
+        "dormant": {
+          "items": {
+            "additionalProperties": {},
+            "properties": {
+              "name": {
+                "type": "string"
+              },
+              "reason": {
+                "type": "string"
+              },
+              "ruleId": {
+                "type": "string"
+              }
+            },
+            "required": [
+              "ruleId",
+              "name",
+              "reason"
+            ],
+            "type": "object"
+          },
+          "type": "array"
+        },
+        "inputs": {
+          "additionalProperties": {
+            "type": "boolean"
+          },
+          "propertyNames": {
+            "type": "string"
+          },
+          "type": "object"
+        },
+        "questions": {
+          "items": {
+            "additionalProperties": {},
+            "properties": {
+              "id": {
+                "enum": [
+                  "safe_output",
+                  "grounded",
+                  "complete",
+                  "relevant",
+                  "task_completed",
+                  "tool_use_correct",
+                  "within_budget"
+                ],
+                "type": "string"
+              },
+              "status": {
+                "enum": [
+                  "judged",
+                  "unjudged",
+                  "not_applicable"
+                ],
+                "type": "string"
+              },
+              "why": {
+                "type": "string"
+              }
+            },
+            "required": [
+              "id",
+              "status"
+            ],
+            "type": "object"
+          },
+          "type": "array"
+        }
+      },
+      "required": [
+        "inputs",
+        "questions"
+      ],
+      "type": "object"
+    },
+    "critical_failures": {
+      "description": "critical rules that failed and vetoed passed",
+      "items": {
+        "type": "string"
+      },
+      "type": "array"
+    },
+    "critical_skipped": {
+      "description": "critical rules that could not judge; treat as unknown",
+      "items": {
+        "type": "string"
+      },
+      "type": "array"
+    },
+    "erased_at": {
+      "description": "set once the linked trace was deleted",
+      "type": "string"
+    },
+    "eval_type": {
+      "description": "the bundle that ran",
+      "enum": [
+        "completeness",
+        "relevance",
+        "safety",
+        "cost",
+        "custom",
+        "all"
+      ],
+      "type": "string"
+    },
+    "id": {
+      "description": "the evaluation id, readable at iris://evaluations/{id}",
+      "type": "string"
+    },
+    "insufficient_data": {
+      "description": "true when no rule could judge",
+      "type": "boolean"
+    },
+    "note": {
+      "description": "present when eval_type was omitted",
+      "type": "string"
+    },
+    "passed": {
+      "description": "the ship verdict; false when nothing was judged",
+      "type": "boolean"
+    },
+    "provenance": {
+      "additionalProperties": {},
+      "description": "Iris version, ruleset and config hashes, thresholds, corpus version, time",
+      "properties": {
+        "configHash": {
+          "type": "string"
+        },
+        "corpusVersion": {
+          "type": "string"
+        },
+        "irisVersion": {
+          "type": "string"
+        },
+        "judgedAt": {
+          "type": "string"
+        },
+        "rulesetHash": {
+          "type": "string"
+        },
+        "thresholds": {
+          "additionalProperties": {},
+          "properties": {
+            "default": {
+              "type": "number"
+            },
+            "perRule": {
+              "additionalProperties": {},
+              "propertyNames": {
+                "type": "string"
+              },
+              "type": "object"
+            }
+          },
+          "required": [
+            "default"
+          ],
+          "type": "object"
+        }
+      },
+      "required": [
+        "irisVersion",
+        "rulesetHash",
+        "configHash",
+        "thresholds",
+        "corpusVersion",
+        "judgedAt"
+      ],
+      "type": "object"
+    },
+    "rule_results": {
+      "description": "per rule: verdict, message, kind, role, question, saw, evidence, uncertainty",
+      "items": {
+        "additionalProperties": {},
+        "properties": {
+          "budgetExceeded": {
+            "type": "boolean"
+          },
+          "category": {
+            "enum": [
+              "completeness",
+              "relevance",
+              "safety",
+              "cost",
+              "custom"
+            ],
+            "type": "string"
+          },
+          "classes": {
+            "items": {
+              "type": "string"
+            },
+            "type": "array"
+          },
+          "configInvalid": {
+            "type": "boolean"
+          },
+          "critical": {
+            "type": "boolean"
+          },
+          "criticalSource": {
+            "enum": [
+              "default",
+              "config"
+            ],
+            "type": "string"
+          },
+          "evidence": {
+            "items": {
+              "oneOf": [
+                {
+                  "additionalProperties": {},
+                  "properties": {
+                    "end": {
+                      "maximum": 9007199254740991,
+                      "minimum": 0,
+                      "type": "integer"
+                    },
+                    "label": {
+                      "type": "string"
+                    },
+                    "source": {
+                      "type": "string"
+                    },
+                    "start": {
+                      "maximum": 9007199254740991,
+                      "minimum": 0,
+                      "type": "integer"
+                    },
+                    "type": {
+                      "const": "span",
+                      "type": "string"
+                    }
+                  },
+                  "required": [
+                    "type",
+                    "source",
+                    "start",
+                    "end",
+                    "label"
+                  ],
+                  "type": "object"
+                },
+                {
+                  "additionalProperties": {},
+                  "properties": {
+                    "count": {
+                      "maximum": 9007199254740991,
+                      "minimum": 0,
+                      "type": "integer"
+                    },
+                    "name": {
+                      "type": "string"
+                    },
+                    "type": {
+                      "const": "pattern",
+                      "type": "string"
+                    }
+                  },
+                  "required": [
+                    "type",
+                    "name",
+                    "count"
+                  ],
+                  "type": "object"
+                },
+                {
+                  "additionalProperties": {},
+                  "properties": {
+                    "index": {
+                      "maximum": 9007199254740991,
+                      "minimum": 0,
+                      "type": "integer"
+                    },
+                    "label": {
+                      "type": "string"
+                    },
+                    "toolName": {
+                      "type": "string"
+                    },
+                    "type": {
+                      "const": "toolCall",
+                      "type": "string"
+                    }
+                  },
+                  "required": [
+                    "type",
+                    "index",
+                    "toolName",
+                    "label"
+                  ],
+                  "type": "object"
+                },
+                {
+                  "additionalProperties": {},
+                  "properties": {
+                    "status": {
+                      "enum": [
+                        "resolved",
+                        "dead",
+                        "unverifiable",
+                        "supported",
+                        "unsupported"
+                      ],
+                      "type": "string"
+                    },
+                    "type": {
+                      "const": "citation",
+                      "type": "string"
+                    },
+                    "url": {
+                      "type": "string"
+                    }
+                  },
+                  "required": [
+                    "type",
+                    "url",
+                    "status"
+                  ],
+                  "type": "object"
+                },
+                {
+                  "additionalProperties": {},
+                  "properties": {
+                    "stat": {
+                      "type": "string"
+                    },
+                    "threshold": {
+                      "type": "number"
+                    },
+                    "thresholdSource": {
+                      "enum": [
+                        "default",
+                        "config",
+                        "call",
+                        "rule"
+                      ],
+                      "type": "string"
+                    },
+                    "type": {
+                      "const": "count",
+                      "type": "string"
+                    },
+                    "unit": {
+                      "type": "string"
+                    },
+                    "value": {
+                      "type": "number"
+                    }
+                  },
+                  "required": [
+                    "type",
+                    "stat",
+                    "unit",
+                    "value"
+                  ],
+                  "type": "object"
+                }
+              ]
+            },
+            "type": "array"
+          },
+          "kind": {
+            "enum": [
+              "measurement",
+              "detection",
+              "inference",
+              "judgment",
+              "policy",
+              "verification"
+            ],
+            "type": "string"
+          },
+          "message": {
+            "type": "string"
+          },
+          "passed": {
+            "type": "boolean"
+          },
+          "question": {
+            "enum": [
+              "safe_output",
+              "grounded",
+              "complete",
+              "relevant",
+              "task_completed",
+              "tool_use_correct",
+              "within_budget"
+            ],
+            "type": "string"
+          },
+          "role": {
+            "enum": [
+              "gate",
+              "veto",
+              "risk",
+              "advisory",
+              "term"
+            ],
+            "type": "string"
+          },
+          "ruleId": {
+            "type": "string"
+          },
+          "ruleName": {
+            "type": "string"
+          },
+          "ruleVersion": {
+            "maximum": 9007199254740991,
+            "minimum": -9007199254740991,
+            "type": "integer"
+          },
+          "saw": {
+            "items": {
+              "enum": [
+                "output",
+                "input",
+                "expected",
+                "tool_calls",
+                "tool_outputs",
+                "tools_catalogue",
+                "cost",
+                "tokens",
+                "citations"
+              ],
+              "type": "string"
+            },
+            "type": "array"
+          },
+          "score": {
+            "type": "number"
+          },
+          "skipClass": {
+            "enum": [
+              "not_applicable",
+              "defeated",
+              "config_invalid"
+            ],
+            "type": "string"
+          },
+          "skipReason": {
+            "type": "string"
+          },
+          "skipped": {
+            "type": "boolean"
+          },
+          "uncertainty": {
+            "oneOf": [
+              {
+                "additionalProperties": {},
+                "properties": {
+                  "basis": {
+                    "const": "published_accuracy",
+                    "type": "string"
+                  },
+                  "corpus": {
+                    "additionalProperties": {},
+                    "properties": {
+                      "fn": {
+                        "maximum": 9007199254740991,
+                        "minimum": -9007199254740991,
+                        "type": "integer"
+                      },
+                      "fp": {
+                        "maximum": 9007199254740991,
+                        "minimum": -9007199254740991,
+                        "type": "integer"
+                      },
+                      "labelling": {
+                        "enum": [
+                          "same-model",
+                          "human-verified"
+                        ],
+                        "type": "string"
+                      },
+                      "n": {
+                        "maximum": 9007199254740991,
+                        "minimum": -9007199254740991,
+                        "type": "integer"
+                      },
+                      "release": {
+                        "type": "string"
+                      },
+                      "tn": {
+                        "maximum": 9007199254740991,
+                        "minimum": -9007199254740991,
+                        "type": "integer"
+                      },
+                      "tp": {
+                        "maximum": 9007199254740991,
+                        "minimum": -9007199254740991,
+                        "type": "integer"
+                      },
+                      "version": {
+                        "type": "string"
+                      }
+                    },
+                    "required": [
+                      "n",
+                      "tp",
+                      "fp",
+                      "fn",
+                      "tn",
+                      "version",
+                      "release",
+                      "labelling"
+                    ],
+                    "type": "object"
+                  },
+                  "fired": {
+                    "type": "boolean"
+                  },
+                  "missRate": {
+                    "additionalProperties": {},
+                    "properties": {
+                      "hi": {
+                        "type": "number"
+                      },
+                      "lo": {
+                        "type": "number"
+                      },
+                      "point": {
+                        "type": "number"
+                      }
+                    },
+                    "required": [
+                      "point",
+                      "lo",
+                      "hi"
+                    ],
+                    "type": "object"
+                  },
+                  "ppv": {
+                    "additionalProperties": {},
+                    "properties": {
+                      "hi": {
+                        "type": "number"
+                      },
+                      "lo": {
+                        "type": "number"
+                      },
+                      "point": {
+                        "type": "number"
+                      }
+                    },
+                    "required": [
+                      "point",
+                      "lo",
+                      "hi"
+                    ],
+                    "type": "object"
+                  },
+                  "prior": {
+                    "additionalProperties": {},
+                    "properties": {
+                      "pi": {
+                        "type": "number"
+                      },
+                      "source": {
+                        "enum": [
+                          "default",
+                          "config",
+                          "estimated"
+                        ],
+                        "type": "string"
+                      }
+                    },
+                    "required": [
+                      "pi",
+                      "source"
+                    ],
+                    "type": "object"
+                  }
+                },
+                "required": [
+                  "basis",
+                  "fired",
+                  "prior",
+                  "corpus"
+                ],
+                "type": "object"
+              },
+              {
+                "additionalProperties": {},
+                "properties": {
+                  "basis": {
+                    "const": "definition",
+                    "type": "string"
+                  },
+                  "conformance": {
+                    "additionalProperties": {},
+                    "properties": {
+                      "matched": {
+                        "maximum": 9007199254740991,
+                        "minimum": -9007199254740991,
+                        "type": "integer"
+                      },
+                      "n": {
+                        "maximum": 9007199254740991,
+                        "minimum": -9007199254740991,
+                        "type": "integer"
+                      }
+                    },
+                    "required": [
+                      "n",
+                      "matched"
+                    ],
+                    "type": "object"
+                  }
+                },
+                "required": [
+                  "basis",
+                  "conformance"
+                ],
+                "type": "object"
+              },
+              {
+                "additionalProperties": {},
+                "properties": {
+                  "basis": {
+                    "const": "self_consistency",
+                    "type": "string"
+                  },
+                  "samples": {
+                    "maximum": 9007199254740991,
+                    "minimum": -9007199254740991,
+                    "type": "integer"
+                  },
+                  "scoreSd": {
+                    "type": "number"
+                  },
+                  "voteFraction": {
+                    "type": "number"
+                  }
+                },
+                "required": [
+                  "basis",
+                  "samples",
+                  "voteFraction",
+                  "scoreSd"
+                ],
+                "type": "object"
+              },
+              {
+                "additionalProperties": {},
+                "properties": {
+                  "basis": {
+                    "const": "local_labels",
+                    "type": "string"
+                  },
+                  "n": {
+                    "maximum": 9007199254740991,
+                    "minimum": -9007199254740991,
+                    "type": "integer"
+                  },
+                  "precision": {
+                    "additionalProperties": {},
+                    "properties": {
+                      "hi": {
+                        "type": "number"
+                      },
+                      "lo": {
+                        "type": "number"
+                      },
+                      "point": {
+                        "type": "number"
+                      }
+                    },
+                    "required": [
+                      "point",
+                      "lo",
+                      "hi"
+                    ],
+                    "type": "object"
+                  }
+                },
+                "required": [
+                  "basis",
+                  "precision",
+                  "n"
+                ],
+                "type": "object"
+              },
+              {
+                "additionalProperties": {},
+                "properties": {
+                  "basis": {
+                    "const": "policy",
+                    "type": "string"
+                  }
+                },
+                "required": [
+                  "basis"
+                ],
+                "type": "object"
+              },
+              {
+                "additionalProperties": {},
+                "properties": {
+                  "basis": {
+                    "const": "unmeasured",
+                    "type": "string"
+                  },
+                  "why": {
+                    "type": "string"
+                  }
+                },
+                "required": [
+                  "basis",
+                  "why"
+                ],
+                "type": "object"
+              }
+            ]
+          },
+          "value": {
+            "additionalProperties": {},
+            "properties": {
+              "stat": {
+                "type": "string"
+              },
+              "unit": {
+                "type": "string"
+              },
+              "value": {
+                "type": "number"
+              }
+            },
+            "required": [
+              "stat",
+              "unit",
+              "value"
+            ],
+            "type": "object"
+          }
+        },
+        "required": [
+          "ruleName",
+          "passed",
+          "score",
+          "message"
+        ],
+        "type": "object"
+      },
+      "type": "array"
+    },
+    "rules_evaluated": {
+      "description": "rules that judged",
+      "maximum": 9007199254740991,
+      "minimum": -9007199254740991,
+      "type": "integer"
+    },
+    "rules_skipped": {
+      "description": "rules that skipped",
+      "maximum": 9007199254740991,
+      "minimum": -9007199254740991,
+      "type": "integer"
+    },
+    "score": {
+      "description": "0..1 weighted quality over the rules that ran",
+      "type": "number"
+    },
+    "suggestions": {
+      "description": "what to change",
+      "items": {
+        "type": "string"
+      },
+      "type": "array"
+    },
+    "trace_id": {
+      "description": "the linked trace, when named",
+      "type": "string"
+    },
+    "verdict": {
+      "additionalProperties": {},
+      "description": "state, passed, basis (which layer decided), by (the rules), risk",
+      "properties": {
+        "basis": {
+          "enum": [
+            "policy_gate",
+            "detector_veto",
+            "critical_unknown",
+            "required_evidence_missing",
+            "risk_over_loss",
+            "score_below_threshold",
+            "clean",
+            "no_rules"
+          ],
+          "type": "string"
+        },
+        "by": {
+          "items": {
+            "type": "string"
+          },
+          "type": "array"
+        },
+        "confidence": {
+          "enum": [
+            "decisive",
+            "marginal"
+          ],
+          "type": "string"
+        },
+        "passed": {
+          "type": "boolean"
+        },
+        "risk": {
+          "anyOf": [
+            {
+              "additionalProperties": {},
+              "properties": {
+                "hi": {
+                  "type": "number"
+                },
+                "lo": {
+                  "type": "number"
+                },
+                "pBad": {
+                  "type": "number"
+                }
+              },
+              "required": [
+                "pBad",
+                "lo",
+                "hi"
+              ],
+              "type": "object"
+            },
+            {
+              "type": "null"
+            }
+          ]
+        },
+        "state": {
+          "enum": [
+            "pass",
+            "fail",
+            "unknown"
+          ],
+          "type": "string"
+        }
+      },
+      "required": [
+        "state",
+        "passed",
+        "basis",
+        "by",
+        "risk"
+      ],
+      "type": "object"
+    }
+  },
+  "required": [
+    "id",
+    "eval_type",
+    "score",
+    "passed",
+    "rule_results",
+    "suggestions",
+    "rules_evaluated",
+    "rules_skipped",
+    "insufficient_data"
+  ],
+  "type": "object"
+}

Changed19 schema fields changedv0.8.0Sep 4, 2026

addedInput schema / additionalProperties
```
Added value: +false
```

changedInput schema / properties / cost_usd / description

Previous value: -"Cost in USD — only consulted when eval_type=\"cost\" (compared against cost_threshold rules)"New value: +"Cost in USD — consulted by the cost bundle (eval_type=\"cost\" or \"all\") AND by any cost_threshold custom rule regardless of eval_type; omit it and such a rule skips rather than passes (a critical one is listed in critical_skipped)"

changedInput schema / properties / custom_rules / description

Previous value: -"Custom evaluation rules — fires REGARDLESS of eval_type; pass eval_type=\"custom\" if you want ONLY these"New value: +"Custom evaluation rules, max 10 per call (deploy persistent rule sets via deploy_rule instead) — fires REGARDLESS of eval_type; pass eval_type=\"custom\" if you want ONLY these. Each entry accepts exactly name, type, config, weight — an unknown key (e.g. a misspelled weight) is rejected"

addedInput schema / properties / custom_rules / items / additionalProperties
```
Added value: +false
```

addedInput schema / properties / custom_rules / items / properties / config / description

Added value: +"Check configuration; keys depend on type (pattern, min_length, keywords, max_cost, …)"

addedInput schema / properties / custom_rules / items / properties / name / description
```
Added value: +"Rule name as it will appear in rule_results"
```
addedInput schema / properties / custom_rules / items / properties / name / minLength
```
Added value: +1
```
addedInput schema / properties / custom_rules / items / properties / type / description
```
Added value: +"Check type — decides which config keys the rule reads"
```
addedInput schema / properties / custom_rules / items / properties / weight / description
```
Added value: +"Weight in the weighted score (default 1; must be > 0)"
```
addedInput schema / properties / custom_rules / items / properties / weight / exclusiveMinimum
```
Added value: +0
```
addedInput schema / properties / custom_rules / maxItems
```
Added value: +10
```
removedInput schema / properties / eval_type / default
```
Removed value: -"completeness"
```

changedInput schema / properties / eval_type / description

Previous value: -"Rule bundle to apply: completeness | relevance | safety | cost | custom — picks which built-in rules fire"New value: +"Rule bundle to apply: completeness | relevance | safety | cost | custom | all — picks which built-in rules fire. \"all\" runs every bundle in one call and adds a per-category breakdown. Defaults to \"all\" when omitted — every bundle runs, safety included, and the response carries a note saying the default ran"

changedInput schema / properties / eval_type / enum

Previous value: -[
-  "completeness",
-  "relevance",
-  "safety",
-  "cost",
-  "custom"
-]New value: +[
+  "completeness",
+  "relevance",
+  "safety",
+  "cost",
+  "custom",
+  "all"
+]

changedInput schema / properties / expected / description

Previous value: -"Expected output for comparison — REQUIRED when eval_type=\"relevance\" (used as keyword-overlap target)"New value: +"Expected output for comparison — consulted only by the completeness bundle's expected_coverage rule; NOT used by relevance (the relevance rules compare the output against `input`)"

changedInput schema / properties / input / description

Previous value: -"Original input for context — improves relevance scoring (keyword overlap vs input)"New value: +"Original input for context (the ask + any source material the agent was given) — REQUIRED when eval_type=\"relevance\" (keyword_overlap and topic_consistency compare the output against it and skip without it); also grounds the safety bundle's hallucination signals"

changedInput schema / properties / token_usage / description

Previous value: -"Token usage breakdown — only consulted when eval_type=\"cost\" (used for token-budget rules)"New value: +"Token usage breakdown — only consulted by the cost bundle (eval_type=\"cost\" or \"all\"; used for token-budget rules)"

addedInput schema / properties / tool_calls

Added value: +{
+  "description": "What the agent DID — the tool calls it made, in order, each { tool_name, input?, output?, latency_ms?, error? } exactly as log_trace records them. Read by the trajectory rules — the rules that judge what the agent DID rather than what it wrote. Omit it and those rules SKIP rather than pass — an evaluation with no trajectory data reports \"not judged\", never \"clean\". When trace_id names a stored trace and this argument is omitted, the tool_calls stored on that trace are loaded and used, so a caller who already logged them need not resend them",
+  "items": {
+    "additionalProperties": false,
+    "properties": {
+      "error": {
+        "type": "string"
+      },
+      "input": {},
+      "latency_ms": {
+        "type": "number"
+      },
+      "output": {},
+      "tool_name": {
+        "type": "string"
+      }
+    },
+    "required": [
+      "tool_name"
+    ],
+    "type": "object"
+  },
+  "type": "array"
+}

changedInput schema / properties / trace_id / description

Previous value: -"Link evaluation to a trace — surfaces this eval in the dashboard's trace drill-through"New value: +"Link evaluation to a trace — surfaces this eval in the dashboard's trace drill-through. Must be the id of a stored trace (from log_trace / get_traces); an unknown id is rejected before anything is evaluated"

Changed4 schema fields changedv0.4.6Aug 11, 2026
- removedInput schema / additionalProperties
```
Removed value: -false
```
- removedInput schema / properties / custom_rules / items / additionalProperties
```
Removed value: -false
```
- addedInput schema / properties / custom_rules / items / properties / config / propertyNames
```
Added value: +{
+  "type": "string"
+}
```
- removedInput schema / properties / token_usage / additionalProperties
```
Removed value: -false
```

Changed8 schema fields changedv0.1.10May 3, 2026

changedInput schema / properties / cost_usd / description

Previous value: -"Cost for cost evaluation"New value: +"Cost in USD — only consulted when eval_type=\"cost\" (compared against cost_threshold rules)"

changedInput schema / properties / custom_rules / description

Previous value: -"Custom evaluation rules"New value: +"Custom evaluation rules — fires REGARDLESS of eval_type; pass eval_type=\"custom\" if you want ONLY these"

changedInput schema / properties / eval_type / description

Previous value: -"Type of evaluation"New value: +"Rule bundle to apply: completeness | relevance | safety | cost | custom — picks which built-in rules fire"

changedInput schema / properties / expected / description

Previous value: -"Expected output for comparison"New value: +"Expected output for comparison — REQUIRED when eval_type=\"relevance\" (used as keyword-overlap target)"

changedInput schema / properties / input / description

Previous value: -"Original input for context"New value: +"Original input for context — improves relevance scoring (keyword overlap vs input)"

changedInput schema / properties / output / description

Previous value: -"The output text to evaluate"New value: +"The output text to evaluate (the agent's response that gets scored against rules)"

changedInput schema / properties / token_usage / description

Previous value: -"Token usage for cost evaluation"New value: +"Token usage breakdown — only consulted when eval_type=\"cost\" (used for token-budget rules)"

changedInput schema / properties / trace_id / description

Previous value: -"Link evaluation to a trace"New value: +"Link evaluation to a trace — surfaces this eval in the dashboard's trace drill-through"

Addedv0.1.8Apr 14, 2026
Removedv0.1.7Apr 7, 2026
First observedMar 22, 2026

TDQS

A4.8/5.0

Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Beyond annotations, it discloses important behavior: missing rule inputs skip rather than pass, input is required for relevance, trace_id alone scores the stored trace, and alternate inputs create a side result rather than replacing the trace verdict. It also names error codes and says failures return isError with a recovery path.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is well structured and front-loads core behavior, alternatives, returns, errors, and siblings. The long return-field list is somewhat redundant given the output schema, but it remains compact and scannable.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For an 11-parameter tool with nested objects, the description provides enough context on purpose, alternatives, required conditions, error handling, and outcome shape. The schema and output schema handle detailed field structure, while the description fills in operational semantics.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is already 100%, so the schema carries most parameter documentation. The description still adds meaningful cross-parameter semantics, especially the relevance input requirement, default eval_type behavior, skip behavior for omitted tool_calls or cost_usd, and how trace_id changes result persistence.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with a specific verb and resource: score an output with deterministic rules, producing a ship verdict, per-rule evidence, and unjudged items. It clearly distinguishes deterministic rule evaluation from semantic judgment by naming evaluate_with_llm_judge as the alternative.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It gives explicit when-not-to-use guidance for semantic judgment and input-firewall use, and explains the eval_type bundle behavior, the relevance input requirement, and trace_id scoring semantics. It also routes the agent to log_trace and list_rules with concrete conditions.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Deploy Server

Evaluate Output

Instructions

Input Schema

Output Schema

Schema Changelog

TDQS

Other Tools