evalsmith-mcp

Detect prompt regression

detect_prompt_regression

Read-only

Detect prompt regressions by comparing baseline and candidate outputs against golden-set deterministic checks before shipping prompt or model changes.

Instructions

Run a golden set's deterministic checks (exact, regex, contains, JSON Schema, length) against baseline and candidate outputs. A regression is a case that passes on baseline and fails on candidate. An LLM judge can be enabled per call for a second opinion; the verdict only depends on the checks. Use it before shipping a prompt or model change.

Input Schema

Name	Required	Description
`judge`	No	Optional pairwise LLM judge on the server's configured OpenAI-compatible endpoint (e.g. Ollama). Off by default; it never changes the verdict.
`golden`	Yes	Golden cases with deterministic checks.
`baseline`	Yes	Outputs of the current prompt/model, keyed by case id.
`candidate`	Yes	Outputs of the proposed prompt/model, keyed by case id.

Output Schema

Name	Required	Description
`cases`	Yes
`judge`	No
`perCase`	Yes
`summary`	Yes
`verdict`	Yes
`regressions`	Yes	Cases that pass on baseline and fail on candidate.
`improvements`	Yes	Cases that fail on baseline and pass on candidate.
`baselinePassRate`	Yes
`candidatePassRate`	Yes

Schema Changelog

Changes observed during successful MCP inspections.

Changed5 schema fields changedOct 29, 2026

addedInput schema / properties / golden / items / properties / checks / description
```
Added value: +"Checks the output must pass, all of them."
```

changedInput schema / properties / golden / items / properties / checks / items / oneOf

Previous value: -[
-  {
-    "properties": {
-      "trim": {
-        "default": true,
-        "type": "boolean"
-      },
-      "type": {
-        "const": "exact",
-        "type": "string"
-      },
-      "value": {
-        "type": "string"
-      }
-    },
-    "required": [
-      "type",
-      "value"
-    ],
-    "type": "object"
-  },
-  {
-    "properties": {
-      "flags": {
-        "type": "string"
-      },
-      "pattern": {
-        "type": "string"
-      },
-      "type": {
-        "const": "regex",
-        "type": "string"
-      }
-    },
-    "required": [
-      "type",
-      "pattern"
-    ],
-    "type": "object"
-  },
-  {
-    "properties": {
-      "caseSensitive": {
-        "default": false,
-        "type": "boolean"
-      },
-      "type": {
-        "const": "contains",
-        "type": "string"
-      },
-      "value": {
-        "type": "string"
-      }
-    },
-    "required": [
-      "type",
-      "value"
-    ],
-    "type": "object"
-  },
-  {
-    "properties": {
-      "schema": {
-        "additionalProperties": {},
-        "description": "JSON Schema the output must satisfy.",
-        "propertyNames": {
-          "type": "string"
-        },
-        "type": "object"
-      },
-      "type": {
-        "const": "json_schema",
-        "type": "string"
-      }
-    },
-    "required": [
-      "type",
-      "schema"
-    ],
-    "type": "object"
-  },
-  {
-    "properties": {
-      "max": {
-        "maximum": 9007199254740991,
-        "minimum": 0,
-        "type": "integer"
-      },
-      "min": {
-        "maximum": 9007199254740991,
-        "minimum": 0,
-        "type": "integer"
-      },
-      "type": {
-        "const": "length",
-        "type": "string"
-      }
-    },
-    "required": [
-      "type"
-    ],
-    "type": "object"
-  }
-]New value: +[
+  {
+    "description": "The output must equal value, trimmed unless trim is false.",
+    "properties": {
+      "trim": {
+        "default": true,
+        "type": "boolean"
+      },
+      "type": {
+        "const": "exact",
+        "type": "string"
+      },
+      "value": {
+        "type": "string"
+      }
+    },
+    "required": [
+      "type",
+      "value"
+    ],
+    "type": "object"
+  },
+  {
+    "description": "The output must match the JavaScript regex pattern.",
+    "properties": {
+      "flags": {
+        "type": "string"
+      },
+      "pattern": {
+        "type": "string"
+      },
+      "type": {
+        "const": "regex",
+        "type": "string"
+      }
+    },
+    "required": [
+      "type",
+      "pattern"
+    ],
+    "type": "object"
+  },
+  {
+    "description": "The output must contain value, case-insensitive by default.",
+    "properties": {
+      "caseSensitive": {
+        "default": false,
+        "type": "boolean"
+      },
+      "type": {
+        "const": "contains",
+        "type": "string"
+      },
+      "value": {
+        "type": "string"
+      }
+    },
+    "required": [
+      "type",
+      "value"
+    ],
+    "type": "object"
+  },
+  {
+    "properties": {
+      "schema": {
+        "additionalProperties": {},
+        "description": "JSON Schema the output must satisfy.",
+        "propertyNames": {
+          "type": "string"
+        },
+        "type": "object"
+      },
+      "type": {
+        "const": "json_schema",
+        "type": "string"
+      }
+    },
+    "required": [
+      "type",
+      "schema"
+    ],
+    "type": "object"
+  },
+  {
+    "description": "The output length in characters must be within min and max.",
+    "properties": {
+      "max": {
+        "maximum": 9007199254740991,
+        "minimum": 0,
+        "type": "integer"
+      },
+      "min": {
+        "maximum": 9007199254740991,
+        "minimum": 0,
+        "type": "integer"
+      },
+      "type": {
+        "const": "length",
+        "type": "string"
+      }
+    },
+    "required": [
+      "type"
+    ],
+    "type": "object"
+  }
+]

addedInput schema / properties / golden / items / properties / id / description
```
Added value: +"Case id; baseline and candidate outputs are keyed by it."
```
addedInput schema / properties / judge / properties / enabled / description
```
Added value: +"Set true to ask the judge."
```

addedInput schema / properties / judge / properties / model / description

Added value: +"Judge model; defaults to EVALSMITH_JUDGE_MODEL on the server."

First observedv0.1.0Oct 28, 2026

TDQS

A4/5.0

Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Beyond the annotations, it explains that a regression is a baseline pass plus candidate fail, and that the optional LLM judge is a second opinion whose output never changes the verdict. That is meaningful behavioral context for an open-world judge call. The idempotentHint=false annotation is not explained; the deterministic-check wording could leave an agent wondering whether repeated calls may differ.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Four short sentences, all of which earn their place: purpose, regression definition, judge caveat, and usage. The purpose is front-loaded with no filler.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

An output schema exists, so return values need not be explained, and the annotations carry the safety profile. The description covers the check semantics and judge behavior adequately for a tool with nested parameters. The main gap is the missing guidance on when to choose this tool over its siblings.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already documents golden, baseline, candidate, and judge. The description's definition of baseline and candidate as the current and proposed outputs adds useful conceptual framing, but it gives no parameter-level detail beyond that. A baseline 3 applies when the schema carries the detail.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific verb and resource: run a golden set's deterministic checks (exact, regex, contains, JSON Schema, length) against baseline and candidate outputs. It is unambiguous about what the tool does. It does not differentiate from siblings such as compare_configs or run_suite, which an agent could plausibly confuse it with.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It gives a clear usage context: "Use it before shipping a prompt or model change." That tells the agent when the tool is appropriate. It names no exclusions or alternative tools, so the agent must infer that compare_configs or run_suite handle other cases.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Deploy Server