evalsmith-mcp

Check agent tool calls

check_tool_calls

Read-onlyIdempotent

Compare an agent's tool-call trace to expected calls to verify tool choice, order, and arguments, and get a pass flag, score, and mismatch reasons.

Instructions

Compare an agent's tool-call trace (evalsmith, Anthropic Messages or OpenAI format, as logged) with the expected calls: right tool, right order (strict or lax) and arguments that satisfy exact / regex / subset / any matchers. Use it to test an agent change against a known-good trace. Returns a pass flag, a score and a reason for every mismatch.

Input Schema

Name	Required	Description	Default
`order`	No	strict: the trace must be exactly the expected calls, in order. lax: every expected call must appear somewhere; order and extra calls are allowed.	strict
`trace`	Yes	Tool calls the agent made, in order. Accepts evalsmith calls ({ "tool", "args" }), Anthropic Messages (messages or content blocks with type "tool_use"), OpenAI Chat messages with "tool_calls" and OpenAI Responses items with type "function_call"; other conversation items are skipped.
`expected`	Yes	Expected calls. Argument matchers: {"type":"exact","value":...}, {"type":"regex","pattern":"..."}, {"type":"subset","value":{...}}, {"type":"any"}.

Output Schema

Name	Required	Description
`calls`	No	Tool names of the trace, in call order. Used to draw the expected vs. actual view.
`order`	Yes	strict: the trace must be exactly the expected calls, in order. lax: every expected call must appear somewhere; order and extra calls are allowed.
`score`	Yes	Matched expected calls / expected calls.
`steps`	Yes
`passed`	Yes
`summary`	Yes
`unexpected`	Yes

Schema Changelog

Changes observed during successful MCP inspections.

Changed8 schema fields changedOct 29, 2026

changedInput schema / properties / expected / items / properties / args / additionalProperties / oneOf

Previous value: -[
-  {
-    "properties": {
-      "type": {
-        "const": "exact",
-        "type": "string"
-      },
-      "value": {}
-    },
-    "required": [
-      "type",
-      "value"
-    ],
-    "type": "object"
-  },
-  {
-    "properties": {
-      "flags": {
-        "type": "string"
-      },
-      "pattern": {
-        "type": "string"
-      },
-      "type": {
-        "const": "regex",
-        "type": "string"
-      }
-    },
-    "required": [
-      "type",
-      "pattern"
-    ],
-    "type": "object"
-  },
-  {
-    "properties": {
-      "type": {
-        "const": "subset",
-        "type": "string"
-      },
-      "value": {}
-    },
-    "required": [
-      "type",
-      "value"
-    ],
-    "type": "object"
-  },
-  {
-    "properties": {
-      "type": {
-        "const": "any",
-        "type": "string"
-      }
-    },
-    "required": [
-      "type"
-    ],
-    "type": "object"
-  }
-]New value: +[
+  {
+    "description": "The argument must deep-equal value.",
+    "properties": {
+      "type": {
+        "const": "exact",
+        "type": "string"
+      },
+      "value": {}
+    },
+    "required": [
+      "type",
+      "value"
+    ],
+    "type": "object"
+  },
+  {
+    "description": "The argument, as a string, must match the JavaScript regex pattern.",
+    "properties": {
+      "flags": {
+        "type": "string"
+      },
+      "pattern": {
+        "type": "string"
+      },
+      "type": {
+        "const": "regex",
+        "type": "string"
+      }
+    },
+    "required": [
+      "type",
+      "pattern"
+    ],
+    "type": "object"
+  },
+  {
+    "description": "Every key in value must be present in the argument with an equal value.",
+    "properties": {
+      "type": {
+        "const": "subset",
+        "type": "string"
+      },
+      "value": {}
+    },
+    "required": [
+      "type",
+      "value"
+    ],
+    "type": "object"
+  },
+  {
+    "description": "The argument must be present; any value passes.",
+    "properties": {
+      "type": {
+        "const": "any",
+        "type": "string"
+      }
+    },
+    "required": [
+      "type"
+    ],
+    "type": "object"
+  }
+]

addedInput schema / properties / expected / items / properties / tool / description
```
Added value: +"Name of the tool the agent should call."
```

changedInput schema / properties / trace / description

Previous value: -"Tool calls the agent made, in order."New value: +"Tool calls the agent made, in order. Accepts evalsmith calls ({ \"tool\", \"args\" }), Anthropic Messages (messages or content blocks with type \"tool_use\"), OpenAI Chat messages with \"tool_calls\" and OpenAI Responses items with type \"function_call\"; other conversation items are skipped."

addedInput schema / properties / trace / items / additionalProperties
```
Added value: +{}
```

removedInput schema / properties / trace / items / properties

Removed value: -{
-  "args": {
-    "additionalProperties": {},
-    "default": {},
-    "propertyNames": {
-      "type": "string"
-    },
-    "type": "object"
-  },
-  "tool": {
-    "minLength": 1,
-    "type": "string"
-  }
-}

addedInput schema / properties / trace / items / propertyNames
```
Added value: +{
+  "type": "string"
+}
```
removedInput schema / properties / trace / items / required
```
Removed value: -[
-  "tool"
-]
```

addedOutput schema / properties / calls

Added value: +{
+  "description": "Tool names of the trace, in call order. Used to draw the expected vs. actual view.",
+  "items": {
+    "type": "string"
+  },
+  "type": "array"
+}

First observedv0.1.0Oct 28, 2026

TDQS

A3.9/5.0

Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnly/idempotent/non-destructive, and the description is consistent with a pure comparison. It adds value beyond the annotations by disclosing the return shape ('a pass flag, a score and a reason for every mismatch') and which trace formats are accepted and how non-tool conversation items are skipped.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three tightly packed sentences, front-loaded with the core comparison and then what it checks and returns. Little is wasted, though the third sentence's return-format summary slightly overlaps the output schema.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With an output schema present and annotations covering the safety profile, the description fills the remaining gaps: supported input formats and ordering semantics. It leaves only minor details (e.g., how multiple mismatches are scored) to the output schema.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so parameters are already well documented, and the description largely restates the matcher list (exact/regex/subset/any) and ordering modes rather than extending them. The one genuine addition is the note that the trace is read 'as logged' across evalsmith/Anthropic/OpenAI shapes, which is also covered in the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description gives a specific verb ('Compare') and a specific resource ('an agent's tool-call trace') against expected calls, and enumerates what is checked (tool, order, argument matchers). It is clearly distinguishable from siblings like compare_configs or evaluate_retrieval, though it never names an alternative explicitly.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It supplies a concrete usage context: 'Use it to test an agent change against a known-good trace,' and the 'strict or lax' ordering split tells the agent which mode fits which intent. It stops short of stating when NOT to use it or naming competing siblings such as detect_prompt_regression.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Deploy Server