IA-QA — 152 QA & Dev Tools for AI Agents

compare_responses

Read-onlyIdempotent

Compare two ALREADY-PRODUCED outputs (e.g. model A vs model B on the same task) side by side. Returns deterministic metrics (token cosine, ROUGE-L, Jaccard, length/structure deltas, JSON diff) and a verdict. If a reference (ground truth) is given, scores each output against it and picks the closer one. If model + api_key are given, an LLM judge also picks a qualitative winner for the task. No re-execution — you bring the outputs.

Input Schema

Name	Required	Description
`task`	No	The task/prompt both outputs were answering — used by the LLM judge for context
`model`	No	Optional judge model id (BYOK). When set with api_key, an LLM judge picks a qualitative winner.
`api_key`	No	Optional API key for the judge model (BYOK). Used only for the judge call; never stored.
`label_a`	No	Label for output A (e.g. "GPT-4o", "v1.0")
`label_b`	No	Label for output B (e.g. "GPT-5-nano", "v1.1")
`reference`	No	Optional ground-truth / expected answer. If set, each output is scored against it and the closer one wins (deterministic).
`check_json`	No	Try to parse as JSON and compare structurally (keys, types, values)
`response_a`	Yes	First output (e.g. model A's answer)
`response_b`	Yes	Second output (e.g. model B's answer)

Output Schema

Name	Required	Description	Default
`judge`	No
`labelA`	No
`labelB`	No
`metrics`	No
`summary`	No
`verdict`	No

Schema Changelog

Changes observed during successful MCP inspections.

Changed1 schema field changedOct 31, 2026

addedInput schema / examples

Added value: +[
+  {
+    "response_a": "Paris is the capital.",
+    "response_b": "Paris is the capital."
+  }
+]

Changed12 schema fields changedJun 27, 2026

addedInput schema / properties / api_key

Added value: +{
+  "description": "Optional API key for the judge model (BYOK). Used only for the judge call; never stored.",
+  "type": "string"
+}

changedInput schema / properties / label_a / description

Previous value: -"Label for response A (e.g. \"GPT-4o\", \"v1.0\")"New value: +"Label for output A (e.g. \"GPT-4o\", \"v1.0\")"

changedInput schema / properties / label_b / description

Previous value: -"Label for response B (e.g. \"Claude\", \"v1.1\")"New value: +"Label for output B (e.g. \"GPT-5-nano\", \"v1.1\")"

addedInput schema / properties / model

Added value: +{
+  "description": "Optional judge model id (BYOK). When set with api_key, an LLM judge picks a qualitative winner.",
+  "type": "string"
+}

addedInput schema / properties / reference

Added value: +{
+  "description": "Optional ground-truth / expected answer. If set, each output is scored against it and the closer one wins (deterministic).",
+  "type": "string"
+}

changedInput schema / properties / response_a / description

Previous value: -"First response (baseline / control)"New value: +"First output (e.g. model A's answer)"

changedInput schema / properties / response_b / description

Previous value: -"Second response (variant / test)"New value: +"Second output (e.g. model B's answer)"

addedInput schema / properties / task

Added value: +{
+  "description": "The task/prompt both outputs were answering — used by the LLM judge for context",
+  "type": "string"
+}

removedOutput schema / properties / diffs
```
Removed value: -{}
```
addedOutput schema / properties / judge
```
Added value: +{}
```
addedOutput schema / properties / metrics
```
Added value: +{}
```
addedOutput schema / properties / verdict
```
Added value: +{}
```

Changed1 schema field changedJun 21, 2026

addedOutput schema / properties

Added value: +{
+  "diffs": {},
+  "labelA": {},
+  "labelB": {},
+  "summary": {
+    "type": "string"
+  }
+}

Changed1 schema field changedJun 10, 2026

changedOutput schema / (root)

Previous value: -nullNew value: +{
+  "additionalProperties": true,
+  "type": "object"
+}

First observedMay 26, 2026

TDQS

A4.5/5.0

Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already establish readOnly/idempotent/non-destructive, and the description adds real context beyond them: the deterministic metric set, that a verdict is returned, that the LLM judge is opt-in and requires BYOK credentials, and that inputs are never re-executed. It does not mention the cost/latency implications of the judge call, which is the main remaining behavioral unknown.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three sentences, front-loaded with what it does, followed by the two optional modes, ending with the key scoping constraint. No filler and no repetition of the name.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With an output schema present, the description needn't explain return shapes, and it still summarizes them usefully. All 9 parameters are schema-documented, annotations cover the safety profile, and the mode logic is fully explained — nothing an agent needs to invoke this correctly is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the baseline is 3, but the description adds cross-parameter meaning the schema cannot: reference triggers per-output scoring and picks the closer one, while model+api_key activates a qualitative judge. That conditional wiring is genuinely useful beyond the individual field docs; only check_json and task go unmentioned.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ("Compare two ALREADY-PRODUCED outputs ... side by side") and enumerates the exact artifacts it produces (cosine, ROUGE-L, Jaccard, deltas, JSON diff, verdict). The closing "No re-execution — you bring the outputs" sharply separates it from execution-based siblings like compare_models.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Clearly lays out the three operating modes: bare comparison, reference-based scoring, and model+api_key LLM judging, so an agent knows which parameter combination selects which behavior. It stops short of naming sibling alternatives (similarity_score, diff_text, analyze_responses) explicitly for the cases where those would be preferable.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

All IA-QA — 152 QA & Dev Tools for AI Agents tools Inspect

Try in Browser

One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

Glama MCP Gateway

Add one secure layer between your agents and this server.