Skip to main content
Glama
mcp-ml-tools
by adeoluo

Compare two models' per-example scores

eval_compare
Read-onlyIdempotent

Compare two models on the same paired scores to get mean difference, bootstrap confidence intervals, p-values, win/tie counts, and McNemar tests for binary outcomes.

Instructions

Paired comparison of model A and model B on the same examples: mean difference (B minus A), seeded paired bootstrap confidence interval and two-sided p-value, win/tie counts, and McNemar's test (exact and chi-square) when every score is 0 or 1.

Tool call template

Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {
      "scoresA": "<scoresA>",
      "scoresB": "<scoresB>"
    },
    "name": "eval_compare"
  }
}

Input Schema

NameRequiredDescriptionDefault
seedNoRandom seed (default 0)
scoresAYesPer-example scores for model A
scoresBYesPer-example scores for model B, same order
confidenceLevelNoConfidence level (default 0.95)
bootstrapSamplesNoBootstrap resamples (default 10000)

Output Schema

NameRequiredDescriptionDefault
nYes
tiesYes
meanAYes
meanBYes
winsAYes
winsBYes
mcnemarYes
bootstrapYes
standardErrorYes
interpretationYes
meanDifferenceYes

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observedv0.2.2

TDQS

A4.1/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The annotations already establish that this is a read-only, idempotent, closed operation, but the description adds meaningful behavioral detail: the B-minus-A direction, seeded paired bootstrap inference, win/tie counts, and McNemar's test only when all scores are binary. It does not mention failure conditions such as unequal array lengths, but the statistical behavior is otherwise well disclosed.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single dense sentence, but it is front-loaded with the core purpose and every clause carries substantive information. Readability could be improved by splitting the output-statistics list from the core comparison statement, but there is little wasted wording.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a statistical comparison tool with an output schema and annotations, the description is largely complete: it explains the comparison, key outputs, and the binary-score condition for McNemar's test. It could still state that score arrays must have equal length and same ordering, but the core behavior is adequately covered.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already documents the five parameters, establishing a baseline of 3. The description adds useful meaning beyond the schema by clarifying that the scores are paired by example and that the mean difference is calculated as B minus A.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description uses a specific verb and resource: a paired comparison of model A and model B on the same examples. It also enumerates the statistics produced, making the tool's purpose unambiguous.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies the tool is used for paired model comparison on shared examples, but it does not explicitly say when to use it versus alternatives or when not to use it. No exclusions or routing guidance are provided.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.