evalsmith-mcp

Compare retrieval configurations

compare_configs

Read-onlyIdempotent

Compare 2–3 retrieval configurations on the same qrels, rank them by a primary metric, and use seeded paired-bootstrap confidence intervals to check whether the winner's lead is significant before switching.

Instructions

Compare 2-3 retrieval configurations on the same qrels. Ranks them on a primary metric, attaches seeded paired-bootstrap confidence intervals, and says whether the winner's lead over the runner-up is significant (interval of the gap excludes zero). Use it before switching retriever, chunking or reranker, so a small gap on a few queries is not mistaken for a win.

Input Schema

Name	Required	Description	Default
`k`	No	Cut-offs to evaluate. Defaults to [1, 3, 5, 10].
`qrels`	Yes	Relevance judgements: { queryId: { docId: grade } }. Grade 0 means not relevant; higher grades are more relevant (used by nDCG). Example: { "q1": { "d3": 2, "d7": 1 } }
`metric`	No	Primary metric that decides the winner, e.g. ndcg@10, recall@5, mrr.	ndcg@10
`configs`	Yes	Two or three runs over the same queries.
`bootstrap`	No	Paired bootstrap over queries. Defaults: 1000 samples, seed 42, 95% confidence.

Output Schema

Name	Required	Description
`delta`	Yes	Paired bootstrap interval of winner minus runner-up.
`metric`	Yes
`winner`	Yes
`configs`	Yes
`summary`	Yes
`runnerUp`	Yes
`bootstrap`	Yes
`significant`	Yes	True when the delta interval excludes zero.

Schema Changelog

Changes observed during successful MCP inspections.

Changed3 schema fields changedOct 29, 2026
- addedInput schema / properties / bootstrap / properties / confidence / description
```
Added value: +"Confidence level of the intervals, e.g. 0.95."
```
- addedInput schema / properties / bootstrap / properties / samples / description
```
Added value: +"Bootstrap resamples of the query set. More is slower and steadier."
```
- addedInput schema / properties / bootstrap / properties / seed / description
```
Added value: +"Random seed; the same seed gives the same intervals."
```
First observedv0.1.0Oct 28, 2026

TDQS

A4.1/5.0

Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnly/idempotent/non-destructive, and the description adds genuinely new behavioral facts: the intervals are paired-bootstrap over queries and are seeded (reproducible), and significance is defined precisely as the gap interval excluding zero. That is real methodological disclosure, though nothing is said about runtime cost or how ties/near-zero gaps are reported.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two tight sentences: the first front-loads what the tool does and its output, the second front-loads the decision it supports. No filler and no repetition of schema or annotation content.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a statistical comparison tool with an output schema, full schema coverage, and rich annotations, everything an agent needs to select and call it is present: input shape is implied, the significance criterion is stated, and the decision context is given.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so qrels, configs, k, metric and the bootstrap block are all documented in the schema itself. The description only echoes that comparisons run on qrels with a primary metric and seeded bootstrap, adding no syntax or constraint detail beyond the schema. Baseline 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('Compare 2-3 retrieval configurations on the same qrels') plus the three things it produces: ranking, bootstrap intervals, and a significance verdict. It does not name the nearest sibling (evaluate_retrieval), so an agent must infer the boundary between comparing configs and evaluating one, which keeps it just short of a 5.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

'Use it before switching retriever, chunking or reranker' gives a concrete decision point, and the trailing clause explains the failure mode it prevents ('a small gap on a few queries is not mistaken for a win'). No explicit when-not or named alternative is offered, so it stops at clear context rather than full routing.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Deploy Server