You can use this MCP server to run deterministic evaluations for RAG retrieval, retrieval configurations, agent tool calls, and prompt changes, then produce CI-ready reports.
Score retrieval quality against qrels with recall@k, precision@k, nDCG@k, hit rate@k, MRR, per-query metrics, and skipped/missing-query reporting.
Compare 2–3 retrieval configurations on a primary metric with seeded paired-bootstrap confidence intervals and a significance check for the winner vs. runner-up.
Check agent tool calls against expected calls using strict or lax ordering and exact/regex/subset/any argument matchers, returning pass/fail, score, mismatch reasons, and unexpected calls.
Detect prompt regressions across golden cases using deterministic checks (exact, regex, contains, JSON Schema, length), with an optional LLM judge that does not change the verdict.
Generate reports as Markdown, JSON, or JUnit XML, optionally applying metric gates to decide overall pass/fail.
Provides a GitHub Action that runs evalsmith suites in GitHub workflows, applies metric gates, and publishes outputs, job summaries, and reports as a merge gate.
Can call a configured Ollama OpenAI-compatible endpoint as an optional LLM judge for a second opinion on prompt regressions, without changing the deterministic verdict.
Supports agent tool-call traces logged in OpenAI Chat/Responses formats and can use an OpenAI-compatible endpoint for the optional LLM judge.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type @ followed by the MCP server name and your instructions, e.g., "@evalsmith-mcp compare my two retrieval configs and check if the gap is significant"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
Evaluate RAG retrieval and AI agents with deterministic metrics, from any MCP client or from CI.
evalsmith answers four questions that come up every time a retriever, prompt or agent changes:
Is retrieval good enough? recall@k, precision@k, nDCG@k, hit rate and MRR from qrels and a ranked run.
Which configuration is better, and is the gap real? 2-3 configurations ranked with seeded paired-bootstrap confidence intervals.
Did the agent call the right tools? Trace vs. expected calls, strict or lax order, argument matchers.
Did the prompt change break anything? Golden-set checks on baseline vs. candidate outputs.
Everything is deterministic and runs offline. An LLM judge exists for a second opinion, but it is off by default and never decides the verdict. Results come out as a terminal table, Markdown, JSON, JUnit XML or a self-contained HTML report, and the GitHub Action turns them into a merge gate.
flowchart LR
subgraph Clients
A[MCP client<br/>desktop app, IDE, agent]
B[Remote MCP client<br/>or service]
C[Terminal / CI job]
D[GitHub workflow]
end
subgraph evalsmith
S1[MCP server<br/>stdio]
S2[MCP server<br/>Streamable HTTP, stateless]
S3[CLI<br/>evalsmith run]
S4[GitHub Action<br/>uses: evalsmith-mcp@v0]
CORE[(src/core<br/>metrics, bootstrap, matchers,<br/>checks, gates, reports)]
end
A --> S1 --> CORE
B --> S2 --> CORE
C --> S3 --> CORE
D --> S4 --> CORE
S3 -. exit code + JUnit + HTML .-> C
S4 -. outputs + job summary .-> DInterface | Entry point | Who uses it | Why it exists |
MCP over stdio |
| Desktop MCP clients, IDE assistants, local agents | An assistant can score a run, compare configs or audit a trace mid-conversation, with typed |
MCP over Streamable HTTP |
| Remote clients, internal platforms, the MCP Inspector | Same tools over the network. Stateless: no sessions to store, scales horizontally. |
CLI / CI |
| Pipelines and terminals |
|
GitHub Action |
| GitHub workflows | The same gates as a step, with outputs, a job summary and no install. |
All four call the same functions in src/core. The MCP tools are thin adapters: parse input with Zod, call the core, return structuredContent validated against an output schema.
Related MCP server: mcp-llm-eval
flowchart TB
subgraph Tools
T1[evaluate_retrieval]
T2[compare_configs]
T3[check_tool_calls]
T4[detect_prompt_regression]
T5[generate_report]
T6[run_suite]
end
subgraph Resources
R1["evalsmith://examples/{name}<br/>12 synthetic datasets + suite"]
end
subgraph Prompts
P1[compare_rag_configs]
P2[audit_agent_trace]
P3[review_prompt_change]
end
T1 -- structuredContent --> T5
T2 -- structuredContent --> T5
T3 -- structuredContent --> T5
T4 -- structuredContent --> T5
classDef tool fill:#ffedd5,stroke:#c2410c,color:#431407
classDef res fill:#e0f2fe,stroke:#0369a1,color:#082f49
classDef prom fill:#ede9fe,stroke:#6d28d9,color:#2e1065
class T1,T2,T3,T4,T5,T6 tool
class R1 res
class P1,P2,P3 promEvery tool declares an input and an output Zod schema, returns structuredContent plus a JSON text block, and is annotated readOnly and non-destructive. Four tools are also idempotent and closed-world. detect_prompt_regression and run_suite are declared open-world and non-idempotent: annotations are per tool, and with the judge enabled they call an external model whose answers can vary. With the judge off (the default) they behave like the others.
Tool | Input | Output |
|
| Mean recall/precision/nDCG/hit rate at every k, MRR, per-query values, skipped and missing queries |
|
| Ranking with intervals, winner, runner-up, paired interval of the gap, |
|
|
|
|
| Verdict, pass rates, regressions, improvements, per-case failures, judge votes when enabled |
| Any of the results above, |
|
| Optional |
|
run_suite runs the same suites as evalsmith run, so a client can evaluate data that lives in files instead of pasting it into the conversation. It only reads inside the workspace root (EVALSMITH_ROOT, the directory the server started in by default), following symlinks before it checks, and runs the bundled example suite when no config is given.
Bad input (unknown metric, duplicate config names, qrels with no relevant documents) comes back as an InvalidParams tool error with a message the model can act on. A judge endpoint that is down or returns an HTTP error is an InternalError.
evalsmith://examples/{name} lists and serves the synthetic datasets in examples/datasets, so a client can try every tool before wiring in its own data:
Name | Content |
| 20 help-center articles of a fictional invoicing app, 12 questions, graded judgements |
| Top-10 rankings from three retrieval configurations |
| A support agent refunding a duplicate charge, and the calls it should make |
| Six golden cases and three sets of outputs |
| The example |
Prompt | Argument | What it asks the model to do |
|
| Compare the configurations, report the gap with its interval and the weakest queries |
|
| Check the trace and propose the smallest fix for each failed step |
|
| Detect regressions, build a Markdown report, decide ship / do not ship |
sequenceDiagram
autonumber
actor User
participant Host as MCP client (LLM host)
participant ES as evalsmith (stdio)
participant Core as src/core
User->>Host: "Is the reranker worth it on our help-center set?"
Host->>ES: initialize, tools/list, resources/list
ES-->>Host: 6 tools + JSON Schemas, 13 resources
Host->>ES: resources/read evalsmith://examples/rag-qrels (+ 3 runs)
ES-->>Host: qrels and runs as JSON
Host->>ES: tools/call compare_configs {metric: "ndcg@5", configs: [...]}
ES->>Core: compareConfigs(qrels, configs, "ndcg@5", {seed: 42})
Core-->>ES: ranking, intervals, paired delta
ES-->>Host: structuredContent {winner: "hybrid+rerank", significant: true, ...}
Host->>ES: tools/call generate_report {format: "markdown", comparison}
ES-->>Host: Markdown report
Host-->>User: "Yes: +0.153 nDCG@5 over hybrid, 95% CI [0.069, 0.251]."For a query $q$ with relevant set $R_q$ (documents with grade $> 0$) and ranking $D_q$, with $D_q^{k}$ its first $k$ documents:
\mathrm{Recall@}k = \frac{|R_q \cap D_q^{k}|}{|R_q|}
\qquad
\mathrm{Precision@}k = \frac{|R_q \cap D_q^{k}|}{k}
\qquad
\mathrm{HitRate@}k = \mathbb{1}\left[\,R_q \cap D_q^{k} \neq \varnothing\,\right]\mathrm{RR}_q = \frac{1}{\text{rank of the first relevant document}}
\qquad
\mathrm{MRR} = \frac{1}{|Q|}\sum_{q \in Q} \mathrm{RR}_q\mathrm{DCG@}k = \sum_{i=1}^{k} \frac{\mathrm{rel}_i}{\log_2(i+1)}
\qquad
\mathrm{nDCG@}k = \frac{\mathrm{DCG@}k}{\mathrm{IDCG@}k}$\mathrm{rel}_i$ is the graded judgement of the document at rank $i$ (linear gain, as in trec_eval), and IDCG is the DCG of the judgements sorted in descending order. Reported values are means over judged queries.
Conventions, all covered by tests:
Queries whose judgements are all 0 are skipped and listed in skippedQueries.
Judged queries missing from the run score 0 and are listed in missingFromRun.
Precision divides by $k$, not by the ranking length.
Repeated document ids in a ranking keep only their first position, as in trec_eval, so a run cannot score above 1 by repeating a relevant document.
Cut-offs start at 1: recall@0 is rejected.
Worked example (tests/core/retrieval.test.ts): judgements a=2, b=1, c=0, ranking d, a, e, b, c.
DCG@5 $= 2/\log_2 3 + 1/\log_2 5 = 1.69254$, IDCG@5 $= 2/\log_2 2 + 1/\log_2 3 = 2.63093$, so nDCG@5 $= 0.64332$; recall@3 $= 0.5$, precision@5 $= 0.4$, RR $= 0.5$.
Bootstrap. compare_configs resamples queries with replacement, using the same indices for every configuration (paired), samples times, from a seeded PRNG (mulberry32). Intervals are percentiles of the resampled means. The lead is significant when the interval of winner minus runner-up excludes zero. The same seed gives the same intervals on every machine.
{
"tool": "create_refund",
"args": {
"invoice_id": { "type": "exact", "value": "INV-2041" },
"reason": { "type": "any" },
"metadata": { "type": "subset", "value": { "ticket": "T-5532" } },
"to": { "type": "regex", "pattern": "@acme-robotics\\.example$" }
}
}strict requires the trace to be exactly the expected calls in order; calls past the expected sequence fail the check and appear as failed test cases in JUnit. lax ignores order and pairs expected calls with trace calls by maximum bipartite matching (augmenting paths), so a loose matcher never steals the only call a stricter one accepts. Extra calls are reported without failing. Every matcher, exact included, fails when the argument is missing.
exact (trimmed by default), regex, contains (case-insensitive by default), json_schema (the output is parsed and validated with Ajv) and length (min / max characters). A regression is a case that passes on the baseline and fails on the candidate. Missing outputs count as failures.
The optional judge sends both outputs to any OpenAI-compatible /chat/completions endpoint, Ollama included, at temperature 0. To cancel position bias each case is judged twice, with baseline as answer A and then as answer B. A side wins only when both orderings agree; a disagreement counts as a tie. That costs two requests per case. Its votes are reported next to the verdict.
npx -y evalsmith-mcp # MCP server on stdio
npm install -g evalsmith-mcp # or install the `evalsmith` CLI and `evalsmith-mcp` server
docker build -t evalsmith-mcp . && docker run --rm -i evalsmith-mcp # or run the server in a containerTo run the bundled examples and the tests from source:
git clone https://github.com/diegoperezg7/evalsmith-mcp.git
cd evalsmith-mcp
npm ci # also builds dist/ through the prepare script
npx playwright install chromium # the HTML report tests open it in a real browser
npm test # metrics, traces, MCP in-memory + HTTP, CLI exit codes, HTML report, action
node dist/cli/index.js run examples/evalsmith.yaml
From a terminal:
claude mcp add evalsmith -- npx -y evalsmith-mcp # Claude Code
codex mcp add evalsmith -- npx -y evalsmith-mcp # Codex CLI
code --add-mcp '{"name":"evalsmith","command":"npx","args":["-y","evalsmith-mcp"]}' # VS CodeOther desktop MCP clients read an mcpServers block like this (examples/mcp-client-config.json):
{
"mcpServers": {
"evalsmith": {
"command": "npx",
"args": ["-y", "evalsmith-mcp"],
"env": { "LOG_LEVEL": "warn" }
}
}
}Restart the client, then try the compare_rag_configs prompt or ask it to read evalsmith://examples/agent-trace and check it.
EVALSMITH_TRANSPORT=http EVALSMITH_HTTP_PORT=3333 node dist/mcp/server.js
curl -s localhost:3333/healthz # {"status":"ok","version":"0.1.0"}
npx @modelcontextprotocol/inspector # connect to http://127.0.0.1:3333/mcpVariable | Default | Purpose |
|
|
|
|
| HTTP bind address |
|
| OpenAI-compatible endpoint for the optional judge |
|
| Judge model |
| unset | Only for hosted endpoints |
| current directory | Directory |
|
| pino level; logs always go to stderr |
evalsmith run <config> [--fail-under metric=value ...] [--format table|json|markdown|html]
[--report file.{md,json,xml,html}] [--report-format markdown|json|junit|html]
evalsmith doctor [config] [--judge]Gates accept recall@k, precision@k, ndcg@k, hit_rate@k, mrr, tool_calls (score of check_tool_calls) and prompt_pass_rate. Gates in the YAML file apply too; a command-line gate with the same name overrides it. A prompt regression or a failed tool-call check fails the run even without gates.
Exit code | Meaning |
| Every gate passed, no regression |
| A gate is below its threshold, the candidate regressed, or the tool calls do not match |
| Usage error, invalid config, unknown metric, unreadable dataset, judge endpoint down |
node dist/cli/index.js run examples/evalsmith.yaml on the bundled datasets (examples/evalsmith.yaml):
evalsmith · ledgerly-support-assistant
Retrieval (12 queries, MRR 0.8403)
┌────┬────────┬───────────┬────────┬──────────┐
│ k │ recall │ precision │ nDCG │ hit rate │
├────┼────────┼───────────┼────────┼──────────┤
│ 1 │ 0.4583 │ 0.7500 │ 0.6667 │ 0.7500 │
│ 3 │ 0.7500 │ 0.4167 │ 0.7442 │ 0.9167 │
│ 5 │ 0.9167 │ 0.3000 │ 0.8059 │ 1.0000 │
│ 10 │ 1.0000 │ 0.1667 │ 0.8285 │ 1.0000 │
└────┴────────┴───────────┴────────┴──────────┘
Comparison on ndcg@5 (paired bootstrap, 2000 samples, seed 42)
┌───┬───────────────┬────────┬──────────────────┐
│ # │ config │ ndcg@5 │ 95% CI │
├───┼───────────────┼────────┼──────────────────┤
│ 1 │ hybrid+rerank │ 0.9588 │ [0.8950, 1.0000] │
│ 2 │ hybrid │ 0.8059 │ [0.6875, 0.9141] │
│ 3 │ bm25 │ 0.5596 │ [0.3782, 0.7354] │
└───┴───────────────┴────────┴──────────────────┘
hybrid+rerank wins on ndcg@5 (0.9588), +0.1528 over hybrid; 95% interval of the gap [0.0692, 0.2506] excludes zero.
Agent tool calls (strict)
┌──────────────────┬─────────┬────────┬───────┐
│ expected │ matched │ status │ notes │
├──────────────────┼─────────┼────────┼───────┤
│ search_customers │ #1 │ pass │ │
│ get_invoice │ #2 │ pass │ │
│ create_refund │ #3 │ pass │ │
│ send_email │ #4 │ pass │ │
└──────────────────┴─────────┴────────┴───────┘
All 4 expected call(s) matched (strict order).
Prompt regression
┌───────────────────────┬──────────┬───────────┬────────────────────┐
│ case │ baseline │ candidate │ candidate failures │
├───────────────────────┼──────────┼───────────┼────────────────────┤
│ classify-intent │ pass │ pass │ │
│ extract-invoice-id │ pass │ pass │ │
│ refund-policy-summary │ fail │ pass │ │
│ due-date-format │ pass │ pass │ │
│ refuse-card-number │ pass │ pass │ │
│ refund-payload │ pass │ pass │ │
└───────────────────────┴──────────┴───────────┴────────────────────┘
No regressions across 6 case(s). Pass rate 83.3% -> 100.0%.
Gates
┌────────────┬────────┬────────┬────────┐
│ metric │ min │ actual │ status │
├────────────┼────────┼────────┼────────┤
│ recall@5 │ 0.8000 │ 0.9167 │ pass │
│ ndcg@5 │ 0.7500 │ 0.8059 │ pass │
│ tool_calls │ 1.0000 │ 1.0000 │ pass │
└────────────┴────────┴────────┴────────┘
PASSRaising the bar makes the same run fail the build:
$ node dist/cli/index.js run examples/evalsmith.yaml --fail-under recall@5=0.95 --fail-under mrr=0.9 --report report.xml
...
Gates
┌────────────┬────────┬────────┬────────┐
│ metric │ min │ actual │ status │
├────────────┼────────┼────────┼────────┤
│ recall@5 │ 0.9500 │ 0.9167 │ FAIL │
│ ndcg@5 │ 0.7500 │ 0.8059 │ pass │
│ tool_calls │ 1.0000 │ 1.0000 │ pass │
│ mrr │ 0.9000 │ 0.8403 │ FAIL │
└────────────┴────────┴────────┴────────┘
FAIL · 2 gate(s) below threshold
Report written to report.xml
$ echo $?
1--report report.html (or --format html on stdout) writes the whole run to one HTML file that opens offline and follows the system light or dark mode:
the verdict, and a card per gate with its value, threshold and status;
retrieval metrics per k as bars with a bootstrap interval over the queries, and gate thresholds marked on them;
the configuration comparison with each interval, the winner and the gap to the runner-up;
expected vs. actual tool calls, with the matched pairs linked and mismatches flagged;
baseline vs. candidate per golden case, with the failing checks one click away.
Dataset text (case ids, tool names, failure messages) is embedded as JSON in a non-executable <script type="application/json"> block and rendered with textContent only. A Content-Security-Policy allows no network access and only the page's own script, by hash. Tests open a report full of markup in Chromium and check that nothing from it runs. npm run docs:screenshots regenerates the pictures above.
- uses: actions/checkout@v5
- uses: diegoperezg7/evalsmith-mcp@v0
id: evals
with:
config: evals/evalsmith.yaml
fail-under: |
recall@5=0.8
ndcg@10=0.7
tool_calls=1
html-report: evalsmith-report.htmlThe step fails when a gate is below its threshold, the prompt candidate regresses or the tool calls do not match, and writes the Markdown report with the verdict to the job summary. It runs the same core as the CLI from a bundled node24 script, with no install step and no shell. examples/github-workflow.yml is a complete workflow that also keeps the HTML report as an artifact and publishes the JUnit XML as a check run.
Input | Default | Description |
|
| Suite file. Dataset paths inside it are relative to the file |
| One | |
| Report file: | |
| from extension |
|
| Path for the self-contained HTML report | |
|
| Append the Markdown report to the job summary |
Output | Description |
|
|
| Gates below their threshold |
| Retrieval metrics as JSON, e.g. |
| MRR of the retrieval run |
| Tool-call score, matched / expected |
| Candidate pass rate on the golden set |
| Cases that pass on baseline and fail on candidate |
| Winning configuration of the comparison |
| Path of |
| Path of the HTML report, empty when not requested |
Outputs of a section the suite does not have are empty. Dataset text printed to the log is wrapped in ::stop-commands::, so a case id or tool name cannot issue workflow commands. The judge reads EVALSMITH_JUDGE_* from the step environment, as the CLI does.
Without the action, the CLI does the same from any CI:
npx evalsmith run evals/evalsmith.yaml --fail-under recall@5=0.8 --format markdown --report evalsmith-report.xml >> "$GITHUB_STEP_SUMMARY"name: ledgerly-support-assistant
retrieval:
qrels: datasets/rag/qrels.json # { queryId: { docId: grade } }
run: datasets/rag/runs/hybrid.json # { queryId: [docId, ...] }
k: [1, 3, 5, 10]
compare:
metric: ndcg@5
configs:
- { name: bm25, run: datasets/rag/runs/bm25.json }
- { name: hybrid, run: datasets/rag/runs/hybrid.json }
- { name: hybrid+rerank, run: datasets/rag/runs/hybrid-rerank.json }
bootstrap: { samples: 2000, seed: 42, confidence: 0.95 }
agent:
trace: datasets/agent/trace.json
expected: datasets/agent/expected.json
order: strict
prompts:
golden: datasets/prompts/golden.json
baseline: datasets/prompts/baseline.json
candidate: datasets/prompts/candidate.json
judge: { enabled: false }
gates:
recall@5: 0.8
ndcg@5: 0.75
tool_calls: 1.0Every section is optional. Paths are relative to the YAML file.
ragcompass profiles a corpus and recommends how to configure the RAG pipeline: parser, chunking, top-k, hybrid search, reranking. evalsmith checks whether that configuration holds up:
flowchart LR
A[Documents] --> B[ragcompass analyze<br/>recommended configs]
B --> C[Index 2-3 candidate configs]
C --> D[Run the eval queries<br/>save rankings]
D --> E[evalsmith compare_configs<br/>winner + intervals]
E --> F[evalsmith run in CI<br/>--fail-under gates]Deterministic first. Metrics, matchers and checks are pure functions with hand-computed tests. A model is never needed to get a verdict, so results do not drift between runs and no paid API key is required.
Judge off by default. When enabled it goes through any OpenAI-compatible endpoint (local Ollama works), and its votes sit next to the verdict instead of changing it.
Intervals, not just means. Twelve queries can make a 0.03 lead look decisive. A paired bootstrap with a fixed seed makes the uncertainty visible and reproducible.
Stateless HTTP done properly. A stateless StreamableHTTPServerTransport cannot be reused across requests, so every POST gets its own server and transport. Registration is cheap, and a test makes two consecutive calls over real HTTP to prove it.
stdout belongs to the protocol. pino writes to stderr and ESLint forbids console in src/, so no stray byte breaks the stdio transport. The CLI keeps stdout for results only.
One schema, three uses. The Zod schemas in src/core/schemas.ts validate MCP input, generate the JSON Schemas clients see, and type the core.
src/
core/ metrics, bootstrap, tool-call matchers, prompt checks, judge, gates, reports (incl. HTML), suite runner
mcp/ server (stdio + HTTP), tools/, resources/, prompts/
cli/ evalsmith run | doctor, table/json/markdown/html output
action/ GitHub Action entry point, bundled into dist/action
shared/ config (Zod), pino logger on stderr, MCP errors
tests/
core/ hand-computed metrics, bootstrap, matchers, checks, reports, HTML in Chromium
mcp/ InMemoryTransport client and real Streamable HTTP client
cli/ exit codes 0 / 1 / 2, JUnit file, doctor
examples/
datasets/ synthetic RAG corpus + qrels + 3 runs, agent trace, golden prompts
evalsmith.yaml, github-workflow.yml, mcp-client-config.jsonnpm run check # typecheck + eslint + prettier + vitest
npm run inspect # MCP Inspector against the built stdio serverReleases come from version tags: release.yml publishes to npm with provenance, to the MCP Registry and to GitHub Releases, and the action is published to the Marketplace from the release. docs/RELEASING.md has the steps, including the action bundle and the v0 tag. Changes are listed in CHANGELOG.md.
Compare an agent's tool-call trace (evalsmith, Anthropic Messages or OpenAI format, as logged) with the expected calls: right tool, right order (strict or lax) and arguments that satisfy exact / regex / subset / any matchers. Use it to test an agent change against a known-good trace. Returns a pass flag, a score and a reason for every mismatch.
| Name | Required | Description | Default |
|---|---|---|---|
| order | No | strict: the trace must be exactly the expected calls, in order. lax: every expected call must appear somewhere; order and extra calls are allowed. | strict |
| trace | Yes | Tool calls the agent made, in order. Accepts evalsmith calls ({ "tool", "args" }), Anthropic Messages (messages or content blocks with type "tool_use"), OpenAI Chat messages with "tool_calls" and OpenAI Responses items with type "function_call"; other conversation items are skipped. | |
| expected | Yes | Expected calls. Argument matchers: {"type":"exact","value":...}, {"type":"regex","pattern":"..."}, {"type":"subset","value":{...}}, {"type":"any"}. |
| Name | Required | Description |
|---|---|---|
| calls | No | Tool names of the trace, in call order. Used to draw the expected vs. actual view. |
| order | Yes | strict: the trace must be exactly the expected calls, in order. lax: every expected call must appear somewhere; order and extra calls are allowed. |
| score | Yes | Matched expected calls / expected calls. |
| steps | Yes | |
| passed | Yes | |
| summary | Yes | |
| unexpected | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnly/idempotent/non-destructive, and the description is consistent with a pure comparison. It adds value beyond the annotations by disclosing the return shape ('a pass flag, a score and a reason for every mismatch') and which trace formats are accepted and how non-tool conversation items are skipped.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three tightly packed sentences, front-loaded with the core comparison and then what it checks and returns. Little is wasted, though the third sentence's return-format summary slightly overlaps the output schema.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With an output schema present and annotations covering the safety profile, the description fills the remaining gaps: supported input formats and ordering semantics. It leaves only minor details (e.g., how multiple mismatches are scored) to the output schema.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so parameters are already well documented, and the description largely restates the matcher list (exact/regex/subset/any) and ordering modes rather than extending them. The one genuine addition is the note that the trace is read 'as logged' across evalsmith/Anthropic/OpenAI shapes, which is also covered in the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description gives a specific verb ('Compare') and a specific resource ('an agent's tool-call trace') against expected calls, and enumerates what is checked (tool, order, argument matchers). It is clearly distinguishable from siblings like compare_configs or evaluate_retrieval, though it never names an alternative explicitly.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It supplies a concrete usage context: 'Use it to test an agent change against a known-good trace,' and the 'strict or lax' ordering split tells the agent which mode fits which intent. It stops short of stating when NOT to use it or naming competing siblings such as detect_prompt_regression.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Compare 2-3 retrieval configurations on the same qrels. Ranks them on a primary metric, attaches seeded paired-bootstrap confidence intervals, and says whether the winner's lead over the runner-up is significant (interval of the gap excludes zero). Use it before switching retriever, chunking or reranker, so a small gap on a few queries is not mistaken for a win.
| Name | Required | Description | Default |
|---|---|---|---|
| k | No | Cut-offs to evaluate. Defaults to [1, 3, 5, 10]. | |
| qrels | Yes | Relevance judgements: { queryId: { docId: grade } }. Grade 0 means not relevant; higher grades are more relevant (used by nDCG). Example: { "q1": { "d3": 2, "d7": 1 } } | |
| metric | No | Primary metric that decides the winner, e.g. ndcg@10, recall@5, mrr. | ndcg@10 |
| configs | Yes | Two or three runs over the same queries. | |
| bootstrap | No | Paired bootstrap over queries. Defaults: 1000 samples, seed 42, 95% confidence. |
| Name | Required | Description |
|---|---|---|
| delta | Yes | Paired bootstrap interval of winner minus runner-up. |
| metric | Yes | |
| winner | Yes | |
| configs | Yes | |
| summary | Yes | |
| runnerUp | Yes | |
| bootstrap | Yes | |
| significant | Yes | True when the delta interval excludes zero. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnly/idempotent/non-destructive, and the description adds genuinely new behavioral facts: the intervals are paired-bootstrap over queries and are seeded (reproducible), and significance is defined precisely as the gap interval excluding zero. That is real methodological disclosure, though nothing is said about runtime cost or how ties/near-zero gaps are reported.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two tight sentences: the first front-loads what the tool does and its output, the second front-loads the decision it supports. No filler and no repetition of schema or annotation content.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a statistical comparison tool with an output schema, full schema coverage, and rich annotations, everything an agent needs to select and call it is present: input shape is implied, the significance criterion is stated, and the decision context is given.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so qrels, configs, k, metric and the bootstrap block are all documented in the schema itself. The description only echoes that comparisons run on qrels with a primary metric and seeded bootstrap, adding no syntax or constraint detail beyond the schema. Baseline 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource ('Compare 2-3 retrieval configurations on the same qrels') plus the three things it produces: ranking, bootstrap intervals, and a significance verdict. It does not name the nearest sibling (evaluate_retrieval), so an agent must infer the boundary between comparing configs and evaluating one, which keeps it just short of a 5.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
'Use it before switching retriever, chunking or reranker' gives a concrete decision point, and the trailing clause explains the failure mode it prevents ('a small gap on a few queries is not mistaken for a win'). No explicit when-not or named alternative is offered, so it stops at clear context rather than full routing.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Run a golden set's deterministic checks (exact, regex, contains, JSON Schema, length) against baseline and candidate outputs. A regression is a case that passes on baseline and fails on candidate. An LLM judge can be enabled per call for a second opinion; the verdict only depends on the checks. Use it before shipping a prompt or model change.
| Name | Required | Description | Default |
|---|---|---|---|
| judge | No | Optional pairwise LLM judge on the server's configured OpenAI-compatible endpoint (e.g. Ollama). Off by default; it never changes the verdict. | |
| golden | Yes | Golden cases with deterministic checks. | |
| baseline | Yes | Outputs of the current prompt/model, keyed by case id. | |
| candidate | Yes | Outputs of the proposed prompt/model, keyed by case id. |
| Name | Required | Description |
|---|---|---|
| cases | Yes | |
| judge | No | |
| perCase | Yes | |
| summary | Yes | |
| verdict | Yes | |
| regressions | Yes | Cases that pass on baseline and fail on candidate. |
| improvements | Yes | Cases that fail on baseline and pass on candidate. |
| baselinePassRate | Yes | |
| candidatePassRate | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Beyond the annotations, it explains that a regression is a baseline pass plus candidate fail, and that the optional LLM judge is a second opinion whose output never changes the verdict. That is meaningful behavioral context for an open-world judge call. The idempotentHint=false annotation is not explained; the deterministic-check wording could leave an agent wondering whether repeated calls may differ.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Four short sentences, all of which earn their place: purpose, regression definition, judge caveat, and usage. The purpose is front-loaded with no filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
An output schema exists, so return values need not be explained, and the annotations carry the safety profile. The description covers the check semantics and judge behavior adequately for a tool with nested parameters. The main gap is the missing guidance on when to choose this tool over its siblings.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the schema already documents golden, baseline, candidate, and judge. The description's definition of baseline and candidate as the current and proposed outputs adds useful conceptual framing, but it gives no parameter-level detail beyond that. A baseline 3 applies when the schema carries the detail.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb and resource: run a golden set's deterministic checks (exact, regex, contains, JSON Schema, length) against baseline and candidate outputs. It is unambiguous about what the tool does. It does not differentiate from siblings such as compare_configs or run_suite, which an agent could plausibly confuse it with.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It gives a clear usage context: "Use it before shipping a prompt or model change." That tells the agent when the tool is appropriate. It names no exclusions or alternative tools, so the agent must infer that compare_configs or run_suite handle other cases.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Score a ranked retrieval run against relevance judgements (qrels). Returns recall@k, precision@k, nDCG@k (graded, linear gain), hit rate@k and MRR, averaged over judged queries, plus per-query values. Use it to check whether one retriever is good enough; to choose between several, use compare_configs. Deterministic; no model calls.
| Name | Required | Description | Default |
|---|---|---|---|
| k | No | Cut-offs to evaluate. Defaults to [1, 3, 5, 10]. | |
| run | Yes | Ranked retrieval output: { queryId: [docId, ...] }, best match first. Example: { "q1": ["d7", "d3", "d9"] } | |
| qrels | Yes | Relevance judgements: { queryId: { docId: grade } }. Grade 0 means not relevant; higher grades are more relevant (used by nDCG). Example: { "q1": { "d3": 2, "d7": 1 } } |
| Name | Required | Description |
|---|---|---|
| metrics | Yes | Metric name to value, e.g. { "recall@5": 0.83, "mrr": 0.71 } |
| queries | Yes | Queries with at least one relevant document. |
| perQuery | Yes | |
| missingFromRun | Yes | Judged queries absent from the run (scored as an empty ranking). |
| skippedQueries | Yes | Queries in qrels with no relevant documents (excluded from averages). |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare read-only, idempotent, non-destructive, closed-world. The description adds genuinely new behavioral facts: determinism, no model calls, and that scores are averaged over judged queries with per-query values also returned. Minor gap: nothing about how unjudged queries or missing qrels entries are handled, which matters for interpreting the averages.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three tight sentences: what it computes, how to use it versus the alternative, and its execution properties. Front-loaded with the core action and metrics; no filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Complete for a three-parameter evaluation tool: outputs described, aggregation semantics stated, sibling routing explicit, and an output schema covers return structure. Nothing needed to call it correctly is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, with defaults for k and worked examples for both qrels and run documented in the schema. The description adds no parameter detail beyond that, so the baseline 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource (score a ranked retrieval run against qrels) and enumerates the exact metrics returned. It names the sibling tool it is not (compare_configs) so an agent can distinguish the two without opening schemas.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly states when to use it ('check whether one retriever is good enough') and when to use the alternative ('to choose between several, use compare_configs'). No inference required.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Turn the results of the other evalsmith tools into a Markdown summary, a JSON document or JUnit XML for CI dashboards. Use it last, passing the structuredContent of each tool you ran. Optional gates (metric -> minimum) decide the overall pass/fail; returns the rendered report and the pass flag.
| Name | Required | Description | Default |
|---|---|---|---|
| name | No | Suite name shown in the report title. | evalsmith |
| gates | No | Minimum values, e.g. { "recall@5": 0.8, "tool_calls": 1 }. | |
| format | No | markdown for a PR comment or job summary, json for storage and diffs, junit for CI test dashboards. | markdown |
| retrieval | No | structuredContent of evaluate_retrieval. | |
| toolCalls | No | structuredContent of check_tool_calls. | |
| comparison | No | structuredContent of compare_configs. | |
| regression | No | structuredContent of detect_prompt_regression. |
| Name | Required | Description |
|---|---|---|
| format | Yes | |
| passed | Yes | |
| content | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare read-only, idempotent, non-destructive behavior, so the description's job is lighter. It adds genuinely useful behavior: gates map metrics to minimums and determine the overall pass/fail, and the tool returns both the rendered report and the pass flag. It does not say what happens when a gated metric is absent or input is partial.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three sentences, each earning its place: what it produces, when to call it and with what input, then gates and return values. No repetition of schema fields or annotation restatements.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a seven-parameter aggregator with an output schema and rich annotations, the description covers the essential wiring between sibling tools and the gate semantics. The main remaining gap is edge-case behavior: missing metrics, partial reports, or how optional inputs combine.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the baseline is 3, but the description adds meaning the schema cannot: the retrieval, toolCalls, comparison, and regression objects are meant to be the structuredContent of specific sibling tools. It also clarifies that gates are metric-to-minimum pairs that control the overall verdict. Format and name parameters remain schema-documented only.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource: turning evalsmith tool results into a Markdown summary, JSON document, or JUnit XML report. The three output formats and the aggregation role distinguish it clearly from the sibling evaluation tools.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly says to use it last and to pass the structuredContent of each tool already run. That is a concrete sequencing rule that an agent can follow without inference; no competing alternative exists for this terminal step.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Run a whole evalsmith.yaml suite from the workspace, the same as evalsmith run in CI: retrieval metrics, config comparison, tool-call checks and prompt regression, read from the files the suite points at, then the gates. Use it when the data lives in files rather than in the conversation. Reads files only inside the workspace root; with the judge on it reaches the configured model.
| Name | Required | Description | Default |
|---|---|---|---|
| gates | No | Minimums that override the suite's gates of the same name, e.g. { "recall@5": 0.8, "tool_calls": 1 }. | |
| judge | No | Run the LLM judge when the suite enables it, on the server's configured endpoint. Off by default; it never changes the verdict. | |
| config | No | Path to an evalsmith.yaml suite, relative to the server's workspace root (EVALSMITH_ROOT, the directory the server started in by default). Every file the suite references must also be inside the root. Omit it to run the bundled example suite. | |
| format | No | Format of the report returned with the results: markdown, json or junit. | markdown |
| Name | Required | Description |
|---|---|---|
| suite | Yes | Every section's results and the evaluated gates. |
| format | Yes | |
| passed | Yes | True when every gate passes and no prompt regressed. |
| report | Yes | The suite rendered in the requested format. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnly, openWorld, non-idempotent, non-destructive. The description adds behavior beyond that: execution is confined to files inside the workspace root, the judge flag causes a call to the configured model, and a gate-evaluation phase runs after the metrics. It does not say what happens when a gate fails (error vs. report with failing status), which is the one behavioral detail an agent would want.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loaded with the action and its component checks, then usage, then execution constraints. The first sentence is long and packs a lot, and the file-reading constraint is stated twice in slightly different words (files the suite points at / files only inside workspace root), but there is no filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Output schema exists, so return shape need not be described, and annotations cover the safety profile. The description supplies what structured fields cannot: the workspace-root containment rule, the external model reach when judge is on, and the gate phase. Missing only the failure semantics of unmet gates.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, with per-parameter descriptions covering gates overrides, judge, config path and format, so the description is not required to re-explain them. It only reinforces that the suite is read "from the files the suite points at" and that gates run at the end. Baseline 3 for a fully documented schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource (run an evalsmith.yaml suite) and enumerates exactly what it composes: retrieval metrics, config comparison, tool-call checks, prompt regression, then gates. That enumeration maps onto the sibling tools (evaluate_retrieval, compare_configs, check_tool_calls, detect_prompt_regression) and makes clear this is the bundled all-in-one runner rather than any single check.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
"Use it when the data lives in files rather than in the conversation" gives a concrete selection rule against the conversation-oriented siblings, and "the same as `evalsmith run` in CI" anchors the use case. It does not explicitly name the individual check tools as alternatives for single-metric runs, so the routing is implied rather than spelled out.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
check_tool_calls8 fields changedInput schema / properties / expected / items / properties / args / additionalProperties / oneOfPrevious value: -[
- {
- "properties": {
- "type": {
- "const": "exact",
- "type": "string"
- },
- "value": {}
- },
- "required": [
- "type",
- "value"
- ],
- "type": "object"
- },
- {
- "properties": {
- "flags": {
- "type": "string"
- },
- "pattern": {
- "type": "string"
- },
- "type": {
- "const": "regex",
- "type": "string"
- }
- },
- "required": [
- "type",
- "pattern"
- ],
- "type": "object"
- },
- {
- "properties": {
- "type": {
- "const": "subset",
- "type": "string"
- },
- "value": {}
- },
- "required": [
- "type",
- "value"
- ],
- "type": "object"
- },
- {
- "properties": {
- "type": {
- "const": "any",
- "type": "string"
- }
- },
- "required": [
- "type"
- ],
- "type": "object"
- }
-]New value: +[
+ {
+ "description": "The argument must deep-equal value.",
+ "properties": {
+ "type": {
+ "const": "exact",
+ "type": "string"
+ },
+ "value": {}
+ },
+ "required": [
+ "type",
+ "value"
+ ],
+ "type": "object"
+ },
+ {
+ "description": "The argument, as a string, must match the JavaScript regex pattern.",
+ "properties": {
+ "flags": {
+ "type": "string"
+ },
+ "pattern": {
+ "type": "string"
+ },
+ "type": {
+ "const": "regex",
+ "type": "string"
+ }
+ },
+ "required": [
+ "type",
+ "pattern"
+ ],
+ "type": "object"
+ },
+ {
+ "description": "Every key in value must be present in the argument with an equal value.",
+ "properties": {
+ "type": {
+ "const": "subset",
+ "type": "string"
+ },
+ "value": {}
+ },
+ "required": [
+ "type",
+ "value"
+ ],
+ "type": "object"
+ },
+ {
+ "description": "The argument must be present; any value passes.",
+ "properties": {
+ "type": {
+ "const": "any",
+ "type": "string"
+ }
+ },
+ "required": [
+ "type"
+ ],
+ "type": "object"
+ }
+]Input schema / properties / expected / items / properties / tool / descriptionAdded value: +"Name of the tool the agent should call."Input schema / properties / trace / descriptionPrevious value: -"Tool calls the agent made, in order."New value: +"Tool calls the agent made, in order. Accepts evalsmith calls ({ \"tool\", \"args\" }), Anthropic Messages (messages or content blocks with type \"tool_use\"), OpenAI Chat messages with \"tool_calls\" and OpenAI Responses items with type \"function_call\"; other conversation items are skipped."Input schema / properties / trace / items / additionalPropertiesAdded value: +{}Input schema / properties / trace / items / propertiesRemoved value: -{
- "args": {
- "additionalProperties": {},
- "default": {},
- "propertyNames": {
- "type": "string"
- },
- "type": "object"
- },
- "tool": {
- "minLength": 1,
- "type": "string"
- }
-}Input schema / properties / trace / items / propertyNamesAdded value: +{
+ "type": "string"
+}Input schema / properties / trace / items / requiredRemoved value: -[
- "tool"
-]Output schema / properties / callsAdded value: +{
+ "description": "Tool names of the trace, in call order. Used to draw the expected vs. actual view.",
+ "items": {
+ "type": "string"
+ },
+ "type": "array"
+}compare_configs3 fields changedInput schema / properties / bootstrap / properties / confidence / descriptionAdded value: +"Confidence level of the intervals, e.g. 0.95."Input schema / properties / bootstrap / properties / samples / descriptionAdded value: +"Bootstrap resamples of the query set. More is slower and steadier."Input schema / properties / bootstrap / properties / seed / descriptionAdded value: +"Random seed; the same seed gives the same intervals."detect_prompt_regression5 fields changedInput schema / properties / golden / items / properties / checks / descriptionAdded value: +"Checks the output must pass, all of them."Input schema / properties / golden / items / properties / checks / items / oneOfPrevious value: -[
- {
- "properties": {
- "trim": {
- "default": true,
- "type": "boolean"
- },
- "type": {
- "const": "exact",
- "type": "string"
- },
- "value": {
- "type": "string"
- }
- },
- "required": [
- "type",
- "value"
- ],
- "type": "object"
- },
- {
- "properties": {
- "flags": {
- "type": "string"
- },
- "pattern": {
- "type": "string"
- },
- "type": {
- "const": "regex",
- "type": "string"
- }
- },
- "required": [
- "type",
- "pattern"
- ],
- "type": "object"
- },
- {
- "properties": {
- "caseSensitive": {
- "default": false,
- "type": "boolean"
- },
- "type": {
- "const": "contains",
- "type": "string"
- },
- "value": {
- "type": "string"
- }
- },
- "required": [
- "type",
- "value"
- ],
- "type": "object"
- },
- {
- "properties": {
- "schema": {
- "additionalProperties": {},
- "description": "JSON Schema the output must satisfy.",
- "propertyNames": {
- "type": "string"
- },
- "type": "object"
- },
- "type": {
- "const": "json_schema",
- "type": "string"
- }
- },
- "required": [
- "type",
- "schema"
- ],
- "type": "object"
- },
- {
- "properties": {
- "max": {
- "maximum": 9007199254740991,
- "minimum": 0,
- "type": "integer"
- },
- "min": {
- "maximum": 9007199254740991,
- "minimum": 0,
- "type": "integer"
- },
- "type": {
- "const": "length",
- "type": "string"
- }
- },
- "required": [
- "type"
- ],
- "type": "object"
- }
-]New value: +[
+ {
+ "description": "The output must equal value, trimmed unless trim is false.",
+ "properties": {
+ "trim": {
+ "default": true,
+ "type": "boolean"
+ },
+ "type": {
+ "const": "exact",
+ "type": "string"
+ },
+ "value": {
+ "type": "string"
+ }
+ },
+ "required": [
+ "type",
+ "value"
+ ],
+ "type": "object"
+ },
+ {
+ "description": "The output must match the JavaScript regex pattern.",
+ "properties": {
+ "flags": {
+ "type": "string"
+ },
+ "pattern": {
+ "type": "string"
+ },
+ "type": {
+ "const": "regex",
+ "type": "string"
+ }
+ },
+ "required": [
+ "type",
+ "pattern"
+ ],
+ "type": "object"
+ },
+ {
+ "description": "The output must contain value, case-insensitive by default.",
+ "properties": {
+ "caseSensitive": {
+ "default": false,
+ "type": "boolean"
+ },
+ "type": {
+ "const": "contains",
+ "type": "string"
+ },
+ "value": {
+ "type": "string"
+ }
+ },
+ "required": [
+ "type",
+ "value"
+ ],
+ "type": "object"
+ },
+ {
+ "properties": {
+ "schema": {
+ "additionalProperties": {},
+ "description": "JSON Schema the output must satisfy.",
+ "propertyNames": {
+ "type": "string"
+ },
+ "type": "object"
+ },
+ "type": {
+ "const": "json_schema",
+ "type": "string"
+ }
+ },
+ "required": [
+ "type",
+ "schema"
+ ],
+ "type": "object"
+ },
+ {
+ "description": "The output length in characters must be within min and max.",
+ "properties": {
+ "max": {
+ "maximum": 9007199254740991,
+ "minimum": 0,
+ "type": "integer"
+ },
+ "min": {
+ "maximum": 9007199254740991,
+ "minimum": 0,
+ "type": "integer"
+ },
+ "type": {
+ "const": "length",
+ "type": "string"
+ }
+ },
+ "required": [
+ "type"
+ ],
+ "type": "object"
+ }
+]Input schema / properties / golden / items / properties / id / descriptionAdded value: +"Case id; baseline and candidate outputs are keyed by it."Input schema / properties / judge / properties / enabled / descriptionAdded value: +"Set true to ask the judge."Input schema / properties / judge / properties / model / descriptionAdded value: +"Judge model; defaults to EVALSMITH_JUDGE_MODEL on the server."generate_report3 fields changedInput schema / properties / format / descriptionAdded value: +"markdown for a PR comment or job summary, json for storage and diffs, junit for CI test dashboards."Input schema / properties / name / descriptionAdded value: +"Suite name shown in the report title."Input schema / properties / toolCalls / properties / callsAdded value: +{
+ "description": "Tool names of the trace, in call order. Used to draw the expected vs. actual view.",
+ "items": {
+ "type": "string"
+ },
+ "type": "array"
+}run_suitev0.1.0check_tool_callscompare_configsdetect_prompt_regressionevaluate_retrievalgenerate_reportScored across 6 tools
Each tool targets a distinct evaluation task, and descriptions explicitly distinguish close cases like evaluate_retrieval vs. compare_configs. The only overlap is run_suite, which subsumes the other tools when data lives in files, but its scope is clarified as file/workspace-driven.
All names follow a consistent snake_case verb_noun pattern: evaluate_retrieval, compare_configs, check_tool_calls, detect_prompt_regression, generate_report, run_suite. No mixed conventions or vague verbs.
Six tools is well within the ideal 3-15 range and each tool has a clear, non-redundant purpose in the evaluation workflow. No tool feels like filler or a thin wrapper.
The surface covers retrieval scoring, config comparison, tool-call checking, prompt regression, reporting, and full-suite execution, which covers the core evaluation lifecycle. Minor gaps exist around dataset/qrels management and suite discovery, but agents can work around them via workspace files.
MCP-native AI evaluation: rubric audits, eval suites, and proof reports for AI/LLM output.
Prompt evals over MCP: run a prompt on your dataset, score each output 1-5 with an LLM judge.
MCP server providing access to the Scorecard API to evaluate and optimize LLM systems.
152 free QA tools for agents: injection scan, evals, RAG, web audits. Smaller packs: /mcp/{pack}.
