evalsmith-mcp
Server Configuration
Describes the environment variables required to run the server.
| Name | Required | Description | Default |
|---|---|---|---|
No arguments | |||
Instructions
Guidance the server publishes about itself, which clients place ahead of the tool catalog so the model reads it before choosing anything.
This server publishes no instructions, or was last inspected before Glama recorded them.
Capabilities
Features and capabilities supported by this server
Protocol revision2025-11-25
| Capability | Details |
|---|---|
| tools | {
"listChanged": true
} |
| prompts | {
"listChanged": true
} |
| resources | {
"listChanged": true
} |
| completions | {} |
Tools
Functions exposed to the LLM to take actions
| Name | Description |
|---|---|
| evaluate_retrievalA | Score a ranked retrieval run against relevance judgements (qrels). Returns recall@k, precision@k, nDCG@k (graded, linear gain), hit rate@k and MRR, averaged over judged queries, plus per-query values. Use it to check whether one retriever is good enough; to choose between several, use compare_configs. Deterministic; no model calls. |
| compare_configsA | Compare 2-3 retrieval configurations on the same qrels. Ranks them on a primary metric, attaches seeded paired-bootstrap confidence intervals, and says whether the winner's lead over the runner-up is significant (interval of the gap excludes zero). Use it before switching retriever, chunking or reranker, so a small gap on a few queries is not mistaken for a win. |
| check_tool_callsA | Compare an agent's tool-call trace (evalsmith, Anthropic Messages or OpenAI format, as logged) with the expected calls: right tool, right order (strict or lax) and arguments that satisfy exact / regex / subset / any matchers. Use it to test an agent change against a known-good trace. Returns a pass flag, a score and a reason for every mismatch. |
| detect_prompt_regressionA | Run a golden set's deterministic checks (exact, regex, contains, JSON Schema, length) against baseline and candidate outputs. A regression is a case that passes on baseline and fails on candidate. An LLM judge can be enabled per call for a second opinion; the verdict only depends on the checks. Use it before shipping a prompt or model change. |
| generate_reportA | Turn the results of the other evalsmith tools into a Markdown summary, a JSON document or JUnit XML for CI dashboards. Use it last, passing the structuredContent of each tool you ran. Optional gates (metric -> minimum) decide the overall pass/fail; returns the rendered report and the pass flag. |
| run_suiteA | Run a whole evalsmith.yaml suite from the workspace, the same as |
Prompts
Interactive templates invoked by user choice
| Name | Description |
|---|---|
| compare_rag_configs | Pick the best retrieval configuration and say whether the lead is real. |
| audit_agent_trace | Check that an agent called the right tools, in order, with the right arguments. |
| review_prompt_change | Decide whether a prompt or model change is safe to ship. |
Resources
Contextual data attached and managed by the client
| Name | Description |
|---|---|
| suite | Example suite config used by `evalsmith run` (retrieval, comparison, agent, prompts, gates). |
| rag-corpus | 20 help-center articles of a fictional invoicing app. |
| rag-queries | 12 user questions over the corpus. |
| rag-qrels | Graded relevance judgements (2 = answers the question, 1 = related). |
| rag-run-bm25 | Top-10 ranking per query from a lexical BM25 configuration. |
| rag-run-hybrid | Top-10 ranking per query from a BM25 + dense hybrid configuration. |
| rag-run-hybrid-rerank | Top-10 ranking per query from the hybrid configuration plus a cross-encoder reranker. |
| agent-trace | Tool calls of a support agent handling a duplicate-charge refund. |
| agent-expected | Expected calls for that task, with exact / regex / subset / any matchers. |
| prompts-golden | Six golden cases with deterministic checks. |
| prompts-baseline | Outputs of the current prompt. |
| prompts-candidate | Outputs of an improved prompt (no regressions). |
| prompts-candidate-regressed | Outputs of a prompt change that breaks two cases. |
TDQS
Scored across 6 tools
Each tool targets a distinct evaluation task, and descriptions explicitly distinguish close cases like evaluate_retrieval vs. compare_configs. The only overlap is run_suite, which subsumes the other tools when data lives in files, but its scope is clarified as file/workspace-driven.
All names follow a consistent snake_case verb_noun pattern: evaluate_retrieval, compare_configs, check_tool_calls, detect_prompt_regression, generate_report, run_suite. No mixed conventions or vague verbs.
Six tools is well within the ideal 3-15 range and each tool has a clear, non-redundant purpose in the evaluation workflow. No tool feels like filler or a thin wrapper.
The surface covers retrieval scoring, config comparison, tool-call checking, prompt regression, reporting, and full-suite execution, which covers the core evaluation lifecycle. Minor gaps exist around dataset/qrels management and suite discovery, but agents can work around them via workspace files.