evalsmith-mcp

Generate report

generate_report

Read-onlyIdempotent

Render aggregated evaluation results as Markdown, JSON, or JUnit reports, apply optional metric gates to decide pass/fail, and return the report plus status.

Instructions

Turn the results of the other evalsmith tools into a Markdown summary, a JSON document or JUnit XML for CI dashboards. Use it last, passing the structuredContent of each tool you ran. Optional gates (metric -> minimum) decide the overall pass/fail; returns the rendered report and the pass flag.

Input Schema

Name	Required	Description	Default
`name`	No	Suite name shown in the report title.	evalsmith
`gates`	No	Minimum values, e.g. { "recall@5": 0.8, "tool_calls": 1 }.
`format`	No	markdown for a PR comment or job summary, json for storage and diffs, junit for CI test dashboards.	markdown
`retrieval`	No	structuredContent of evaluate_retrieval.
`toolCalls`	No	structuredContent of check_tool_calls.
`comparison`	No	structuredContent of compare_configs.
`regression`	No	structuredContent of detect_prompt_regression.

Output Schema

Name	Required	Description	Default
`format`	Yes
`passed`	Yes
`content`	Yes

Schema Changelog

Changes observed during successful MCP inspections.

Changed3 schema fields changedOct 29, 2026

addedInput schema / properties / format / description

Added value: +"markdown for a PR comment or job summary, json for storage and diffs, junit for CI test dashboards."

addedInput schema / properties / name / description

Added value: +"Suite name shown in the report title."

addedInput schema / properties / toolCalls / properties / calls

Added value: +{
+  "description": "Tool names of the trace, in call order. Used to draw the expected vs. actual view.",
+  "items": {
+    "type": "string"
+  },
+  "type": "array"
+}

First observedv0.1.0Oct 28, 2026

TDQS

A4.6/5.0

Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare read-only, idempotent, non-destructive behavior, so the description's job is lighter. It adds genuinely useful behavior: gates map metrics to minimums and determine the overall pass/fail, and the tool returns both the rendered report and the pass flag. It does not say what happens when a gated metric is absent or input is partial.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three sentences, each earning its place: what it produces, when to call it and with what input, then gates and return values. No repetition of schema fields or annotation restatements.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a seven-parameter aggregator with an output schema and rich annotations, the description covers the essential wiring between sibling tools and the gate semantics. The main remaining gap is edge-case behavior: missing metrics, partial reports, or how optional inputs combine.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the baseline is 3, but the description adds meaning the schema cannot: the retrieval, toolCalls, comparison, and regression objects are meant to be the structuredContent of specific sibling tools. It also clarifies that gates are metric-to-minimum pairs that control the overall verdict. Format and name parameters remain schema-documented only.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource: turning evalsmith tool results into a Markdown summary, JSON document, or JUnit XML report. The three output formats and the aggregation role distinguish it clearly from the sibling evaluation tools.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly says to use it last and to pass the structuredContent of each tool already run. That is a concrete sequencing rule that an agent can follow without inference; no competing alternative exists for this terminal step.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Deploy Server