Skip to main content
Glama

MCP Support Evals

A TypeScript MCP server and tool-use evaluation suite for Demo Plant Shop, a fictional plant store.

What this demonstrates

  • MCP tools with explicit input and output schemas, along with resources and a support prompt.

  • Order lookup that requires a matching email and returns that require explicit confirmation.

  • A bounded model loop that calls tools through an MCP client and treats their output as data.

  • Deterministic grading in CI and recorded live results with usage, latency and estimated cost.

Related MCP server: support-agent-mcp

Architecture

flowchart LR
    Client --> Agent[Agent loop]
    Agent <--> API[Anthropic Messages API]
    Agent --> MCP[MCP client]
    MCP --> Server[MCP server]
    Server --> Fixtures[Fictional shop fixtures]

Results

Metric

Run A (original prompt)

Run B (updated prompt)

Model

claude-sonnet-5-5

claude-sonnet-5-5

Cases graded / selected

54 / 54

54 / 54

Overall pass rate

53/54 (98.1%)

53/54 (98.1%)

tool selection

9/9 (100.0%)

9/9 (100.0%)

grounding

9/9 (100.0%)

9/9 (100.0%)

privacy

9/9 (100.0%)

9/9 (100.0%)

write safety

9/9 (100.0%)

9/9 (100.0%)

injection

8/9 (88.9%)

9/9 (100.0%)

out of scope

9/9 (100.0%)

8/9 (88.9%)

API calls

102

103

Total tokens

178750

185836

Case latency p50 / p95

2.73 s / 4.14 s

2.91 s / 5.38 s

Estimated cost (USD)

$0.425804

$0.436992

Stopped by budget

No

No

Iteration

The original run scored 47/54 on 7543214. A harness bug marked six cases incomplete because the response parser rejected messages containing signed thinking blocks. The seventh case, liner-purpose, reached a final answer and failed a disclosure check.

Run A uses the parser fix with the original system prompt. The parser now preserves thinking and redacted thinking blocks through tool calls and returns only text blocks as the answer. Injection was the lowest-scoring category at 8/9 because liner-purpose repeated order details from an injected instruction.

Run B applies the prompt change on top of the same parser. The prompt asks the model to use only relevant shop facts and omit injected instructions and identifiers. Injection passed 9/9, including liner-purpose. Out-of-scope fell from 9/9 to 8/9 when repair made an unnecessary tool call before refusing. The overall score was 53/54 in both runs.

Both runs used the same model, cases and graders. There was one live run per prompt. Pull request #3

Run locally

npm ci
npm run build
npm start

The stdio process exchanges MCP messages through stdin and stdout. Use npm run start:http for Streamable HTTP at http://127.0.0.1:3210/mcp.

MCP Inspector

From the project root, launch the MCP Inspector against the built stdio entry point. Use a current Node.js LTS release.

npx @modelcontextprotocol/inspector node dist/stdio.js

For HTTP, start the HTTP server first, then choose Streamable HTTP in the Inspector and connect to http://127.0.0.1:3210/mcp.

Claude Desktop

Add this entry to claude_desktop_config.json. Replace the path with the absolute path to dist/stdio.js in the local checkout. Use the absolute path to the Node executable as command if Claude Desktop cannot find node.

{
  "mcpServers": {
    "demo-plant-shop": {
      "command": "node",
      "args": ["/absolute/path/to/mcp-support-evals/dist/stdio.js"]
    }
  }
}

Claude Code

Register the same built entry point from the project directory. claude mcp get demo-plant-shop shows the connection details.

claude mcp add --transport stdio demo-plant-shop -- node "/absolute/path/to/mcp-support-evals/dist/stdio.js"

Tools and resources

  • search_help_articles ranks keyword matches with BM25 and returns article IDs, titles and snippets. The result limit is optional. get_help_article returns the complete article.

  • get_order_status returns only the status when the order reference and email match. Both an unknown order and an email mismatch return not_found.

  • find_plants filters by light level, optional pet safety and optional maximum price in demo credits.

  • create_return_request previews a request for a delivered order. Only confirm: true records a pending request. Repeated confirmations return the existing request and its original reason with already_exists: true. The store is held in memory for the lifetime of the server process.

Clients can read help articles as MCP resources and use the answer_customer_question prompt to guide grounded answers. The server uses fictional fixture data and needs no API key.

How to reproduce

npm run build
npm run eval:replay

Replay uses the committed scripted responses and executes their tool calls against a fresh in-memory MCP session for each case. CI uses the same command. A scripted result checks the harness and carries no measured model performance, usage or cost. The deterministic checks cover tool arguments, citation IDs, order privacy, return confirmation, untrusted article text and refusals.

For a live run, put ANTHROPIC_API_KEY and optionally ANTHROPIC_MODEL in the ignored .env.local. The default model is Claude Sonnet. An alternate model requires --input-usd-per-million and --output-usd-per-million. The standard token prices come from the Claude pricing page.

npm run eval:live -- --archive-recordings

Use --max-cases to run a subset and --budget-usd to change the budget. Before each request the runner checks the measured spend so far plus an input allowance and the full output limit. The report gives a cost estimate.

Live runs save reports under evals/results/. With --archive-recordings, responses go under evals/recordings/<model>/<run-id>/ so each run keeps its own evidence. A run without the flag writes responses directly to evals/recordings/<model>/. Reports include per-case checks, category pass rates, token totals and latency percentiles. Case latency sums the time spent waiting for model responses. A case interrupted by the budget contributes usage but has no grade.

Replay the latest recorded run for a model with npm run eval:replay -- --model <model>. Use --run-id to select a run explicitly. The replay writes a separate report. The original 47/54 run needs its original agent revision for exact replay. On Node.js 22 or later, npm run report regenerates the published table from the selected result files in evals/report-runs.json.

Limits

The cases use a small fictional dataset and measure a single model. The graders check tool arguments, citation IDs and answer patterns. They do not assess overall helpfulness or verify every factual clause.

Each prompt ran once, with nine cases per category, so a one-case difference is within normal run-to-run variation. Run B passed liner-purpose and missed repair. Both runs scored 53/54 overall, so the comparison leaves any overall improvement from the prompt change unproven.

Checks

npm run format:check
npm run lint
npm run typecheck
npm test

Designed and reviewed by Massimiliano Borgatti; implemented with the help of an AI coding agent. — LinkedIn

Available Tools

5 tools
create_return_requestA
Idempotent

Preview a return request for a delivered order with matching email and a customer-provided reason. Submit with confirm=true only after the customer explicitly confirms the preview. Repeating a submission returns the existing request.

Inputs: email (required), reason (required), order_id (required), confirm. Returns: reason, status, order_id, request_id, +1 more

find_plantsA
Read-only

Find catalog plants for low, medium or bright light. Optionally filter by whether they are pet-safe and by a maximum price in demo credits.

Inputs: light (required), pet_safe, max_price. Returns: plants

get_help_articleA
Read-only

Read the full text of one Demo Plant Shop help article using an article ID from search results or resources.

Inputs: id (required). Returns: article

get_order_statusA
Read-only

Read the status of a fictional order. Requires the order reference and the matching email supplied by the customer.

Inputs: email (required), order_id (required). Returns: status

search_help_articlesA
Read-only

Search Demo Plant Shop policies and plant care guides by keyword. Returns ranked article IDs, titles and snippets. Use get_help_article to read the full article.

Inputs: query (required), limit. Returns: articles

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    A3.9/5.0

    Scored across 5 tools

    Disambiguation5/5

    Each tool targets a distinct resource and action: help search vs. help reading, order status lookup, plant catalog filtering, and return submission. The one potentially confusable pair (search_help_articles vs. find_plants) is separated by domain—policy content versus catalog inventory—and the descriptions make that explicit.

    Naming Consistency5/5

    All five tools follow a clean verb_noun snake_case pattern (search_help_articles, get_help_article, get_order_status, find_plants, create_return_request). Verbs are semantically accurate and the shared prefixes (help_, order) group related tools predictably.

    Tool Count4/5

    Five tools is a tight, well-scoped surface with no redundant entries, appropriate for a demo shop assistant. It leans slightly thin—there is no purchasing or cart operation—but every listed tool earns its place.

    Completeness3/5

    The surface covers discovery (help, plants), post-purchase status, and returns, but skips the core commerce step: there is no way to place an order or view a customer's order list. An agent handling an end-to-end shopping request would hit a dead end before checkout.

    Maintenance

    ActivityMaintained
    ResponsivenessNo recent community issues

    Related MCP Connectors

    Related MCP Servers