MCP Stats Tools Server

A real Model Context Protocol server exposing three statistical tools — survival analysis, paired significance testing, and distribution drift detection — plus an evaluation harness measuring quality, latency, and cost. Built to close two specific gaps that kept recurring across job postings: hands-on MCP experience, and evaluation of AI-adjacent systems beyond just correctness.

app/tools.py     app/server.py           app/evaluate.py
  pure logic  -->  MCP wiring         -->  quality checks (known ground truth)
  no MCP           (real protocol,         latency (real timing, real calls)
  involved         real error contract)    cost (illustrative, clearly labelled)

Running it

pip install -r requirements.txt
make test       # 18 tests
make evaluate   # quality, latency, and cost report
make run        # starts the server over stdio

Inspect it with the official tooling, no code changes needed:

npx @modelcontextprotocol/inspector python -m app.server

Nothing here needs an API key or any paid service. The tools are deterministic statistics, not LLM calls.

Related MCP server: MCP ML Monitor

The tools

• survival_analysis -- Kaplan-Meier curve(s), and a log-rank significance test when exactly two groups are given.

• compare_two_samples -- paired t-test and Wilcoxon signed-rank test, plus a bootstrap confidence interval on the mean difference.

• check_distribution_drift -- Kolmogorov-Smirnov test and Population Stability Index between a reference and current sample.

All three reuse the exact statistical methods validated in the sibling supply-chain-decision-lab project, reimplemented here as self-contained functions (no dependency on that repo) so this server can be launched standalone -- a real requirement for an MCP server, not a style choice.

Evaluation: quality, latency, cost

app/evaluate.py is structured around the JD language it was built to answer, kept genuinely separate because each is a different question:

Quality -- checked against known ground truth, the same discipline used throughout this whole portfolio: two groups generated with a real hazard difference must be flagged significant; two generated identically must not be. A known 10-unit injected difference must be recovered by compare_two_samples. A genuine distribution shift must be caught by check_distribution_drift; a stable one must not. All five checks pass.

Latency -- measured by timing real calls through the actual MCPServer.call_tool() path, not the bare Python function underneath it, so protocol overhead (argument validation, result serialisation) is included in what's reported. Representative numbers from one run: check_distribution_drift ~1.3ms, compare_two_samples ~9ms, survival_analysis ~40ms (it does bootstrap resampling, so the ordering is expected, not a red flag).

Cost -- these tools are deterministic and local, not metered per-token like an LLM call, so there is no real dollar figure to report honestly. What is honest: converting measured latency into an illustrative monthly cost using a clearly-labelled, made-up compute rate (ILLUSTRATIVE_COMPUTE_RATE_USD_PER_VCPU_HOUR), demonstrating the calculation a real deployment would run against its actual cloud invoice, without pretending a fabricated number is a real one.

Bugs found while building this

Left in deliberately, same as every sibling project -- the process of finding these is part of what this project demonstrates.

1. dict return types silently produced no structured output. The MCP SDK needs enough type information to build a schema; a bare -> dict annotation doesn't provide it, so results came back only as JSON-in-text, not structured_content -- not an error, just quietly less useful than it looked. Diagnosed by comparing against a working -> int example, fixed by annotating with dict[str, Any].

2. A plain ValueError was classified as a crash, not a reported error. tools.py raises ordinary ValueError for anticipated problems (mismatched lengths, too few observations) -- correct for a module tested independent of MCP, deliberately. But the SDK's contract treats any un-translated exception as an UnexpectedToolError (a real crash: full traceback logged, generic message reaches the caller), where its own ToolError is reported cleanly. Fixed with one decorator in server.py that translates ValueError into ToolError at the protocol boundary, keeping tools.py itself completely MCP-agnostic, so the separation between "logic" and "protocol contract" stays in exactly one place.

3. A test asserted the wrong layer's contract. An early test expected server.call_tool() to return is_error=True for bad input. The SDK's own docstring says otherwise: called programmatically, call_tool() raises ToolError for anticipated failures -- the graceful is_error=True result form only appears at the actual wire-protocol handler a live client connects through, a layer these in-process tests don't exercise. Fixed by asserting the real, documented behaviour (ToolError is raised, and specifically not its UnexpectedToolError subclass) instead of a plausible-sounding but incorrect assumption.

Layout

app/tools.py       pure statistical logic, zero MCP involved, independently tested
app/server.py      MCP wiring: three tools, error-contract translation
app/evaluate.py    quality / latency / cost harness
tests/test_tools.py   18 tests: logic + real protocol integration

What is not built

• No LLM ever calls these tools in this repo -- that would need a real agent host (Claude Desktop, an ADK agent, etc.) wired to this server over stdio, which is a live-integration step outside what this project needed to prove.

• Cost is illustrative by necessity, not measured against a real bill -- see above.

• No SSE/HTTP transport is exercised, only stdio -- the MCP SDK supports both, and stdio is what a local tool-calling agent typically uses.