# MCP Stats Tools Server

A real Model Context Protocol server exposing three statistical tools —
survival analysis, paired significance testing, and distribution drift
detection — plus an evaluation harness measuring quality, latency, and cost.
Built to close two specific gaps that kept recurring across job postings:
hands-on MCP experience, and evaluation of AI-adjacent systems beyond just
correctness.

```
app/tools.py     app/server.py           app/evaluate.py
  pure logic  -->  MCP wiring         -->  quality checks (known ground truth)
  no MCP           (real protocol,         latency (real timing, real calls)
  involved         real error contract)    cost (illustrative, clearly labelled)
```

## Running it

```bash
pip install -r requirements.txt
make test       # 18 tests
make evaluate   # quality, latency, and cost report
make run        # starts the server over stdio
```

Inspect it with the official tooling, no code changes needed:
```bash
npx @modelcontextprotocol/inspector python -m app.server
```

Nothing here needs an API key or any paid service. The tools are
deterministic statistics, not LLM calls.

## The tools

- **`survival_analysis`** -- Kaplan-Meier curve(s), and a log-rank
  significance test when exactly two groups are given.
- **`compare_two_samples`** -- paired t-test and Wilcoxon signed-rank test,
  plus a bootstrap confidence interval on the mean difference.
- **`check_distribution_drift`** -- Kolmogorov-Smirnov test and Population
  Stability Index between a reference and current sample.

All three reuse the exact statistical methods validated in the sibling
`supply-chain-decision-lab` project, reimplemented here as self-contained
functions (no dependency on that repo) so this server can be launched
standalone -- a real requirement for an MCP server, not a style choice.

## Evaluation: quality, latency, cost

`app/evaluate.py` is structured around the JD language it was built to
answer, kept genuinely separate because each is a different question:

**Quality** -- checked against known ground truth, the same discipline used
throughout this whole portfolio: two groups generated with a real hazard
difference must be flagged significant; two generated identically must not
be. A known 10-unit injected difference must be recovered by
`compare_two_samples`. A genuine distribution shift must be caught by
`check_distribution_drift`; a stable one must not. All five checks pass.

**Latency** -- measured by timing real calls through the actual
`MCPServer.call_tool()` path, not the bare Python function underneath it, so
protocol overhead (argument validation, result serialisation) is included in
what's reported. Representative numbers from one run: `check_distribution_drift`
~1.3ms, `compare_two_samples` ~9ms, `survival_analysis` ~40ms (it does
bootstrap resampling, so the ordering is expected, not a red flag).

**Cost** -- these tools are deterministic and local, not metered per-token
like an LLM call, so there is no real dollar figure to report honestly. What
*is* honest: converting measured latency into an illustrative monthly cost
using a clearly-labelled, made-up compute rate
(`ILLUSTRATIVE_COMPUTE_RATE_USD_PER_VCPU_HOUR`), demonstrating the
calculation a real deployment would run against its actual cloud invoice,
without pretending a fabricated number is a real one.

## Bugs found while building this

Left in deliberately, same as every sibling project -- the process of
finding these is part of what this project demonstrates.

1. **`dict` return types silently produced no structured output.** The MCP
   SDK needs enough type information to build a schema; a bare `-> dict`
   annotation doesn't provide it, so results came back only as JSON-in-text,
   not `structured_content` -- not an error, just quietly less useful than
   it looked. Diagnosed by comparing against a working `-> int` example,
   fixed by annotating with `dict[str, Any]`.

2. **A plain `ValueError` was classified as a crash, not a reported error.**
   `tools.py` raises ordinary `ValueError` for anticipated problems
   (mismatched lengths, too few observations) -- correct for a module tested
   independent of MCP, deliberately. But the SDK's contract treats any
   un-translated exception as an `UnexpectedToolError` (a real crash: full
   traceback logged, generic message reaches the caller), where its own
   `ToolError` is reported cleanly. Fixed with one decorator in `server.py`
   that translates `ValueError` into `ToolError` at the protocol boundary,
   keeping `tools.py` itself completely MCP-agnostic, so the separation
   between "logic" and "protocol contract" stays in exactly one place.

3. **A test asserted the wrong layer's contract.** An early test expected
   `server.call_tool()` to return `is_error=True` for bad input. The SDK's
   own docstring says otherwise: called programmatically, `call_tool()`
   *raises* `ToolError` for anticipated failures -- the graceful
   `is_error=True` result form only appears at the actual wire-protocol
   handler a live client connects through, a layer these in-process tests
   don't exercise. Fixed by asserting the real, documented behaviour
   (`ToolError` is raised, and specifically not its `UnexpectedToolError`
   subclass) instead of a plausible-sounding but incorrect assumption.

## Layout

```
app/tools.py       pure statistical logic, zero MCP involved, independently tested
app/server.py      MCP wiring: three tools, error-contract translation
app/evaluate.py    quality / latency / cost harness
tests/test_tools.py   18 tests: logic + real protocol integration
```

## What is not built

- No LLM ever calls these tools in this repo -- that would need a real agent
  host (Claude Desktop, an ADK agent, etc.) wired to this server over stdio,
  which is a live-integration step outside what this project needed to prove.
- Cost is illustrative by necessity, not measured against a real bill -- see
  above.
- No SSE/HTTP transport is exercised, only stdio -- the MCP SDK supports
  both, and stdio is what a local tool-calling agent typically uses.