# how-to-test-an-mcp-server

A runnable companion repo for testing an MCP server: a minimal two-tool MCP Python server, MCP Inspector CLI protocol checks, deterministic pytest unit tests on the tool functions, a Claude-based tool-selection eval harness with semantic assertions across N runs, and transport-specific (stdio/HTTP/SSE) and auth/authorization pytest suites.

> Companion code for the Autonoma blog post: **[How to Test an MCP Server](https://getautonoma.com/blog/how-to-test-an-mcp-server)**

## Requirements

- Python 3.10+
- Node 18+ and `jq` (only for the MCP Inspector CLI checks)
- An `ANTHROPIC_API_KEY` (only for the tool-selection eval harness)

## Quickstart

```bash
git clone https://github.com/Autonoma-Tools/how-to-test-an-mcp-server.git
cd how-to-test-an-mcp-server
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
python src/server.py
```

## Project structure

```
src/
  __init__.py
  tools.py       Plain-Python get_weather / create_ticket -- no MCP, no LLM
  auth.py        Shared JWT auth helpers for the HTTP/SSE transports
  server.py      The MCP server: stdio (default), plus --transport http/sse
scripts/
  inspector_checks.sh   MCP Inspector CLI protocol checks (stdio)
tests/
  test_tools_unit.py        Deterministic unit tests
  test_transport_stdio.py   stdio transport handshake + clean-exit tests
  test_transport_http.py    HTTP transport + bearer-token scoping tests
  test_transport_sse.py     SSE transport incremental delivery + resume tests
  test_auth.py              Auth: rejection, scoping, expiry/refresh
eval/
  eval_tool_selection.py    Claude-based tool-selection eval (N=15 runs/scenario)
examples/
  quickstart_client.py   End-to-end stdio client demo
  run_eval_demo.sh       How to run the eval harness
  run_test_suites.sh     Runs every layer below, in order
conftest.py              Makes `from src.tools import ...` work from any cwd
requirements.txt
```

- `src/` — the MCP server and its plain-Python tool implementations.
- `scripts/` — the MCP Inspector CLI protocol-check script.
- `tests/` — the pytest suites for the unit, transport, and auth layers.
- `eval/` — the Claude-based tool-selection eval harness.
- `examples/` — runnable examples you can execute as-is.

## The five testing layers

This repo mirrors the testing pyramid described in the blog post. Each layer answers a different question about the server.

### 1. Protocol-level checks (MCP Inspector CLI)

*"Does the server speak MCP correctly over stdio?"*

```bash
bash scripts/inspector_checks.sh
```

Starts `src/server.py` under `@modelcontextprotocol/inspector --cli`, asserts `tools/list` returns both tools with non-empty schemas, and that `tools/call` on `get_weather` returns content mentioning `temperature`. Requires Node 18+ and `jq`.

### 2. Deterministic unit tests

*"Is the underlying tool logic correct, independent of MCP entirely?"*

```bash
pytest tests/test_tools_unit.py -v
```

Imports `get_weather` / `create_ticket` directly from `src/tools.py` — no protocol, no transport, no LLM. Exact-match assertions only. This is the fastest, cheapest, most reliable layer, and should catch the large majority of regressions before anything else runs.

### 3. Tool-selection eval harness

*"Does an LLM reliably pick the right tool for a natural-language ask?"*

```bash
export ANTHROPIC_API_KEY=sk-ant-...
python eval/eval_tool_selection.py
```

LLM tool-calling is probabilistic, so this harness runs each scenario 15 times against Claude and reports a per-scenario pass rate rather than a single pass/fail. Scenarios below a 90% pass rate are flagged `FLAKY`. Assertions here are property-based (correct tool, argument satisfies a loose check) rather than exact-match.

### 4. Transport tests (stdio / HTTP / SSE)

*"Does the server behave correctly across every transport it supports?"*

```bash
# stdio: handshake + tool call + clean process exit, no manual setup
pytest tests/test_transport_stdio.py -v

# HTTP: spawns its own server on :8787
pytest tests/test_transport_http.py -v

# SSE: spawns its own server on :8788, tests incremental delivery + resume
pytest tests/test_transport_sse.py -v
```

The HTTP and SSE test files start and tear down their own server subprocess (module-scoped pytest fixtures) — you don't need to run `python src/server.py --transport http` yourself first, though you can (`--transport http --port 8787` / `--transport sse --port 8787`) if you want to poke at it manually with `curl`.

### 5. Auth / authorization tests

*"Is the auth boundary actually enforced — rejection, scoping, and expiry?"*

```bash
pytest tests/test_auth.py -v
```

Covers three angles against the HTTP-transport server (its own instance, on `:8789`): no-token/invalid-token rejection with no tool side effect, scope enforcement (a token scoped to one tool gets 403 on the other), and token expiry/refresh (an expired JWT is rejected; a freshly issued replacement succeeds over the same connection).

### Run everything at once

```bash
bash examples/run_test_suites.sh
```

## The HTTP/SSE auth model (read before poking at it)

The HTTP and SSE transports are a small, hand-rolled, stdlib-only JSON-RPC-over-HTTP and Server-Sent-Events layer (see `src/server.py`) — not part of the official MCP SDK's transports. They exist specifically to give the auth and transport tests something concrete to exercise. Auth is HS256 JWTs verified against a **hardcoded demo secret** in `src/auth.py` (`JWT_SECRET`). This is intentional for a teaching repo and is not production-ready auth — see the module docstrings in `src/auth.py` and `tests/test_auth.py` for specifics.

## About

This repository is maintained by [Autonoma](https://getautonoma.com) as reference material for the linked blog post. Autonoma builds autonomous AI agents that plan, execute, and maintain end-to-end tests directly from your codebase.

If something here is wrong, out of date, or unclear, please [open an issue](https://github.com/Autonoma-Tools/how-to-test-an-mcp-server/issues/new).

## License

Released under the [MIT License](./LICENSE) © 2026 Autonoma Labs.