how-to-test-an-mcp-server

A runnable companion repo for testing an MCP server: a minimal two-tool MCP Python server, MCP Inspector CLI protocol checks, deterministic pytest unit tests on the tool functions, a Claude-based tool-selection eval harness with semantic assertions across N runs, and transport-specific (stdio/HTTP/SSE) and auth/authorization pytest suites.

Companion code for the Autonoma blog post: How to Test an MCP Server

Requirements

• Python 3.10+

• Node 18+ and jq (only for the MCP Inspector CLI checks)

• An ANTHROPIC_API_KEY (only for the tool-selection eval harness)

Related MCP server: MCP Basic Create Tool

Quickstart

git clone https://github.com/Autonoma-Tools/how-to-test-an-mcp-server.git
cd how-to-test-an-mcp-server
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
python src/server.py

Project structure

src/
  __init__.py
  tools.py       Plain-Python get_weather / create_ticket -- no MCP, no LLM
  auth.py        Shared JWT auth helpers for the HTTP/SSE transports
  server.py      The MCP server: stdio (default), plus --transport http/sse
scripts/
  inspector_checks.sh   MCP Inspector CLI protocol checks (stdio)
tests/
  test_tools_unit.py        Deterministic unit tests
  test_transport_stdio.py   stdio transport handshake + clean-exit tests
  test_transport_http.py    HTTP transport + bearer-token scoping tests
  test_transport_sse.py     SSE transport incremental delivery + resume tests
  test_auth.py              Auth: rejection, scoping, expiry/refresh
eval/
  eval_tool_selection.py    Claude-based tool-selection eval (N=15 runs/scenario)
examples/
  quickstart_client.py   End-to-end stdio client demo
  run_eval_demo.sh       How to run the eval harness
  run_test_suites.sh     Runs every layer below, in order
conftest.py              Makes `from src.tools import ...` work from any cwd
requirements.txt

• src/ — the MCP server and its plain-Python tool implementations.

• scripts/ — the MCP Inspector CLI protocol-check script.

• tests/ — the pytest suites for the unit, transport, and auth layers.

• eval/ — the Claude-based tool-selection eval harness.

• examples/ — runnable examples you can execute as-is.

The five testing layers

This repo mirrors the testing pyramid described in the blog post. Each layer answers a different question about the server.

1. Protocol-level checks (MCP Inspector CLI)

"Does the server speak MCP correctly over stdio?"

bash scripts/inspector_checks.sh

Starts src/server.py under @modelcontextprotocol/inspector --cli, asserts tools/list returns both tools with non-empty schemas, and that tools/call on get_weather returns content mentioning temperature. Requires Node 18+ and jq.

2. Deterministic unit tests

"Is the underlying tool logic correct, independent of MCP entirely?"

pytest tests/test_tools_unit.py -v

Imports get_weather / create_ticket directly from src/tools.py — no protocol, no transport, no LLM. Exact-match assertions only. This is the fastest, cheapest, most reliable layer, and should catch the large majority of regressions before anything else runs.

3. Tool-selection eval harness

"Does an LLM reliably pick the right tool for a natural-language ask?"

export ANTHROPIC_API_KEY=sk-ant-...
python eval/eval_tool_selection.py

LLM tool-calling is probabilistic, so this harness runs each scenario 15 times against Claude and reports a per-scenario pass rate rather than a single pass/fail. Scenarios below a 90% pass rate are flagged FLAKY. Assertions here are property-based (correct tool, argument satisfies a loose check) rather than exact-match.

4. Transport tests (stdio / HTTP / SSE)

"Does the server behave correctly across every transport it supports?"

# stdio: handshake + tool call + clean process exit, no manual setup
pytest tests/test_transport_stdio.py -v

# HTTP: spawns its own server on :8787
pytest tests/test_transport_http.py -v

# SSE: spawns its own server on :8788, tests incremental delivery + resume
pytest tests/test_transport_sse.py -v

The HTTP and SSE test files start and tear down their own server subprocess (module-scoped pytest fixtures) — you don't need to run python src/server.py --transport http yourself first, though you can (--transport http --port 8787 / --transport sse --port 8787) if you want to poke at it manually with curl.

5. Auth / authorization tests

"Is the auth boundary actually enforced — rejection, scoping, and expiry?"

pytest tests/test_auth.py -v

Covers three angles against the HTTP-transport server (its own instance, on :8789): no-token/invalid-token rejection with no tool side effect, scope enforcement (a token scoped to one tool gets 403 on the other), and token expiry/refresh (an expired JWT is rejected; a freshly issued replacement succeeds over the same connection).

Run everything at once

bash examples/run_test_suites.sh

The HTTP/SSE auth model (read before poking at it)

The HTTP and SSE transports are a small, hand-rolled, stdlib-only JSON-RPC-over-HTTP and Server-Sent-Events layer (see src/server.py) — not part of the official MCP SDK's transports. They exist specifically to give the auth and transport tests something concrete to exercise. Auth is HS256 JWTs verified against a hardcoded demo secret in src/auth.py (JWT_SECRET). This is intentional for a teaching repo and is not production-ready auth — see the module docstrings in src/auth.py and tests/test_auth.py for specifics.

About

This repository is maintained by Autonoma as reference material for the linked blog post. Autonoma builds autonomous AI agents that plan, execute, and maintain end-to-end tests directly from your codebase.

If something here is wrong, out of date, or unclear, please open an issue.

License

Released under the MIT License © 2026 Autonoma Labs.