auth-floor-rag-demo

An MCP server with three things production agent systems actually need and demo projects usually skip:

1. An auth floor — the server refuses to start without a token, every request is authenticated, and write tools check the caller's tier per call.

2. A hybrid RAG pipeline — TF-IDF cosine retrieval with a pool-local BM25 re-rank fused by Reciprocal Rank Fusion, over a small bundled corpus.

3. The eval harness that gates it — a frozen query set, written before any tuning, scoring both rankers; the eval exits non-zero below its MRR gate, so it can sit in CI.

This is a distilled, sanitized miniature of a real production system (an MCP control plane fronting a large indexed knowledge store). The architecture, the auth pattern, and the failure lessons are the production ones; the scale is deliberately small enough to run anywhere.

Run it (under 5 minutes)

Requires Python 3.11+.

python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

export DEMO_MCP_TOKENS='dev-read:read:reader, dev-admin:admin:me'
python -m app.server
# listening on http://127.0.0.1:8765/mcp (8 documents, auth floor ON)

Try the floor first:

curl http://127.0.0.1:8765/health        # public: liveness + data_mode label
curl -X POST http://127.0.0.1:8765/mcp   # 401 — no token, no service

And the refusal that defines the floor — start it without tokens:

unset DEMO_MCP_TOKENS
python -m app.server
# FATAL: no MCP token configured ... Refusing to start (exit 2)

Connect any MCP client over streamable HTTP with an Authorization: Bearer dev-read header, e.g. a minimal Python client:

from mcp import ClientSession
from mcp.client.streamable_http import streamablehttp_client

async with streamablehttp_client(
    "http://127.0.0.1:8765/mcp",
    headers={"Authorization": "Bearer dev-read"},
) as (read, write, _):
    async with ClientSession(read, write) as session:
        await session.initialize()
        print(await session.call_tool("search_documents",
                                      {"query": "how does bread rise without commercial yeast"}))

Related MCP server: MCP Knowledge Service

Architecture

            Authorization: Bearer <token:tier>
                            │
   MCP client ──HTTP──▶ BearerAuthMiddleware ──401──▶ ✕
                            │  sets principal (contextvar)
                            ▼
                    FastMCP (streamable HTTP)
                    ┌───────────────────────────────┐
                    │ read tools      write tools   │── require admin tier,
                    │ (any token)     (per-call ✓)  │   checked per call
                    └──────────────┬────────────────┘
                                   ▼
                    Index: chunk → TF-IDF cosine pool
                           → pool-local BM25 re-rank → RRF fusion
                                   ▲
                    eval/queries.json (frozen) ── evaluate() scores
                    both rankers; run_eval tool + CLI expose the gate

Tools (9): whoami, search_documents (ranker selectable: hybrid/tfidf/bm25), get_document, list_documents, summarize_document (extractive), index_stats, run_eval — plus the admin-tier writes add_document and remove_document.

Auth model. Tokens are configured as token:tier[:label] pairs in DEMO_MCP_TOKENS (or a single admin token in DEMO_MCP_TOKEN). Tiers are read < admin. Authentication happens in ASGI middleware on every request — the MCP transport never sees an unauthenticated call — and the resulting principal rides a contextvar so write tools can enforce tier per call, not once at login. Token comparison is constant-time. /health is public but serves no data, and its payload carries data_mode: "sample-corpus" — see the second lesson below for why that field exists.

Retrieval. Documents are chunked paragraph-aware (~700 chars), indexed as TF-IDF vectors, and queried in two stages: cosine similarity builds a candidate pool, BM25 computed over that pool only re-ranks it, and RRF fuses the two orderings. TF-IDF is the key-free, deterministic stand-in at the embedding seam — the interface is where a neural embedder drops in.

The eval, honestly

python eval/run_eval.py scores a frozen set of 16 queries (12 keyword, 4 plain-language paraphrases) against the 8-document demo corpus:

Perfect scores — because eight documents on eight disjoint topics is an easy corpus, and pretending otherwise would be exactly the kind of claim this repo exists to avoid. The demo eval proves the harness runs and the gate bites (lower the threshold in app/evaluate.py and watch it fail); it is not evidence about retrieval quality at scale.

For that, here are the production system's numbers, measured on a frozen 16-query headline set against a ~48,500-chunk snapshot of a 4,406-document corpus, before and after the fix this demo's hybrid design distills (pure embedding cosine → + pool-local BM25 re-rank with full-pool RRF):

Two honest caveats the production report itself records: the anchored metrics understate quality (gold chunks were capped at 8 per query while 9–97 were eligible), and on a 3-query broad-control set the anchored numbers for v2.2 were worse even though every control's top-5 stayed inside the expected topic (topic-level hit@5 was 16/16 headline, 3/3 control). The fix was adopted for the headline gain with the tradeoff stated, not hidden.

What broke (the chapter demos usually omit)

1. The pipe, not the destination. A remote host appeared unreachable for hours: the database of "the host is down" evidence kept growing while the host was fine. The actual fault was a local proxy bridge that read its input through a buffered stream — small client writes were swallowed before ever reaching the network, so connections half-opened and died. Fix: pump bytes unbuffered, and add a transport self-test that runs before any "host down" conclusion is allowed. Lesson: verify the layer you're standing on before diagnosing the layer you're looking at.

2. Fixture data wearing live data's clothes. A dashboard rendered sample fixtures through the same code path as live data, visually indistinguishable from the real thing — "it loads" was being read as "it works." Fix: the payload itself now declares its provenance (data_mode in every health/status response — this demo's /health does the same), so any consumer can tell fixture from live in-band, without trusting convention or memory. Lesson: test doubles must be labeled in the data, not in the documentation.

3. Green by inspection isn't green. A large integration branch looked mergeable on review. Running the full suite in a clean environment found exactly one genuine defect — a missing dependency — hiding among failures that turned out to be environment artifacts (a VM clock-resolution quirk making two timing assertions land on exactly 0.0 ms; a test-runner quirk in one sandbox). Both kinds of red look identical in a summary line. The rule that came out of it: no merge without a recorded green run from a clean checkout, and an "environment artifact" must be proven — reproduced and root-caused — never assumed. This repo follows the rule: the test suite spins up the real server process and exercises the floor over HTTP, because the floor only means something at the transport.

Tests

pip install -r requirements-dev.txt
python -m pytest        # 19 tests: auth units, retrieval units,
                        # and end-to-end HTTP tests against the real
                        # server process (401s, tier denial, no-start)
python eval/run_eval.py # the frozen eval + gate

Layout

app/auth.py       token store, tiers, ASGI bearer middleware, no-start floor
app/rag.py        chunking, TF-IDF + pool-local BM25, RRF hybrid search
app/evaluate.py   frozen-eval scoring (hit@1/@3, MRR@5) + the gate
app/server.py     FastMCP wiring: 9 tools, /health, startup
corpus/           8 sample documents (original text, written for this demo)
eval/queries.json the frozen query set · eval/run_eval.py the CLI gate
tests/            unit + live-transport end-to-end tests

License

MIT. The corpus documents are original texts written for this demo.