policy-evidence-mcp

An MCP server that gives AI assistants governed, cited access to two kinds of evidence:

1. Official statistics, live from any SDMX 2.1 API (the OECD's public API by default): find a dataset, understand its dimensions, fetch observations with a citable source URL.

2. Policy documents, through a local retrieval (RAG) index with a sensitivity ceiling: documents above the configured classification are never indexed or returned. Search is by keywords (BM25), or hybrid (keywords plus meaning, from any embedding service).

The server does retrieval and citation. The client model (Claude, Copilot, ChatGPT or any MCP-capable assistant) does the reasoning and writing. No language model runs inside the server; hybrid search only calls an embedding model, which can be local.

Unofficial project. Not affiliated with or endorsed by the OECD or any other data provider. Data retrieved through an API remains subject to that provider's terms of use. The sample document corpus describes a fictional organisation.

Part of a portfolio. Implements patterns P1 and P2 (with P7 for reranking) of the AI engineering framework, at its Build stage. All repositories.

Architecture

flowchart LR
    A["AI assistant<br/>(MCP client)"] -- "MCP over stdio or HTTP" --> V
    subgraph S [policy-evidence-mcp]
      V["Input validation"] --> T["Tools"]
      T --> H["HTTP layer<br/>host allowlist, cache,<br/>rate limit, size cap"]
      T --> R["Retriever: BM25, or hybrid<br/>with embeddings (rank fusion)<br/>sensitivity ceiling"]
    end
    H -- "HTTPS, read-only" --> API[("SDMX API<br/>e.g. OECD")]
    R --> I[("index.json<br/>chunks + citations")]
    C[("Corpus folder<br/>public / internal / restricted")] -- "evidence-mcp ingest<br/>(ceiling applied)" --> I
    R -. "query vectors (optional)" .-> E[("Embedding service<br/>Ollama, LLM gateway,<br/>Azure OpenAI")]

Related MCP server: Statline MCP

Tools

Plus one prompt, evidence_brief, that tells the assistant how to answer from cited evidence. All tools are declared read-only to clients (readOnlyHint).

A typical exchange: "How does R&D spending as a share of GDP compare between France and Germany, and are staff allowed to use restricted budget data with a chatbot?" The assistant calls search_datasets → describe_dataset → get_data for the figures and search_documents for the rule, then answers with a source URL for each figure and a citation for each quotation.

Quick start

Requires Python 3.10+.

python -m venv .venv
# Windows (PowerShell): .venv\Scripts\Activate.ps1      macOS/Linux: source .venv/bin/activate
pip install -e ".[dev]"

pytest                                                   # 93 offline tests, about 2 seconds

# The default ceiling is "public"; allow the sample's internal documents too.
export EVIDENCE_MCP_MAX_CLASSIFICATION=internal   # PowerShell: $env:EVIDENCE_MCP_MAX_CLASSIFICATION="internal"
evidence-mcp ingest --corpus sample_corpus
evidence-mcp search "who approves a high-risk AI use case"
evidence-mcp eval --questions evals/sample_questions.jsonl
python scripts/smoke_live.py                             # 3 live requests to the OECD API

To index your own documents, put .md, .txt or .pdf files in a folder (sub-folders named public, internal or restricted set the classification) and run evidence-mcp ingest --corpus <folder>. The corpus/ and index/ folders are git-ignored.

Connect an assistant

Claude Desktop (claude_desktop_config.json), Windows example:

{
  "mcpServers": {
    "policy-evidence": {
      "command": "C:\\path\\to\\policy-evidence-mcp\\.venv\\Scripts\\evidence-mcp.exe",
      "args": ["serve"],
      "env": {
        "EVIDENCE_MCP_INDEX_PATH": "C:\\path\\to\\policy-evidence-mcp\\index\\index.json",
        "EVIDENCE_MCP_MAX_CLASSIFICATION": "internal"
      }
    }
  }
}

To inspect the tools interactively: npx @modelcontextprotocol/inspector evidence-mcp serve.

A skill tells the assistant how to use the tools well: the three statistics calls in order, narrow keys, a citation on every figure and passage, and passages treated as data. Copy skills/policy-evidence into your assistant's skills folder (for Claude Code, ~/.claude/skills/).

Over HTTP (for clients that connect by URL):

evidence-mcp serve --transport streamable-http --port 8000     # http://127.0.0.1:8000/mcp
docker build -t policy-evidence-mcp . && docker run --rm -p 127.0.0.1:8000:8000 policy-evidence-mcp

Configuration

All settings are environment variables with safe defaults.

Security model

Tool arguments come from a language model, and document text may contain anything, so both are treated as untrusted.

• Read-only by construction. No tool writes, sends or deletes. The only network destination is the configured API base URL; the model cannot supply a URL (no SSRF).

• Strict input validation. Every identifier, key and period is matched against a narrow pattern before it reaches a URL, which rules out path and query injection.

• Sensitivity ceiling, applied twice. Documents above the ceiling are skipped at ingestion, and results are filtered again at query time in case an index was built with a higher clearance than the server runs with. The index records only the number of excluded documents, not their file names. Unknown labels are treated as most sensitive.

• Untrusted content marked as such. Control and zero-width characters are stripped at ingestion, and every search result tells the client that passages are data, not instructions (a mitigation for indirect prompt injection, not a guarantee).

• Bounded responses and resource use. Row, code, passage and byte caps; timeouts; brief retries with backoff; a persisted hourly rate limit so restarts cannot exceed the provider's quota.

• Transport. stdio for local use (logs go to stderr so they never corrupt the protocol stream). The HTTP transport binds to 127.0.0.1 by default and rejects requests whose Host header is not allowed (DNS-rebinding protection).

• Authenticated callers, each with their own clearance (0.3). Over HTTP the server requires a bearer token (its own hashed tokens, or Entra ID access tokens) and refuses to start on a network address without one unless told the network itself is the boundary. Each caller sees documents up to the lower of the server's ceiling and their own clearance. See Access management.

SharePoint through Microsoft Graph (0.3)

evidence-mcp sync-sharepoint copies a SharePoint library into a corpus through Microsoft Graph, with app-only access to that one site (Sites.Selected), and classifies each file by its Purview sensitivity label. Unlabelled files, and labels not in the mapping, are treated as restricted, so a labelling gap can only narrow access. Re-syncs download only changed files, move re-labelled files and remove deleted ones; answers cite the SharePoint link. Word documents (.docx) are now indexed too. Setup and permissions: docs/sharepoint.md; example configuration: sharepoint.example.toml.

evidence-mcp sync-sharepoint --config sharepoint.toml --out corpus-sharepoint
evidence-mcp ingest --corpus corpus-sharepoint --ceiling internal

Microsoft 365 Copilot through Copilot Studio (0.3.1)

integrations/copilot-studio connects a Copilot Studio agent to this server as an MCP tool, published to Microsoft 365 Copilot and Teams. The agent calls the server with the signed-in user's Entra ID token, so each person sees documents up to their own clearance, managed by role assignment in Entra; the agent holds no broader credential. The folder has the custom connector (streamable MCP, OAuth 2.0 with Entra ID), the agent's instructions and the setup steps; tests check that the connector asks for exactly the scope the server requires. It is configuration for a tenant, not a tested deployment.

Together with the SharePoint sync, the chain is: documents labelled in Purview, synced through Microsoft Graph with their labels as classification, served to Copilot users within their roles.

Access management (0.3)

Over HTTP, every request needs a bearer token, and each caller sees documents up to the lower of the server's ceiling and their own clearance. Statistics are public and need only a valid token. The server publishes OAuth protected-resource metadata, so MCP clients that support OAuth discover where to get a token; requests without one get 401 with a pointer to it.

Tokens issued by the server (a team server or a pilot):

evidence-mcp token create --name alice --clearance public     --tokens-file tokens.json
evidence-mcp token create --name bob   --clearance internal   --tokens-file tokens.json
evidence-mcp serve --transport streamable-http --auth tokens --tokens-file tokens.json

Tokens are shown once and stored only as SHA-256 hashes; token revoke --name alice removes one.

Microsoft Entra ID (an organisation): the server validates access tokens issued for its API (signature against the tenant's published keys, issuer, audience, expiry, the Evidence.Read scope) and reads the clearance from app roles assigned in Entra (Evidence.Public, Evidence.Internal, Evidence.Restricted, highest wins). Who may see what is then managed where the organisation already manages access: by assigning users or groups to roles.

export EVIDENCE_MCP_ENTRA_TENANT_ID=<tenant id>
export EVIDENCE_MCP_ENTRA_AUDIENCE=api://policy-evidence
evidence-mcp serve --transport streamable-http --host 0.0.0.0 --auth entra \
    --public-url https://evidence.example.org/mcp --allowed-host evidence.example.org

The app registration (exposed API, scope, app roles, assignment) is described step by step in docs/entra-id.md.

Serving on a network address without authentication is refused, unless --allow-unauthenticated states that a private network is the boundary (as inside the platform's container network, where only the agents service can reach the server). Every document search logs the caller and the ceiling applied, never the query text.

Tracing (0.3)

Set OTEL_EXPORTER_OTLP_ENDPOINT (Jaeger, Grafana Tempo, an OpenTelemetry Collector) and install the tracing extra (in the container image). The MCP SDK opens a tools/call <tool> span for every call and continues the caller's trace from the request's _meta, so an agent's tool call and the server's work appear in one trace. The server adds the access decision to the span: the ceiling applied for this caller, the server's ceiling, the number of passages, the search mode and the most sensitive classification returned. The query and the passages are never put on a span. Without an endpoint, tracing is off.

Cost and sustainability

• Server side: no model calls, so no token cost; statistics are cached (catalogue 24 h, structures 7 days, data 1 h), which also keeps load on the public API low.

• Client side: tokens are driven by what tools return, so outputs are compact. get_data lifts columns that are identical on every row into constant_columns (typically halving the table), and every tool has a size cap with a hint on how to narrow the query.

Evaluation

evals/sample_questions.jsonl holds 16 labelled questions over the sample corpus: 12 direct, 2 paraphrased, and 2 security questions that ask about the restricted document, which must never come back. evidence-mcp eval reports hit@1, hit@k, mean reciprocal rank and leaks; CI fails on any leak or if hit@3 drops below 0.8.

Current baseline (ceiling internal, k = 3):

The one miss is instructive. "Can employees rely on a chatbot to pick which job applicants to hire?" shares no words with the policy, which says "Staff must not use AI tools to make decisions about individuals, such as recruitment". Keyword retrieval cannot bridge that vocabulary gap.

evals/paraphrase_questions.jsonl makes the gap measurable: 40 questions worded the way people ask rather than the way policies are written, about ten per document, plus three security questions. Keyword search finds the right document in the top 3 for 0.68 of them (hit@1 0.53, MRR 0.62, no leaks). The first 14 (13 scored, one leak check) are frozen in evals/paraphrase_questions_v1.jsonl; the other 29 were added in October 2026 and turned out easier for keywords (hit@3 0.74 against 0.54).

Hybrid search (0.2)

Hybrid search adds meaning to keywords. At ingest, every chunk also gets an embedding vector; at query time, the keyword ranking and the similarity ranking are merged by reciprocal rank fusion (each chunk scores the sum of 1 / (60 + rank) over both lists). Any OpenAI-compatible embeddings endpoint works: a local model through Ollama (nothing leaves the machine), an internal LLM gateway (which applies the data policy and records the cost) or Azure OpenAI.

ollama pull nomic-embed-text
export EVIDENCE_MCP_EMBEDDINGS_URL=http://localhost:11434/v1
export EVIDENCE_MCP_MAX_CLASSIFICATION=internal
evidence-mcp ingest --corpus sample_corpus --embeddings --embeddings-cache evals/embeddings.json
evidence-mcp eval --questions evals/paraphrase_questions.jsonl --mode compare \
  --embeddings-cache evals/embeddings.json

The comparison prints hit@1, hit@3, MRR and leaks for both modes. The cache file records every vector, so the same comparison replays offline (--offline) in CI once it is committed.

Results (live run, October 2026)

nomic-embed-text through Ollama on a laptop, ceiling internal, k = 3. CI replays these vectors on every push.

Hybrid search closes the gap on the direct questions (it now finds "Can employees rely on a chatbot to pick which job applicants to hire?") and lifts the paraphrases from about half to three quarters, with no leak in either mode. On all 40 paraphrases, four still miss the top 3, for example "Which committee signs off on AI projects that affect people?", where the answer says "Digital Governance Board". That is the case for a reranker (0.4, below).

Rules that keep it safe:

• Documents above the ceiling are never sent for embedding, and the ceiling filters the meaning side of search exactly like the keyword side.

• One model per index. The index records the model that made its vectors. If the service answers with another model, or is down, search uses keywords only and says so in its result (search_mode), rather than mixing incomparable vectors or failing.

• Each result says whether it was found by keywords, by meaning, or both (matched_by).

In governed-ai-platform, the server runs as an internal container, re-indexes at start with a local embedding model through the LLM gateway (whose local_only policy guarantees document text never leaves), and serves the agents over MCP.

Reranking (0.4)

Hybrid search finds words and meaning; it does not read a passage as an answer. The optional reranker does: one call to a chat model per search grades each of the top 20 passages from 0 (unrelated) to 3 (answers the question), and the passages are re-sorted by grade, ties kept in search order.

It is a bounded judgment, not generation (pattern P7):

• A closed answer set. The reply must give every passage a grade from 0 to 3. A missing, extra or out-of-range grade is no decision: search keeps its own order and says so in search_mode. Grades reorder what search found; they cannot add a passage.

• The ceiling first. The reranking model sees only passages the caller may see, and none above EVIDENCE_MCP_RERANK_MAX_CLASSIFICATION when that is set; those keep their positions.

• Passages are data. Only a list of grades is read back, so an instruction planted in a passage can at most change its own grade.

• Off by default. It adds a model call to every search. Use a local model, or the gateway with a local_only team, when the documents are internal.

ollama pull qwen2.5:7b
export EVIDENCE_MCP_RERANK_URL=http://localhost:11434/v1
export EVIDENCE_MCP_RERANK_TIMEOUT=900          # a laptop CPU; a GPU server needs far less
export EVIDENCE_MCP_EMBEDDINGS_URL=http://localhost:11434/v1
export EVIDENCE_MCP_MAX_CLASSIFICATION=internal
CACHE="--embeddings-cache evals/embeddings.json"
evidence-mcp ingest --corpus sample_corpus --out build/hybrid.json --embeddings $CACHE --offline
# Record the grades once with the live model, then replay them offline (CI does)
evidence-mcp eval --questions evals/paraphrase_questions.jsonl --index build/hybrid.json \
  --mode compare --min-hit 0 $CACHE --rerank --rerank-cache evals/rerank-qwen2.5-7b.json
evidence-mcp eval --questions evals/paraphrase_questions.jsonl --index build/hybrid.json \
  --mode compare --min-hit 0 $CACHE --rerank --rerank-cache evals/rerank-qwen2.5-7b.json --offline

The table gains keywords + rerank and hybrid + rerank rows. If the reranking service does not answer for a question (a timeout, an outage), the run says INCOMPLETE and exits with an error: nothing is recorded for that question, so running the command again asks again. An invalid answer from the model, by contrast, is recorded and counted: it is part of what is measured.

Results (live runs, October 2026)

Qwen 2.5 7B (8k context) through Ollama on a laptop CPU (Intel Core i7-13620H, 16 GB, integrated graphics), grading the top 20 passages of each search, on the 43 paraphrase questions (40 scored, three leak checks). The first 14 were graded in one run and the other 29 in a second; both are in evals/rerank-qwen2.5-7b-ctx8k.json, which CI replays.

The same runs on the first 13 scored questions and on the 27 added later (hit@1 / hit@3):

• Reranking still helps, on three times the questions. Hybrid search goes from 28 to 31 of 40 right at rank 1 and from 36 to 38 in the top 3; keyword search from 21 to 28 at rank 1. No leak in any mode. The gain is smaller than the first run suggested (hit@1 +0.08, not +0.16), which is what a set of 13 could not tell.

• Most hybrid searches got no decision, and list length is why. Qwen graded 1 passage of 20 on most hybrid searches: across both runs, 15 of 44 replies for 20 passages were complete, against 30 of 35 for shorter lists (most keyword searches return fewer than 20). Each incomplete reply was no decision, as designed, so search kept its own order: the gains above come from the 14 hybrid searches that were graded, and the closed set kept the other 29 from doing harm. A next run grades the top 10 (EVIDENCE_MCP_RERANK_DEPTH=10), measured the same way.

• It cannot find what search did not return. Four keyword questions return no passage at all, so there is nothing to grade; only hybrid search reaches them. Reranking is a layer on top of hybrid search, not a replacement for it.

• Two questions miss with hybrid search, reranked or not: "Which committee signs off on AI projects that affect people?" and "How long do we keep the paperwork for suppliers once a deal is over?". Reranked keyword search does find the first.

• The added questions are easier, for keywords and for hybrid search alike (hybrid hit@3 0.96 against 0.77). They were written before any search ran on them, not chosen to be hard, so the larger set leaves less room for a reranker to show a gain at rank 3.

• Too slow on a CPU for interactive use. About a minute per graded search (the second run took about 65 minutes for 55 calls). On a laptop that suits batch evaluation, not a person waiting for an answer; a GPU server or a hosted model behind the gateway is the deployment option, measured the same way. Set EVIDENCE_MCP_RERANK_TIMEOUT to what the hardware needs (the first attempt, at a fixed 60 seconds, timed out on every question).

The list answer format did worse (live run, October 2026)

Same model and hardware, on the first 14 questions only, with EVIDENCE_MCP_RERANK_FORMAT=list (one number per passage, in order), compared with the objects format on the same 14. evals/rerank-qwen2.5-7b-ctx8k-list.json holds the grades; CI replays them on evals/paraphrase_questions_v1.jsonl. The list format was not run on the larger set.

• Worse on every measure but one, and twice as slow. The objects format stays the default.

• The failures changed shape, and the prompt explains it. In 8 of the 10 failed replies Qwen returned exactly three grades, whatever the number of passages. The prompt's example was a list for three passages; a small model copied its length. One reply graded 19 of 20, one used a grade of 5. Each was no decision, as designed: the closed set held on both formats.

• What this teaches beyond the reranker: for a small model, an example in the prompt is part of the specification, length included. A next attempt would show no list at all, or one as long as the real one, and be measured the same way.

Limitations and roadmap

• Hybrid retrieval: embeddings alongside BM25 with rank fusion, compared on the same evaluation (0.2)

• Reranking: a model-based reranker over the fused top 20, graded from a closed set, off by default (0.4). On 40 paraphrases: hit@1 0.70 to 0.78 and hit@3 0.90 to 0.95 on hybrid search, about a minute per search on a laptop CPU, with most 20-passage searches getting no decision. A list-of-grades answer format (0.5) measured worse on the first 14; objects stays the default. Next: grading the top 10, then a GPU or hosted run for latency.

• Codes with data only: use SDMX availableconstraint so describe_dataset lists only codes that actually have observations.

• Structured tool output: typed results with output schemas.

• Identity-aware access: bearer tokens or Entra ID, documents filtered per caller (0.3)

• Tracing: OpenTelemetry spans per tool call with the access decision, joined with the agent's trace (0.3)

• Metrics: cache hit rate and upstream latency.

• More providers: the SDMX client is provider-neutral; test against ECB and Eurostat.

Project layout

src/evidence_mcp/
  server.py       MCP tools and prompt
  sdmx.py         SDMX client and parsers (catalogue, structure, CSV data)
  http_cache.py   host allowlist, disk cache, rate limit, retries, size cap
  documents.py    loading (md/txt/pdf), metadata, heading-aware chunking
  retrieval.py    BM25 and hybrid search, sensitivity ceiling, index files
  rerank.py       optional reranking: grades 0-3 from a chat model, recorded for replay
  embeddings.py   OpenAI-compatible embeddings client, recorded vectors for replay
  evaluation.py   hit@k, MRR, leak detection
  validation.py   input patterns
  config.py       settings and classification levels
  cli.py          serve | ingest | search | eval
skills/           a SKILL.md for assistants that call the MCP tools
tests/            offline unit, protocol and end-to-end tests with recorded fixtures
evals/            labelled question sets
sample_corpus/    fictional documents for tests and demos
docs/             design decisions
AGENTS.md         commands and invariants for coding agents (CLAUDE.md imports it)

Development

pytest              # all tests, offline
ruff check . && ruff format --check .

CI (GitHub Actions) runs lint, tests and the retrieval evaluation on Python 3.10 to 3.13, then builds the Docker image. See docs/design-decisions.md for the reasoning behind the main choices.

License

MIT. See LICENSE.