policy-evidence-mcp
by flam7791
README.md
# policy-evidence-mcp
[](https://github.com/flam7791/policy-evidence-mcp/actions/workflows/ci.yml) [](LICENSE) 
An [MCP](https://modelcontextprotocol.io) server that gives AI assistants **governed, cited access
to two kinds of evidence**:
1. **Official statistics**, live from any SDMX 2.1 API (the OECD's public API by default):
find a dataset, understand its dimensions, fetch observations with a citable source URL.
2. **Policy documents**, through a local retrieval (RAG) index with a **sensitivity ceiling**:
documents above the configured classification are never indexed or returned. Search is by
keywords (BM25), or **hybrid** (keywords plus meaning, from any embedding service).
The server does retrieval and citation. The client model (Claude, Copilot, ChatGPT or any
MCP-capable assistant) does the reasoning and writing. No language model runs inside the server;
hybrid search only calls an embedding model, which can be local.
> **Unofficial project.** Not affiliated with or endorsed by the OECD or any other data provider.
> Data retrieved through an API remains subject to that provider's terms of use. The sample
> document corpus describes a fictional organisation.
> **Part of a portfolio.** Implements patterns [P1][aief-p1] and [P2][aief-p2] (with [P7][aief-p7]
> for reranking) of the [AI engineering framework][aief], at its *Build* stage. [All
> repositories][aief-all].
[aief]: https://github.com/flam7791/ai-engineering-framework
[aief-all]: https://github.com/flam7791
[aief-p1]: https://github.com/flam7791/ai-engineering-framework/blob/main/docs/patterns.md#p1-governed-retrieval-rag-with-a-sensitivity-ceiling
[aief-p2]: https://github.com/flam7791/ai-engineering-framework/blob/main/docs/patterns.md#p2-read-only-mcp-tool-server
[aief-p7]: https://github.com/flam7791/ai-engineering-framework/blob/main/docs/patterns.md#p7-bounded-judgment-a-typed-decision-from-a-closed-answer-set
## Architecture
```mermaid
flowchart LR
A["AI assistant<br/>(MCP client)"] -- "MCP over stdio or HTTP" --> V
subgraph S [policy-evidence-mcp]
V["Input validation"] --> T["Tools"]
T --> H["HTTP layer<br/>host allowlist, cache,<br/>rate limit, size cap"]
T --> R["Retriever: BM25, or hybrid<br/>with embeddings (rank fusion)<br/>sensitivity ceiling"]
end
H -- "HTTPS, read-only" --> API[("SDMX API<br/>e.g. OECD")]
R --> I[("index.json<br/>chunks + citations")]
C[("Corpus folder<br/>public / internal / restricted")] -- "evidence-mcp ingest<br/>(ceiling applied)" --> I
R -. "query vectors (optional)" .-> E[("Embedding service<br/>Ollama, LLM gateway,<br/>Azure OpenAI")]
```
## Tools
| Tool | What it does | Output bounded by |
|---|---|---|
| `search_datasets` | Keyword search over the API's dataset catalogue | `limit` (max 50) |
| `describe_dataset` | Dimensions in key order, code lists, how to write a series key | `max_codes` per dimension |
| `find_codes` | Look up codes in one dimension ("germany" → `DEU`) | `limit` |
| `get_data` | Observations as a compact table, with source URL and citation | `max_rows` (max 1,000) and a 25 MB response cap |
| `search_documents` | Top passages for a question, each with a citation | `top_k` (max 10) × 1,200 characters |
| `list_documents` | Documents the server is cleared to show | clearance level |
Plus one prompt, `evidence_brief`, that tells the assistant how to answer from cited evidence.
All tools are declared read-only to clients (`readOnlyHint`).
A typical exchange: *"How does R&D spending as a share of GDP compare between France and
Germany, and are staff allowed to use restricted budget data with a chatbot?"* The assistant calls
`search_datasets` → `describe_dataset` → `get_data` for the figures and `search_documents` for the
rule, then answers with a source URL for each figure and a citation for each quotation.
## Quick start
Requires Python 3.10+.
```bash
python -m venv .venv
# Windows (PowerShell): .venv\Scripts\Activate.ps1 macOS/Linux: source .venv/bin/activate
pip install -e ".[dev]"
pytest # 93 offline tests, about 2 seconds
# The default ceiling is "public"; allow the sample's internal documents too.
export EVIDENCE_MCP_MAX_CLASSIFICATION=internal # PowerShell: $env:EVIDENCE_MCP_MAX_CLASSIFICATION="internal"
evidence-mcp ingest --corpus sample_corpus
evidence-mcp search "who approves a high-risk AI use case"
evidence-mcp eval --questions evals/sample_questions.jsonl
python scripts/smoke_live.py # 3 live requests to the OECD API
```
To index your own documents, put `.md`, `.txt` or `.pdf` files in a folder (sub-folders named
`public`, `internal` or `restricted` set the classification) and run
`evidence-mcp ingest --corpus <folder>`. The `corpus/` and `index/` folders are git-ignored.
### Connect an assistant
Claude Desktop (`claude_desktop_config.json`), Windows example:
```json
{
"mcpServers": {
"policy-evidence": {
"command": "C:\\path\\to\\policy-evidence-mcp\\.venv\\Scripts\\evidence-mcp.exe",
"args": ["serve"],
"env": {
"EVIDENCE_MCP_INDEX_PATH": "C:\\path\\to\\policy-evidence-mcp\\index\\index.json",
"EVIDENCE_MCP_MAX_CLASSIFICATION": "internal"
}
}
}
}
```
To inspect the tools interactively: `npx @modelcontextprotocol/inspector evidence-mcp serve`.
A skill tells the assistant how to use the tools well: the three statistics calls in order,
narrow keys, a citation on every figure and passage, and passages treated as data. Copy
[`skills/policy-evidence`](skills/policy-evidence/SKILL.md) into your assistant's skills folder
(for Claude Code, `~/.claude/skills/`).
Over HTTP (for clients that connect by URL):
```bash
evidence-mcp serve --transport streamable-http --port 8000 # http://127.0.0.1:8000/mcp
docker build -t policy-evidence-mcp . && docker run --rm -p 127.0.0.1:8000:8000 policy-evidence-mcp
```
## Configuration
All settings are environment variables with safe defaults.
| Variable | Default | Purpose |
|---|---|---|
| `EVIDENCE_MCP_SDMX_BASE_URL` | `https://sdmx.oecd.org/public/rest` | Any SDMX 2.1 REST endpoint (HTTPS only) |
| `EVIDENCE_MCP_INDEX_PATH` | `index/index.json` | Document index to serve |
| `EVIDENCE_MCP_MAX_CLASSIFICATION` | `public` | Highest level indexed and returned: `public`, `internal`, `restricted` |
| `EVIDENCE_MCP_CACHE_DIR` | `~/.cache/evidence-mcp` | HTTP cache and rate-limit state |
| `EVIDENCE_MCP_RATE_LIMIT_PER_HOUR` | `50` | Upstream requests per hour (the OECD allows about 60) |
| `EVIDENCE_MCP_HTTP_TIMEOUT` | `30` | Seconds per upstream request |
| `EVIDENCE_MCP_EMBEDDINGS_URL` | none | OpenAI-compatible embeddings endpoint; unset = keywords only |
| `EVIDENCE_MCP_EMBEDDINGS_MODEL` | `nomic-embed-text` | Embedding model (`auto` through a gateway) |
| `EVIDENCE_MCP_EMBEDDINGS_API_KEY` | none | Key for that endpoint, if it needs one |
| `EVIDENCE_MCP_RERANK_URL` | none | OpenAI-compatible chat endpoint that grades the top passages; unset = no reranking |
| `EVIDENCE_MCP_RERANK_MODEL` | `qwen2.5:7b` | Reranking model (`auto` through a gateway) |
| `EVIDENCE_MCP_RERANK_API_KEY` | none | Key for that endpoint, if it needs one |
| `EVIDENCE_MCP_RERANK_DEPTH` | `20` | Passages graded per search (2 to 50) |
| `EVIDENCE_MCP_RERANK_FORMAT` | `objects` | How the model answers: `objects` (passage number and grade for each) or `list` (one grade per passage, in order, shorter for small models) |
| `EVIDENCE_MCP_RERANK_TIMEOUT` | `60` | Seconds per grading call; after that, search keeps its own order. A 7B model on a laptop CPU needs several minutes for 20 passages |
| `EVIDENCE_MCP_RERANK_MAX_CLASSIFICATION` | the caller's ceiling | Highest level whose passages the reranking model may see |
| `EVIDENCE_MCP_TOKENS_FILE` | `tokens.json` | Hashed tokens for `--auth tokens` |
| `EVIDENCE_MCP_ENTRA_TENANT_ID` / `_AUDIENCE` | none | Entra ID tenant and the API's Application ID URI, for `--auth entra` |
| `EVIDENCE_MCP_ENTRA_SCOPE` | `Evidence.Read` | Delegated scope a token must carry |
| `EVIDENCE_MCP_ENTRA_ROLES` | `Evidence.Public=public,...` | App role to clearance mapping |
## Security model
Tool arguments come from a language model, and document text may contain anything, so both are
treated as untrusted.
- **Read-only by construction.** No tool writes, sends or deletes. The only network destination
is the configured API base URL; the model cannot supply a URL (no SSRF).
- **Strict input validation.** Every identifier, key and period is matched against a narrow
pattern before it reaches a URL, which rules out path and query injection.
- **Sensitivity ceiling, applied twice.** Documents above the ceiling are skipped at ingestion,
and results are filtered again at query time in case an index was built with a higher
clearance than the server runs with. The index records only the *number* of excluded
documents, not their file names. Unknown labels are treated as most sensitive.
- **Untrusted content marked as such.** Control and zero-width characters are stripped at
ingestion, and every search result tells the client that passages are data, not instructions
(a mitigation for indirect prompt injection, not a guarantee).
- **Bounded responses and resource use.** Row, code, passage and byte caps; timeouts; brief
retries with backoff; a persisted hourly rate limit so restarts cannot exceed the provider's
quota.
- **Transport.** stdio for local use (logs go to stderr so they never corrupt the protocol
stream). The HTTP transport binds to 127.0.0.1 by default and rejects requests whose `Host`
header is not allowed (DNS-rebinding protection).
- **Authenticated callers, each with their own clearance (0.3).** Over HTTP the server requires
a bearer token (its own hashed tokens, or Entra ID access tokens) and refuses to start on a
network address without one unless told the network itself is the boundary. Each caller sees
documents up to the lower of the server's ceiling and their own clearance. See
[Access management](#access-management-03).
## SharePoint through Microsoft Graph (0.3)
`evidence-mcp sync-sharepoint` copies a SharePoint library into a corpus through Microsoft Graph,
with app-only access to that one site (`Sites.Selected`), and classifies each file by its
**Purview sensitivity label**. Unlabelled files, and labels not in the mapping, are treated as
`restricted`, so a labelling gap can only narrow access. Re-syncs download only changed files,
move re-labelled files and remove deleted ones; answers cite the SharePoint link. Word documents
(`.docx`) are now indexed too. Setup and permissions: [docs/sharepoint.md](docs/sharepoint.md);
example configuration: [sharepoint.example.toml](sharepoint.example.toml).
```bash
evidence-mcp sync-sharepoint --config sharepoint.toml --out corpus-sharepoint
evidence-mcp ingest --corpus corpus-sharepoint --ceiling internal
```
## Microsoft 365 Copilot through Copilot Studio (0.3.1)
[integrations/copilot-studio](integrations/copilot-studio/) connects a Copilot Studio agent to
this server as an MCP tool, published to Microsoft 365 Copilot and Teams. The agent calls the
server with the **signed-in user's** Entra ID token, so each person sees documents up to their
own clearance, managed by role assignment in Entra; the agent holds no broader credential. The
folder has the custom connector (streamable MCP, OAuth 2.0 with Entra ID), the agent's
instructions and the setup steps; tests check that the connector asks for exactly the scope the
server requires. It is configuration for a tenant, not a tested deployment.
Together with the SharePoint sync, the chain is: documents labelled in Purview, synced through
Microsoft Graph with their labels as classification, served to Copilot users within their roles.
## Access management (0.3)
Over HTTP, every request needs a bearer token, and each caller sees documents up to **the lower
of the server's ceiling and their own clearance**. Statistics are public and need only a valid
token. The server publishes OAuth protected-resource metadata, so MCP clients that support OAuth
discover where to get a token; requests without one get `401` with a pointer to it.
**Tokens issued by the server** (a team server or a pilot):
```bash
evidence-mcp token create --name alice --clearance public --tokens-file tokens.json
evidence-mcp token create --name bob --clearance internal --tokens-file tokens.json
evidence-mcp serve --transport streamable-http --auth tokens --tokens-file tokens.json
```
Tokens are shown once and stored only as SHA-256 hashes; `token revoke --name alice` removes one.
**Microsoft Entra ID** (an organisation): the server validates access tokens issued for its API
(signature against the tenant's published keys, issuer, audience, expiry, the `Evidence.Read`
scope) and reads the clearance from **app roles** assigned in Entra (`Evidence.Public`,
`Evidence.Internal`, `Evidence.Restricted`, highest wins). Who may see what is then managed where
the organisation already manages access: by assigning users or groups to roles.
```bash
export EVIDENCE_MCP_ENTRA_TENANT_ID=<tenant id>
export EVIDENCE_MCP_ENTRA_AUDIENCE=api://policy-evidence
evidence-mcp serve --transport streamable-http --host 0.0.0.0 --auth entra \
--public-url https://evidence.example.org/mcp --allowed-host evidence.example.org
```
The app registration (exposed API, scope, app roles, assignment) is described step by step in
[docs/entra-id.md](docs/entra-id.md).
Serving on a network address without authentication is refused, unless
`--allow-unauthenticated` states that a private network is the boundary (as inside the platform's
container network, where only the agents service can reach the server). Every document search
logs the caller and the ceiling applied, never the query text.
## Tracing (0.3)
Set `OTEL_EXPORTER_OTLP_ENDPOINT` (Jaeger, Grafana Tempo, an OpenTelemetry Collector) and install
the `tracing` extra (in the container image). The MCP SDK opens a `tools/call <tool>` span for
every call and continues the caller's trace from the request's `_meta`, so an agent's tool call
and the server's work appear in one trace. The server adds the access decision to the span:
the ceiling applied for this caller, the server's ceiling, the number of passages, the search
mode and the most sensitive classification returned. **The query and the passages are never put
on a span.** Without an endpoint, tracing is off.
## Cost and sustainability
- **Server side:** no model calls, so no token cost; statistics are cached (catalogue 24 h,
structures 7 days, data 1 h), which also keeps load on the public API low.
- **Client side:** tokens are driven by what tools return, so outputs are compact. `get_data`
lifts columns that are identical on every row into `constant_columns` (typically halving the
table), and every tool has a size cap with a hint on how to narrow the query.
## Evaluation
`evals/sample_questions.jsonl` holds 16 labelled questions over the sample corpus: 12 direct, 2
paraphrased, and 2 *security* questions that ask about the restricted document, which must never
come back. `evidence-mcp eval` reports hit@1, hit@k, mean reciprocal rank and leaks; CI fails on
any leak or if hit@3 drops below 0.8.
Current baseline (ceiling `internal`, k = 3):
| hit@1 | hit@3 | MRR | leaks |
|---|---|---|---|
| 0.93 | 0.93 | 0.93 | 0 |
The one miss is instructive. *"Can employees rely on a chatbot to pick which job applicants to
hire?"* shares no words with the policy, which says *"Staff must not use AI tools to make
decisions about individuals, such as recruitment"*. Keyword retrieval cannot bridge that
vocabulary gap.
`evals/paraphrase_questions.jsonl` makes the gap measurable: 40 questions worded the way people
ask rather than the way policies are written, about ten per document, plus three security
questions. Keyword search finds the right document in the top 3 for **0.68** of them (hit@1
0.53, MRR 0.62, no leaks). The first 14 (13 scored, one leak check) are frozen in
`evals/paraphrase_questions_v1.jsonl`; the other 29 were added in October 2026 and turned out
easier for keywords (hit@3 0.74 against 0.54).
## Hybrid search (0.2)
Hybrid search adds meaning to keywords. At ingest, every chunk also gets an embedding vector; at
query time, the keyword ranking and the similarity ranking are merged by **reciprocal rank
fusion** (each chunk scores the sum of 1 / (60 + rank) over both lists). Any OpenAI-compatible
embeddings endpoint works: a **local model through Ollama** (nothing leaves the machine), an
internal **LLM gateway** (which applies the data policy and records the cost) or **Azure
OpenAI**.
```bash
ollama pull nomic-embed-text
export EVIDENCE_MCP_EMBEDDINGS_URL=http://localhost:11434/v1
export EVIDENCE_MCP_MAX_CLASSIFICATION=internal
evidence-mcp ingest --corpus sample_corpus --embeddings --embeddings-cache evals/embeddings.json
evidence-mcp eval --questions evals/paraphrase_questions.jsonl --mode compare \
--embeddings-cache evals/embeddings.json
```
The comparison prints hit@1, hit@3, MRR and leaks for both modes. The cache file records every
vector, so the same comparison replays offline (`--offline`) in CI once it is committed.
### Results (live run, October 2026)
`nomic-embed-text` through Ollama on a laptop, ceiling `internal`, k = 3. CI replays these
vectors on every push.
| Question set | Mode | hit@1 | hit@3 | MRR | Leaks |
|---|---|---|---|---|---|
| Direct questions (16) | keywords | 0.93 | 0.93 | 0.93 | 0 |
| | **hybrid** | **1.00** | **1.00** | **1.00** | 0 |
| Paraphrases, first 14 | keywords | 0.46 | 0.54 | 0.52 | 0 |
| | **hybrid** | **0.69** | **0.77** | **0.77** | 0 |
| Paraphrases, all 43 | keywords | 0.53 | 0.68 | 0.62 | 0 |
| | **hybrid** | **0.70** | **0.90** | **0.80** | 0 |
Hybrid search closes the gap on the direct questions (it now finds *"Can employees rely on a
chatbot to pick which job applicants to hire?"*) and lifts the paraphrases from about half to
three quarters, with no leak in either mode. On all 40 paraphrases, four still miss the top 3,
for example *"Which committee signs off on AI projects that affect people?"*, where the answer
says "Digital Governance Board". That is the case for a reranker (0.4, below).
Rules that keep it safe:
- **Documents above the ceiling are never sent for embedding**, and the ceiling filters the
meaning side of search exactly like the keyword side.
- **One model per index.** The index records the model that made its vectors. If the service
answers with another model, or is down, search uses keywords only and says so in its result
(`search_mode`), rather than mixing incomparable vectors or failing.
- Each result says whether it was found by keywords, by meaning, or both (`matched_by`).
In [governed-ai-platform](https://github.com/flam7791/governed-ai-platform), the server runs as
an internal container, re-indexes at start with a local embedding model through the LLM gateway
(whose `local_only` policy guarantees document text never leaves), and serves the agents over
MCP.
## Reranking (0.4)
Hybrid search finds words and meaning; it does not read a passage as an answer. The optional
reranker does: one call to a chat model per search grades each of the top 20 passages from 0
(unrelated) to 3 (answers the question), and the passages are re-sorted by grade, ties kept in
search order.
It is a bounded judgment, not generation ([pattern P7](https://github.com/flam7791/ai-engineering-framework/blob/main/docs/patterns.md)):
- **A closed answer set.** The reply must give every passage a grade from 0 to 3. A missing,
extra or out-of-range grade is no decision: search keeps its own order and says so in
`search_mode`. Grades reorder what search found; they cannot add a passage.
- **The ceiling first.** The reranking model sees only passages the caller may see, and none
above `EVIDENCE_MCP_RERANK_MAX_CLASSIFICATION` when that is set; those keep their positions.
- **Passages are data.** Only a list of grades is read back, so an instruction planted in a
passage can at most change its own grade.
- **Off by default.** It adds a model call to every search. Use a local model, or the gateway
with a `local_only` team, when the documents are internal.
```bash
ollama pull qwen2.5:7b
export EVIDENCE_MCP_RERANK_URL=http://localhost:11434/v1
export EVIDENCE_MCP_RERANK_TIMEOUT=900 # a laptop CPU; a GPU server needs far less
export EVIDENCE_MCP_EMBEDDINGS_URL=http://localhost:11434/v1
export EVIDENCE_MCP_MAX_CLASSIFICATION=internal
CACHE="--embeddings-cache evals/embeddings.json"
evidence-mcp ingest --corpus sample_corpus --out build/hybrid.json --embeddings $CACHE --offline
# Record the grades once with the live model, then replay them offline (CI does)
evidence-mcp eval --questions evals/paraphrase_questions.jsonl --index build/hybrid.json \
--mode compare --min-hit 0 $CACHE --rerank --rerank-cache evals/rerank-qwen2.5-7b.json
evidence-mcp eval --questions evals/paraphrase_questions.jsonl --index build/hybrid.json \
--mode compare --min-hit 0 $CACHE --rerank --rerank-cache evals/rerank-qwen2.5-7b.json --offline
```
The table gains `keywords + rerank` and `hybrid + rerank` rows. If the reranking service does
not answer for a question (a timeout, an outage), the run says INCOMPLETE and exits with an
error: nothing is recorded for that question, so running the command again asks again. An
invalid answer from the model, by contrast, is recorded and counted: it is part of what is
measured.
### Results (live runs, October 2026)
Qwen 2.5 7B (8k context) through Ollama on a laptop CPU (Intel Core i7-13620H, 16 GB, integrated
graphics), grading the top 20 passages of each search, on the 43 paraphrase questions (40 scored,
three leak checks). The first 14 were graded in one run and the other 29 in a second; both are in
`evals/rerank-qwen2.5-7b-ctx8k.json`, which CI replays.
| Mode | hit@1 | hit@3 | MRR | Leaks | Searches with no decision |
|---|---|---|---|---|---|
| keywords | 0.53 | 0.68 | 0.62 | 0 | |
| keywords + rerank | 0.70 | 0.80 | 0.75 | 0 | 5 of 36 |
| hybrid | 0.70 | 0.90 | 0.80 | 0 | |
| **hybrid + rerank** | **0.78** | **0.95** | **0.86** | 0 | **29 of 43** |
The same runs on the first 13 scored questions and on the 27 added later (hit@1 / hit@3):
| Mode | First 13 | Added 27 |
|---|---|---|
| keywords | 0.46 / 0.54 | 0.56 / 0.74 |
| keywords + rerank | 0.62 / 0.69 | 0.74 / 0.85 |
| hybrid | 0.69 / 0.77 | 0.70 / 0.96 |
| hybrid + rerank | 0.85 / 0.85 | 0.74 / 1.00 |
- **Reranking still helps, on three times the questions.** Hybrid search goes from 28 to 31 of
40 right at rank 1 and from 36 to 38 in the top 3; keyword search from 21 to 28 at rank 1. No
leak in any mode. The gain is smaller than the first run suggested (hit@1 +0.08, not +0.16),
which is what a set of 13 could not tell.
- **Most hybrid searches got no decision, and list length is why.** Qwen graded 1 passage of
20 on most hybrid searches: across both runs, 15 of 44 replies for 20 passages were complete,
against 30 of 35 for shorter lists (most keyword searches return fewer than 20). Each incomplete
reply was no decision, as designed, so search kept its own order: the gains above come from
the 14 hybrid searches that were graded, and the closed set kept the other 29 from doing harm.
A next run grades the top 10 (`EVIDENCE_MCP_RERANK_DEPTH=10`), measured the same way.
- **It cannot find what search did not return.** Four keyword questions return no passage at
all, so there is nothing to grade; only hybrid search reaches them. Reranking is a layer on top
of hybrid search, not a replacement for it.
- **Two questions miss with hybrid search, reranked or not:** *"Which committee signs off on AI
projects that affect people?"* and *"How long do we keep the paperwork for suppliers once a
deal is over?"*. Reranked keyword search does find the first.
- **The added questions are easier,** for keywords and for hybrid search alike (hybrid hit@3
0.96 against 0.77). They were written before any search ran on them, not chosen to be hard,
so the larger set leaves less room for a reranker to show a gain at rank 3.
- **Too slow on a CPU for interactive use.** About a minute per graded search (the second run
took about 65 minutes for 55 calls). On a laptop that suits batch evaluation, not a person waiting
for an answer; a GPU server or a hosted model behind the gateway is the deployment option,
measured the same way. Set `EVIDENCE_MCP_RERANK_TIMEOUT` to what the hardware needs (the first
attempt, at a fixed 60 seconds, timed out on every question).
### The `list` answer format did worse (live run, October 2026)
Same model and hardware, on the first 14 questions only, with `EVIDENCE_MCP_RERANK_FORMAT=list`
(one number per passage, in order), compared with the `objects` format on the same 14.
`evals/rerank-qwen2.5-7b-ctx8k-list.json` holds the grades; CI replays them on
`evals/paraphrase_questions_v1.jsonl`. The list format was not run on the larger set.
| Mode | Format | hit@1 | hit@3 | MRR | Searches with no decision | Run time |
|---|---|---|---|---|---|---|
| keywords + rerank | objects | 0.62 | 0.69 | 0.65 | 0 | 25 min (both rows) |
| keywords + rerank | list | 0.54 | 0.54 | 0.56 | 3 | 54 min (both rows) |
| hybrid + rerank | objects | **0.85** | **0.85** | **0.87** | 6 of 14 | |
| hybrid + rerank | list | 0.77 | 0.85 | 0.83 | 7 of 14 | |
- **Worse on every measure but one,** and twice as slow. The `objects` format stays the default.
- **The failures changed shape, and the prompt explains it.** In 8 of the 10 failed replies Qwen
returned exactly three grades, whatever the number of passages. The prompt's example was a
list for three passages; a small model copied its length. One reply graded 19 of 20, one used
a grade of 5. Each was no decision, as designed: the closed set held on both formats.
- **What this teaches beyond the reranker:** for a small model, an example in the prompt is
part of the specification, length included. A next attempt would show no list at all, or one
as long as the real one, and be measured the same way.
## Limitations and roadmap
- [x] **Hybrid retrieval**: embeddings alongside BM25 with rank fusion, compared on the same
evaluation (0.2)
- [x] **Reranking**: a model-based reranker over the fused top 20, graded from a closed set,
off by default (0.4). On 40 paraphrases: hit@1 0.70 to 0.78 and hit@3 0.90 to 0.95 on
hybrid search, about a minute per search on a laptop CPU, with most 20-passage searches
getting no decision. A list-of-grades answer format (0.5) measured worse on the first 14;
`objects` stays the default. Next: grading the top 10, then a GPU or hosted run for
latency.
- [ ] **Codes with data only**: use SDMX `availableconstraint` so `describe_dataset` lists only
codes that actually have observations.
- [ ] **Structured tool output**: typed results with output schemas.
- [x] **Identity-aware access**: bearer tokens or Entra ID, documents filtered per caller (0.3)
- [x] **Tracing**: OpenTelemetry spans per tool call with the access decision, joined with the
agent's trace (0.3)
- [ ] **Metrics**: cache hit rate and upstream latency.
- [ ] **More providers**: the SDMX client is provider-neutral; test against ECB and Eurostat.
## Project layout
```
src/evidence_mcp/
server.py MCP tools and prompt
sdmx.py SDMX client and parsers (catalogue, structure, CSV data)
http_cache.py host allowlist, disk cache, rate limit, retries, size cap
documents.py loading (md/txt/pdf), metadata, heading-aware chunking
retrieval.py BM25 and hybrid search, sensitivity ceiling, index files
rerank.py optional reranking: grades 0-3 from a chat model, recorded for replay
embeddings.py OpenAI-compatible embeddings client, recorded vectors for replay
evaluation.py hit@k, MRR, leak detection
validation.py input patterns
config.py settings and classification levels
cli.py serve | ingest | search | eval
skills/ a SKILL.md for assistants that call the MCP tools
tests/ offline unit, protocol and end-to-end tests with recorded fixtures
evals/ labelled question sets
sample_corpus/ fictional documents for tests and demos
docs/ design decisions
AGENTS.md commands and invariants for coding agents (CLAUDE.md imports it)
```
## Development
```bash
pytest # all tests, offline
ruff check . && ruff format --check .
```
CI (GitHub Actions) runs lint, tests and the retrieval evaluation on Python 3.10 to 3.13, then
builds the Docker image. See [docs/design-decisions.md](docs/design-decisions.md) for the
reasoning behind the main choices.
## License
MIT. See [LICENSE](LICENSE).
This server cannot be deployed
Maintenance
ActivityMaintained
ResponsivenessNo issues