# corpus-mcp

A local [MCP](https://modelcontextprotocol.io) server that gives an agent clean,
efficient access to a **local knowledge corpus of offline ZIM archives** —
Wikipedia, medical (MDWiki), developer documentation (DevDocs), and Stack
Exchange — through one uniform interface. No internet, no embeddings, no
vector database: `libzim` full-text search plus deterministic, in-server
content cleaning.

The public MCP surface is exactly two tools:

```text
search(query, limit?)
fetch(ref, sections?)
```

The configured corpus is an **operator** concern, not an agent concern. The
agent only ever:

```text
discover  →  search()
select    →  fetch()
```

## Corpus families

| Corpus | `kind` | Document | Section model |
| --- | --- | --- | --- |
| Wikipedia, MDWiki | `article` | article | heading tree (h2+), lead section id `""` |
| DevDocs (C, CMake, Python) | `documentation` | documentation page | heading tree; in-page TOCs and nav chrome stripped |
| Stack Exchange | `thread` | question + answers | synthetic sections: `question`, `accepted-answer`, `answer-<id>` |

All corpus identification, routing, ZIM access, HTML interpretation, cleanup,
ranking, redirect handling, and normalization remain server responsibilities.
The agent is never required to parse HTML, resolve redirects, construct or
parse references, or know anything about libzim, ZIM namespaces, or corpus
storage internals.

## References

`search()` results carry an opaque `ref` (e.g. `corpus://Wikipedia/Bell_test`);
`fetch()` consumes it. The agent must never construct, parse, or modify a ref,
nor infer the corpus from one:

```text
search() produces ref      fetch() consumes ref
```

## Architecture

```text
Local agent
    │  MCP / Streamable HTTP  →  http://127.0.0.1:8000/mcp
    ▼
┌──────────────────────────────────────────────┐
│ Corpus MCP Server                            │
│  search()  fetch()                           │
│  ├─ CorpusManager (routing, cache,          │
│  │   bounded-concurrency fan-out)           │
│  ├─ federated ranking (RRF + lexical title  │
│  │   reranking + diversity)                 │
│  ├─ adapters: mediawiki / devdocs /         │
│  │   stackexchange                           │
│  ├─ HTML cleaner → Markdown, section trees  │
│  └─ GlobalRef codec (opaque refs)           │
└─────────────┬────────────────────────────────┘
              ▼
      per-library ZIM service (only libzim touchpoint,
      one search lock per archive)
              ▼
      corpus/  (read-only volume, N .zim archives)
      corpus.toml  (manifest: name, adapter, path)
```

The MCP layer exposes no libzim concepts: no namespaces, cluster IDs, raw
entries, MIME types, or raw HTML.

## Prerequisites

* Docker + Docker Compose
* ZIM archives (see below)
* For running the test suite locally: Python 3.12 and `uv` (or pip)

## Corpus layout

The server **never downloads archives itself** — corpus acquisition is
deliberately decoupled from application startup. Default layout:

```text
corpus/
  wikipedia/wikipedia_en_all_nopic_*.zim
  medical/mdwiki_en_all_maxi_*.zim
  devdocs/devdocs_en_cpp_*.zim
  devdocs/devdocs_en_cmake_*.zim
  devdocs/devdocs_en_python_*.zim
  stackexchange/stackoverflow.com_en_all_*.zim
  stackexchange/security.stackexchange.com_en_all_*.zim
  stackexchange/softwareengineering.stackexchange.com_en_all_*.zim
corpus.toml
```

`corpus.toml` names each library, its adapter, its path (relative to the
corpus root), and whether it is enabled:

```toml
version = 1

[[library]]
name = "Wikipedia"
path = "wikipedia/wikipedia_en_all_nopic_2026-06.zim"
adapter = "mediawiki"
enabled = true

[[library]]
name = "CMake-Docs"
path = "devdocs/devdocs_en_cmake_2026-08.zim"
adapter = "devdocs"
enabled = true
```

Disabled entries are not opened or served. Library names must be unique,
adapters must be known, and enabled paths must stay inside the corpus root.
Verify a corpus before starting the server:

```text
make validate-corpus   # opens every archive, reports metadata
make corpus-list       # list configured libraries
```

## Startup / shutdown

```text
make start           # build + start (docker compose, detached)
make logs            # tail logs
make ps              # container status
make stop            # stop (keep containers)
make down            # stop + remove
make restart
make build
```

The MCP endpoint is then available at `http://127.0.0.1:8000/mcp`
(Streamable HTTP). The host port is bound to loopback only by default; the
container listens on `0.0.0.0:8000` internally.

If any configured ZIM cannot be opened, the server **fails to start** and
names the offending library — there is no partially functional mode.

## Tool schemas

### `search(query: str, limit?: int)`

Searches the full-text index of **every configured library** (bounded
concurrency, one worker per archive), fuses the ranked lists with Reciprocal
Rank Fusion, reranks tied cross-corpus candidates by lexical title coverage,
applies a deterministic diversity pass, and returns clean results. `limit`
defaults to 5; the server enforces a hard maximum
(`SEARCH_MAX_LIMIT`, default 10).

```json
{
  "results": [
    {
      "ref": "corpus://Wikipedia/Bell_test",
      "library": "Wikipedia",
      "kind": "article",
      "title": "Bell test",
      "snapshot": "2026-06",
      "snippet": "To close the detection loophole, an apparatus with a high detection efficiency is needed.",
      "relevant_sections": [
        { "id": "Notable_experiments", "title": "Notable experiments" },
        { "id": "Loopholes", "title": "Loopholes" }
      ]
    }
  ]
}
```

* `ref` — opaque global identifier; pass it back to `fetch()`.
* `library` / `kind` / `snapshot` — provenance: which archive, what kind of
  document, and the corpus snapshot (derived from archive metadata).
* `relevant_sections` — 0–3 deterministic lexical hints (empty when no
  section clearly matches). Section IDs are server-derived; the agent must
  not reconstruct them.

One failing library degrades the search (the others still answer); it never
kills it.

### `fetch(ref: str, sections?: list[str])`

Returns the cleaned document as structured Markdown.

* Without `sections`: the whole document (bounded by `MAX_FETCH_CHARS`;
  `truncated: true` if cut at a section boundary).
* With `sections`: only those sections (subtrees included). Section IDs come
  from `search()` hints or from `available_sections`. The lead/intro section
  has id `""`. For threads, sections are `question`, `accepted-answer`, and
  `answer-<id>`; their `metadata` carries score, acceptance, and tags.

```json
{
  "ref": "corpus://Wikipedia/Bell_test",
  "library": "Wikipedia",
  "kind": "article",
  "title": "Bell test",
  "snapshot": "2026-06",
  "sections": [
    { "id": "Loopholes", "title": "Loopholes", "content": "## Loopholes\n\n..." }
  ],
  "available_sections": [
    { "id": "", "title": "Bell test" },
    { "id": "Background", "title": "Background" },
    { "id": "Loopholes", "title": "Loopholes" }
  ],
  "truncated": false
}
```

Errors are concise and actionable:

```json
{ "error": "invalid_ref", "message": "invalid reference: ..." }
{ "error": "not_found", "message": "Document not found in Wikipedia: Foo_bar" }
{
  "error": "section_not_found",
  "missing_sections": ["Experiments"],
  "available_sections": [ { "id": "Loopholes", "title": "Loopholes" }, "..." ]
}
```

## Example agent workflow

```text
search("Bell experiment loopholes")
    ↓
fetch("corpus://Wikipedia/Bell_test", ["Notable_experiments", "Loopholes"])
```

## Configuration

Environment variables (container defaults shown):

| Variable | Default | Meaning |
| --- | --- | --- |
| `CORPUS_ROOT` | `/corpus` | Corpus root **inside** the container (required) |
| `CORPUS_CONFIG` | `/config/corpus.toml` | Manifest path inside the container (required) |
| `MCP_HOST` | `0.0.0.0` | Listen address inside the container |
| `MCP_PORT` | `8000` | Listen port inside the container |
| `SEARCH_LIMIT` | `5` | Default `limit` for `search()` |
| `SEARCH_MAX_LIMIT` | `10` | Hard maximum for `search(limit=…)` |
| `MAX_FETCH_CHARS` | `100000` | Output budget for fetched content |
| `SEARCH_WORKERS` | `8` | Concurrent archive searches during fan-out |
| `SEARCH_MAX_CONSECUTIVE` | `2` | Diversity pass: max consecutive results from one library |
| `LOG_QUERIES` | `true` | Log search query text (privacy) |
| `ZIM_CHECK` | `false` | Run libzim's full checksum verification at startup (reads the entire corpus: opt-in, slow on large archives) |

Host-side compose variables: `CORPUS_ROOT` (default `./corpus`) and
`CORPUS_CONFIG` (default `./corpus.toml`).

The server fails fast on invalid configuration.

## Tests

```text
make test     # unit + integration + MCP surface tests (needs .venv)
make lint
make format
```

Setup for a local test run:

```text
uv venv .venv --python 3.12
uv pip install -e . --python .venv/bin/python
uv pip install --python .venv/bin/python pytest pytest-asyncio ruff
make test
```

Tests build their own small ZIM fixtures with libzim's writer (one per corpus
family); no corpus is required. The MCP surface regression test asserts that
the server exposes exactly the two tools `search` and `fetch`, and no prompts
or resources.

## Security posture

Local service by design: host binding is loopback-only by default, the corpus
volumes are read-only, the container runs as a non-root user, no privileged
mode, no Docker socket, no arbitrary filesystem access, no URL fetching, no
shell execution. Neither tool accepts filesystem paths, URLs, commands, or
executable content — `ref` is an opaque corpus identifier only.