corpus-mcp

An MCP server that gives an agent keyword search over a directory of documents. Point it at a folder and it works — no model download, no API key, no GPU, no vector database running alongside it. One dependency: the MCP SDK.

pip install -e .
corpus-mcp --root ./docs serve

The interesting part is not the retrieval. It is the tool design: what an agent can actually do with a search tool, and what makes one usable rather than a context-window bonfire.

Try it in ten seconds

$ make demo
1. reference/glossary.md  (score 1.973, f700ededcfdd:0)
   # Glossary

   **Extraction** — the process of dissolving soluble compounds out of ground
   coffee. Under-extraction tastes sour and thin; over-extraction tastes bitter …

2. guides/brewing.md  (score 1.774, 71c6f092dbcb:0)
   # Pour-over brewing
   …

That query was "why does my coffee taste sour". The document says tastes, the query said taste, and the glossary entry that actually answers it ranks first. Both of those are deliberate; see below.

The tools

Documents are also exposed as MCP resources at corpus://<relative-path>.

Design decisions worth arguing with

Search and fetch are separate tools. One search returning full chunks is simpler to write and much worse to use: ten results at 1,200 characters each is most of a context window spent before the agent has decided which one it wants. So search returns snippets — enough to triage — and fetch widens a chosen result on demand. The agent pays for detail only where it decided detail was worth having.

Snippets are centred on the match, not the top of the chunk. Returning the first N characters fails constantly, because the matching sentence is usually in the middle: the agent sees an unrelated preamble and either discards a good hit or fetches everything to find out. The snippet window is chosen to cover as many query-term occurrences as possible.

Every limit is clamped server-side. Tool output lands directly in a context window, so an unbounded tool is a denial-of-service on the thing calling it. A caller asking for 10,000 results is exactly the case the cap exists for, so limits are enforced rather than trusted. When output is truncated the response says so, so the agent can narrow its query instead of assuming it saw everything.

Empty results explain themselves. A bare empty list is a dead end. The response reports how many chunks and documents exist, which distinguishes "your query missed" from "nothing is indexed" — two situations with different next moves.

Stale identifiers are an expected outcome, not an error. Chunk ids change when a document is edited, so an id from earlier in a long session can go bad. fetch says exactly that and tells the agent to search again.

Overlap is stripped when chunks are joined. Chunks overlap so no passage is split across a boundary, but handing that overlap back means the agent reads the same sentences twice and may read the repetition as emphasis. Chunks carry absolute offsets, so the overlap is removed by position rather than by string matching.

BM25, not embeddings. For the keyword-ish queries an agent issues while navigating a corpus it already knows something about, lexical retrieval is strong, and it has the property that matters most in an agent loop: fast, and it never silently costs money. Semantic search is a worthwhile addition, not a precondition for the thing being useful.

Light stemming, not a real stemmer. Plurals and common verb endings are folded so tastes matches taste. A full Porter implementation is a hundred lines and a maintenance surface, and its long tail (operational → oper) is as likely to hurt as help on short queries. Indexing and querying share one tokeniser, since any divergence between them silently costs recall.

Security

The server is pointed at a root directory and never reads outside it. This matters more than it might seem: tool arguments come from model output, so a document identifier is untrusted input, and ../../.ssh/id_rsa is a thing a confused or adversarial agent will eventually ask for.

Every path crossing the boundary goes through one containment check that resolves symlinks before comparing — a symlink inside the root pointing outside it defeats a prefix check done on the unresolved path. Absolute-looking arguments are interpreted relative to the root rather than as real absolute paths. Resource URIs get the same treatment as tool arguments.

Non-UTF-8 files, oversized files, and vendor directories (.git, node_modules, …) are skipped rather than indexed as noise.

Connecting it to a client

Claude Desktop, or any MCP host, launches the server as a subprocess:

{
  "mcpServers": {
    "my-docs": {
      "command": "corpus-mcp",
      "args": ["--root", "/absolute/path/to/docs", "serve"]
    }
  }
}

The corpus is re-read when it changes on disk, so files edited during a session become searchable without a restart — reindexing is incremental on modification time rather than rebuilding on every call.

Development

make install   # server plus dev tools
make demo      # one query against the example corpus
make test      # 89 tests, no network required
make smoke     # launch the installed server as a subprocess and exercise it
make lint

Two layers of testing, because they catch different failures:

• tests/test_server.py drives a real MCP client against a real server in-process. What is exercised is wire behaviour — tool schemas, structured results, error shapes — not the Python functions underneath. A server whose functions are correct but whose tool surface is wrong is still broken, and only this level catches that.

• scripts/stdio_smoke.py launches the installed console script as a subprocess and talks JSON-RPC to it over stdio, the way a host does. That covers packaging, the entry point, and the transport — including the classic failure where something writes to stdout and corrupts the protocol stream.

Limitations

• Lexical retrieval only. A query sharing no vocabulary with the document will not find it. Adding an embedding backend behind the same tool surface is the obvious next step.

• Text formats only — .md, .txt, .rst, .csv, .json, .yaml and friends. No PDF or DOCX extraction.

• The whole index lives in memory and is rebuilt in full when the corpus changes. Fine for the thousands-of-documents case this is built for; a corpus in the millions wants a real index that updates per file.

• English only. The stopword list and the suffix folding both assume it.

• No access control beyond the root. Every file under the root is visible to anything the server is connected to.

License

MIT. Built by Aion Innovations.