rookhavenretrieval

An MCP (Model Context Protocol) server that does retrieval for Ilkeston Holdings (Project Rookhaven). Given a query and a corpus of documents, it breaks the query into subqueries, gives each subquery a slice of the total recall budget, retrieves against the corpus, and merges the picks into one ranked list.

The problem it solves

A single dense query ("crane faults and berth delays and tide windows") mixes several information needs. Ranking the whole corpus against it once lets the strongest need crowd out the others. This server decomposes the query and spends a separate recall budget per subquery, so every need gets guaranteed representation in the result before the lists are merged.

The hard constraint is auditability under reproducibility: every result must be bit-reproducible from a recorded seed. Port operations decisions get reviewed after the fact, so a stored (seed, query, corpus) must reproduce the exact ranking that was acted on — byte for byte.

How reproducibility is enforced

• No Math.random anywhere on the retrieval path. Every tie-break uses a seeded generator (src/rng.ts: mulberry32 with cyrb128 string-to-seed hashing).

• Allocation and merge use two RNG streams seeded distinctly from seed + query, so budget ties and ranking ties never correlate.

• Float scores are rounded to 9 decimal places in the scorer, erasing last-bit noise that would otherwise make two runs differ.

• Every result carries an audit record: the seed, the decomposition, the per-subquery budgets, and the docs each subquery contributed. The test suite asserts JSON.stringify(runA) === JSON.stringify(runB).

Architecture

src/
  types.ts          domain types (Document, RetrievalRequest, RetrievalResult, ...)
  rng.ts            seeded PRNG (mulberry32 + cyrb128)
  tokenize.ts       shared tokenization + stop-word filtering
  decompose.ts      query -> subqueries, largest-remainder budget allocation
  retrieve.ts       core algorithm: score, select per budget, merge, cap
  providers/
    base.ts         ScoringProvider interface (deterministic scoring contract)
    stub.ts         offline cosine scorer used by the whole test suite
  config.ts         env-driven config with validation and resource limits
  corpus.ts         corpus file loading + validation (bad JSON, missing file...)
  logging.ts        structured JSON logging to stderr
  server.ts         pure JSON-RPC / MCP dispatcher (initialize, tools/*)
  main.ts           stdio transport wiring + startup

Scoring runs behind ScoringProvider (src/providers/base.ts) so the retrieval core never depends on a concrete model. The only implementation shipped is StubProvider — a deterministic cosine scorer over term frequencies that needs no network. The interface is intentionally synchronous (see docs/adr/0003-synchronous-provider-interface.md); a network-backed provider is not yet implemented, and the tradeoff that blocks it is recorded there.

Install

npm ci

Requires Node 22+, which runs TypeScript directly — there is no build step and no dist/. npm ci uses the committed package-lock.json.

Quickstart

Use it as a library:

import { retrieve, StubProvider } from './src/index.ts';

const corpus = [
  { id: 'd1', text: 'crane maintenance schedule for berth 7' },
  { id: 'd2', text: 'berth delays caused by tide windows' },
];

const result = retrieve(
  { query: 'crane faults and tide windows', recallBudget: 4, seed: 'run-1' },
  corpus,
  new StubProvider(),
);

console.log(result.hits);        // ranked ScoredHit[]
console.log(result.audit.seed);  // 'run-1' — replay this to reproduce

Or run the MCP server over stdio (newline-delimited JSON-RPC 2.0). stdout carries protocol traffic only; all logs are JSON lines on stderr:

printf '[{"id":"d1","text":"crane maintenance for berth 7"},{"id":"d2","text":"tide windows"}]' > corpus.json
printf '%s\n%s\n' \
  '{"jsonrpc":"2.0","id":1,"method":"initialize"}' \
  '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"retrieve","arguments":{"query":"crane and tide","recallBudget":2,"seed":"demo"}}}' \
  | ROOKHAVEN_CORPUS_PATH=./corpus.json node src/main.ts

The retrieve tool takes query (required), recallBudget (optional, defaults to config) and seed (optional, defaults to config). The response's structuredContent is the full RetrievalResult, so a run replays from audit.seed.

Algorithm

1. decompose splits the query on connectors (and, then, commas, ...) into subqueries, weights each by its distinct content-term count, and distributes the recall budget by the largest-remainder method. Fractional-share ties are broken by the seeded RNG.

2. For each subquery, retrieve scores every document via the provider, sorts with a seeded comparator (score desc, then a seeded shuffle key, then id), and takes up to that subquery's budget.

3. Picks are merged, deduplicated to the strongest score per document, and capped at the total recall budget.

Configuration (environment variables)

The corpus file is a JSON array of { "id": string, "text": string }. With no ROOKHAVEN_CORPUS_PATH the server starts with an empty corpus and logs a warning (every query then returns zero hits). Bad config or an unreadable corpus is a fatal startup error: the process logs the reason to stderr and exits non-zero.

Commands

npm ci             # install from the committed lockfile
npm test           # run the node:test suite
npm run typecheck  # tsc --noEmit

Known limitations

• Decomposition is lexical, not semantic. It splits on explicit connectors and punctuation; it will not break "options for reducing quayside idle time" into distinct needs, because there is no connector. See ADR 0002.

• Only the stub scorer exists. StubProvider is term-frequency cosine — it has no synonymy or spelling tolerance. A real model provider is blocked on the synchronous interface decision in ADR 0003.

• The whole corpus is scored per subquery (linear scan, no index). Fine for the low-thousands of documents this is sized for; ROOKHAVEN_MAX_CORPUS_DOCS guards against accidentally loading something far larger.

• The corpus is loaded fully into memory at startup. There is no incremental or streaming ingest.

Architecture decision records

See docs/adr/ for the contested calls: the custom seeded PRNG (0001), lexical vs model-based decomposition (0002), the synchronous provider interface (0003), and the stdio transport framing (0004).

Ilkeston Holdings is an illustrative client; this repository is a self-directed reference implementation built to work end to end.