# rookhavenretrieval

An MCP (Model Context Protocol) server that does retrieval for Ilkeston Holdings
(Project Rookhaven). Given a query and a corpus of documents, it breaks the query
into subqueries, gives each subquery a slice of the total recall budget, retrieves
against the corpus, and merges the picks into one ranked list.

## The problem it solves

A single dense query ("crane faults and berth delays and tide windows") mixes
several information needs. Ranking the whole corpus against it once lets the
strongest need crowd out the others. This server decomposes the query and spends
a separate recall budget per subquery, so every need gets guaranteed
representation in the result before the lists are merged.

The hard constraint is auditability under reproducibility: **every result must be
bit-reproducible from a recorded seed.** Port operations decisions get reviewed
after the fact, so a stored `(seed, query, corpus)` must reproduce the exact
ranking that was acted on — byte for byte.

## How reproducibility is enforced

- No `Math.random` anywhere on the retrieval path. Every tie-break uses a seeded
  generator (`src/rng.ts`: mulberry32 with cyrb128 string-to-seed hashing).
- Allocation and merge use two RNG streams seeded distinctly from
  `seed + query`, so budget ties and ranking ties never correlate.
- Float scores are rounded to 9 decimal places in the scorer, erasing
  last-bit noise that would otherwise make two runs differ.
- Every result carries an `audit` record: the seed, the decomposition, the
  per-subquery budgets, and the docs each subquery contributed. The test suite
  asserts `JSON.stringify(runA) === JSON.stringify(runB)`.

## Architecture

```
src/
  types.ts          domain types (Document, RetrievalRequest, RetrievalResult, ...)
  rng.ts            seeded PRNG (mulberry32 + cyrb128)
  tokenize.ts       shared tokenization + stop-word filtering
  decompose.ts      query -> subqueries, largest-remainder budget allocation
  retrieve.ts       core algorithm: score, select per budget, merge, cap
  providers/
    base.ts         ScoringProvider interface (deterministic scoring contract)
    stub.ts         offline cosine scorer used by the whole test suite
  config.ts         env-driven config with validation and resource limits
  corpus.ts         corpus file loading + validation (bad JSON, missing file...)
  logging.ts        structured JSON logging to stderr
  server.ts         pure JSON-RPC / MCP dispatcher (initialize, tools/*)
  main.ts           stdio transport wiring + startup
```

Scoring runs behind `ScoringProvider` (`src/providers/base.ts`) so the retrieval
core never depends on a concrete model. The only implementation shipped is
`StubProvider` — a deterministic cosine scorer over term frequencies that needs
no network. The interface is intentionally synchronous (see
`docs/adr/0003-synchronous-provider-interface.md`); a network-backed provider is
not yet implemented, and the tradeoff that blocks it is recorded there.

## Install

```sh
npm ci
```

Requires Node 22+, which runs TypeScript directly — there is no build step and no
`dist/`. `npm ci` uses the committed `package-lock.json`.

## Quickstart

Use it as a library:

```ts
import { retrieve, StubProvider } from './src/index.ts';

const corpus = [
  { id: 'd1', text: 'crane maintenance schedule for berth 7' },
  { id: 'd2', text: 'berth delays caused by tide windows' },
];

const result = retrieve(
  { query: 'crane faults and tide windows', recallBudget: 4, seed: 'run-1' },
  corpus,
  new StubProvider(),
);

console.log(result.hits);        // ranked ScoredHit[]
console.log(result.audit.seed);  // 'run-1' — replay this to reproduce
```

Or run the MCP server over stdio (newline-delimited JSON-RPC 2.0). stdout carries
protocol traffic only; all logs are JSON lines on stderr:

```sh
printf '[{"id":"d1","text":"crane maintenance for berth 7"},{"id":"d2","text":"tide windows"}]' > corpus.json
printf '%s\n%s\n' \
  '{"jsonrpc":"2.0","id":1,"method":"initialize"}' \
  '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"retrieve","arguments":{"query":"crane and tide","recallBudget":2,"seed":"demo"}}}' \
  | ROOKHAVEN_CORPUS_PATH=./corpus.json node src/main.ts
```

The `retrieve` tool takes `query` (required), `recallBudget` (optional, defaults
to config) and `seed` (optional, defaults to config). The response's
`structuredContent` is the full `RetrievalResult`, so a run replays from
`audit.seed`.

## Algorithm

1. `decompose` splits the query on connectors (`and`, `then`, commas, ...) into
   subqueries, weights each by its distinct content-term count, and distributes
   the recall budget by the largest-remainder method. Fractional-share ties are
   broken by the seeded RNG.
2. For each subquery, `retrieve` scores every document via the provider, sorts
   with a seeded comparator (score desc, then a seeded shuffle key, then id),
   and takes up to that subquery's budget.
3. Picks are merged, deduplicated to the strongest score per document, and capped
   at the total recall budget.

## Configuration (environment variables)

| Variable | Default | Meaning |
| --- | --- | --- |
| `ROOKHAVEN_CORPUS_PATH` | (none) | Path to the corpus JSON file |
| `ROOKHAVEN_DEFAULT_RECALL` | `10` | Recall budget when a request omits one |
| `ROOKHAVEN_MAX_RECALL` | `1000` | Upper bound on any request's recall budget |
| `ROOKHAVEN_MAX_QUERY_LEN` | `4096` | Reject queries longer than this |
| `ROOKHAVEN_MAX_CORPUS_DOCS` | `100000` | Reject corpora larger than this |
| `ROOKHAVEN_DEFAULT_SEED` | `rookhaven-default` | Seed used when a request omits one |
| `ROOKHAVEN_LOG_LEVEL` | `info` | `debug` \| `info` \| `warn` \| `error` |

The corpus file is a JSON array of `{ "id": string, "text": string }`. With no
`ROOKHAVEN_CORPUS_PATH` the server starts with an empty corpus and logs a warning
(every query then returns zero hits). Bad config or an unreadable corpus is a
fatal startup error: the process logs the reason to stderr and exits non-zero.

## Commands

```sh
npm ci             # install from the committed lockfile
npm test           # run the node:test suite
npm run typecheck  # tsc --noEmit
```

## Known limitations

- **Decomposition is lexical, not semantic.** It splits on explicit connectors
  and punctuation; it will not break "options for reducing quayside idle time"
  into distinct needs, because there is no connector. See ADR 0002.
- **Only the stub scorer exists.** `StubProvider` is term-frequency cosine — it
  has no synonymy or spelling tolerance. A real model provider is blocked on the
  synchronous interface decision in ADR 0003.
- **The whole corpus is scored per subquery** (linear scan, no index). Fine for
  the low-thousands of documents this is sized for; `ROOKHAVEN_MAX_CORPUS_DOCS`
  guards against accidentally loading something far larger.
- **The corpus is loaded fully into memory at startup.** There is no incremental
  or streaming ingest.

## Architecture decision records

See `docs/adr/` for the contested calls: the custom seeded PRNG (0001), lexical
vs model-based decomposition (0002), the synchronous provider interface (0003),
and the stdio transport framing (0004).

---

*Ilkeston Holdings is an illustrative client; this repository is a self-directed reference implementation built to work end to end.*