# Cairn

**A self-hosted MCP server that remembers what it has already told you.**

*A cairn is a stone you leave on a path so you can find your place when you
come back.*

---

## The problem

On a screen you can skim. You glance at a list, see the three things you
already read this morning, and your eye jumps past them in under a second.

In voice you cannot. The channel is serial, it runs at about 150 words a
minute, and you cannot look ahead. So when a voice assistant answers *"what's
new?"* by reading you the top of a feed, you sit through the same items you
heard yesterday, at full length, before you reach anything you did not know.

The scarce resource in a spoken interface is not answers. It is **not being
told the same thing twice**, and **not having to re-establish where you were**.

Cairn is a small MCP server that supplies both, and nothing else.

## What it does

Ask it what's new and it queries several public sources at once, then **removes
everything you have already been told** — in this session, in yesterday's, in
any session. What it reads you is what is actually new.

Then it stays useful afterwards. A briefing is numbered, and the numbers keep
working:

> — *"What's new on protein folding?"*
> — *"Three new things. First: …  Second: …  Third: …"*
>
> …two days later, a different conversation…
>
> — *"Tell me more about the second one."*
> — *"The second one. …"*

That follow-up is the part that does not exist today. An MCP session begins at
`initialize` and dies with the connection. A person's memory of what they were
told does not. Conflating the two is exactly why assistants repeat themselves,
and Cairn's whole design is the refusal to conflate them: **state is keyed to
the user, never to the session.**

## What it is measured to do

Every number below is produced by a script in this repository, on real data.
Re-run them and you will get your own.

### It stops repeats — 57.3% → 0%

`scripts/measure_impact.py` replays the **last five calendar days** of a real,
timestamped arXiv corpus across three topics, one "what's new" per day. The day
boundaries and the feed's turnover both come from `Item.published`; nothing
about the arrival rate is assumed. **Both arms read the same feed, to the same
depth, and may speak at most five items per session** — that symmetry is
enforced in the script, not asserted here.

| | stateless assistant | Cairn |
|---|---|---|
| spoken slots used | 75 | 32 |
| **repeats** | **43 (57.3%)** | **0 (0%)** |
| **distinct things heard** | **32** | **32** |
| distinct per spoken slot | 0.43 | 1.0 |
| speech spent on repeats | 202 s | 0 s |

The last row is timed by the product's own renderer: `cairn.speech.spoken_seconds`
over the line `render_brief` actually speaks for each repeated item, at its own
150 words per minute. It used to be a hand-rolled word count over titles, in a
script that imported `spoken_seconds` and never called it.

**The claim is the third and fourth rows together: the same thirty-two things,
heard in thirty-two slots instead of seventy-five.** Cairn does not find the
user more; it stops spending their time on what they already have. The
baseline always fills its five slots, because a repeat still costs a slot.
Cairn stops when the feed holds nothing it has not already said.

Two earlier versions of this table were wrong, and both corrections are worth
stating because the second one is the more interesting failure. The first
printed "75 | 75" and called it the same speaking budget; it was not. The
second fixed that and then quietly gave Cairn a **fifteen**-item feed while the
baseline read the top **five** of it, which made the "distinct things heard"
row read 32 against 55 — a gap produced by the feed depth, not by
deduplication. At equal depth the two arms deliver exactly the same thirty-two
items, and that is what the table now reports.

Cairn *can* legitimately read deeper than a stateless assistant, precisely
because it can skip what it has already said, and that is a real advantage —
at a fifteen-deep feed it delivers 55 distinct items to the baseline's 32. But
that is a different claim from deduplication, so it is measured as a labelled
variant by `scripts/sensitivity.py` (**F**) rather than folded into the
headline.

The measured arrival rates over the replay differ by more than an order of
magnitude between topics — 3.2, 1.0 and 0.2 items a day — and that is the
mechanism: a recency-ranked top five on a slow topic is mostly yesterday's top
five. The script reports the rate it measured for each topic rather than
assuming one.

**What this figure does and does not depend on.** It is a function of the
corpus's *timestamps*, not its *text*: swap the titles for random strings and
keep the dates, and you get 57.3% again, because a
repeat is the same item resurfacing. Strip the dates and the measurement
collapses to zero rather than inventing a rate; compress every arrival into one
day and it goes to 0%; spread arrivals one per day and it rises to 64%.
`scripts/sensitivity.py` runs all four variants, and
`tests/test_claims.py` keeps them as regression tests. This matters because
the previous implementation did **not** pass that test: it hardcoded two
arrivals per day, never read `Item.published`, and reproduced its headline
48% exactly on a corpus of 180 random strings. The corpus contributes its
timestamps to this figure and its text to the partial-recall and wasted-speech
figures below.

**What it also depends on: the shape of the session.** The 57.3% is measured
at five days of five spoken items. That is a choice, and it moves the number:
across sessions in {3, 5, 7, 10, 14} and items-per-session in {3, 5, 8} the
baseline repeat rate ranges from **37.0% to 77.1%**, rising with both. The
direction never changes — the baseline repeats substantially in all fifteen
cells and Cairn is at 0% in all fifteen — but the point estimate is one cell of
that grid and the whole grid is printed by `scripts/sensitivity.py` (**G**).
Those two constants used to be undocumented and unswept, which meant a reader
told "sensitivity.py runs all the variants" would have believed the headline
was stress-tested on every axis when it was stress-tested on one.

### Follow-ups survive the session — 100% by number

Deliver twelve briefings to a **file-backed** store, close it, and resolve the
references from a **separate operating-system process** that opens the file for
the first time. The parent keeps no handle; the process ids are printed and
recorded. `tests/test_transport.py` does the same thing one level up, across a
real `initialize` / `DELETE` / `initialize` boundary over HTTP.

(An earlier version of this script claimed to "close that store's session and
open a new one" while holding a single `Store(":memory:")` object throughout —
an in-memory SQLite database cannot be reopened at all. It measured a live
object answering its own writes. That is fixed, and the numbers below are from
the cross-process run.)

| resolution method | result |
|---|---|
| by ordinal — *"the second one"* | **60 / 60 (100%)** |
| by description, verbatim words | 25 / 25 (100%) — *an upper bound, not a finding* |
| by description, partial recall | **20 / 25 (80%)** |
| a stateless server | **0%, by construction** — with no referent frame, an ordinal from an earlier session indexes into nothing |

The two description rows are separated on purpose. Handing the matcher three
words copied straight out of the title scores 100%, and that number means
nothing: it is the mechanism being fed its own input. The row worth reading is
the second one — two words from the *second half* of the title, modelling a
listener who has lost the distinctive opening, which is exactly what a spoken
channel takes from you. That scores **80%**, and it is reported at 80%.

### It is on the protocol revision it claims

`scripts/conformance.py` drives a running server over real HTTP — no SDK, no
mocks — and validates every response against `spec/schema-2025-11-25.json`, the
unmodified schema published by the MCP project (vendored with attribution in
`spec/NOTICE`).

```
$ python3 scripts/conformance.py --spawn
...
40 passed, 0 failed, 40 checks
```

Those 40 include the things that are easy to claim and awkward to prove:
`Origin` rejection with 403, session `404` after `DELETE`, `400` on an
unsupported `MCP-Protocol-Version`, batch rejection, and the whole
2025-11-25 **task** surface — `tasks/get`, `tasks/list`, `tasks/result`,
`tasks/cancel`, the `-32602` on cancelling a terminal task, and the
`io.modelcontextprotocol/related-task` metadata on a task result.

## Why tasks, and why they are the honest version-check

A five-source fan-out takes seconds. A voice turn cannot block on it. With
task augmentation, `brief` returns a task handle immediately — the assistant
can say *"hold on, checking five sources"* — and the result is collected when
it is ready.

Tasks are also the reason the version claim is checkable. Any server can put
`"2025-11-25"` in a string. Tasks **did not exist in 2025-06-18**, so a working
`tasks/result` is evidence the revision is real rather than asserted.

## Reproducing all of it, offline

Every figure above comes from a script in this repository, and the inputs those
scripts read are committed.

```bash
python3 -m pytest -q                                     # 187 tests
python3 scripts/conformance.py --spawn                   # 40 checks over real HTTP
python3 scripts/measure_impact.py --numbers data/measured.json
python3 scripts/check_readme_numbers.py                  # this README vs that file
python3 scripts/sensitivity.py                           # try to break the headline number
python3 scripts/mutants.py                               # 17 deliberate defects, all caught
```

`data/cassette.json` holds recorded responses for three topics, so the fan-out
and the briefing pipeline run **with no network at all**. Every source that
issues a request for a topic is recorded for it; USGS appears once, for the
hazards topic, because for the other two it declines to ask (see the source
table below), and `tests/test_replay.py` asserts that each of the five
delivers items for at least one recorded topic:

```bash
CAIRN_HTTP_MODE=replay CAIRN_CASSETTE=data/cassette.json python3 -m cairn
python3 scripts/demo.py                                  # the whole demo, offline, ~50s
python3 scripts/record_cassette.py                       # re-record it (needs network)
```

`tests/test_replay.py` runs that path with `socket.socket` monkeypatched to
raise, so a live request escaping during replay fails the suite rather than
quietly returning different items.

### What the tests do and do not cover

`pytest` reports **79% statement coverage** of `cairn/`. The largest single
block of the gap is `cairn/transport.py` at **41%** of the missed statements —
the SSE streaming and task-notification paths, driven by
`scripts/conformance.py` over a real socket (`C4.1`–`C4.14`) rather than by
`pytest`. The second is `cairn/__main__.py`, which `pytest` never imports at
all and which is exercised only by `conformance.py --spawn`. An earlier
revision of this paragraph said the gap was "mostly" transport; that overstated
its share by about a factor of two and did not mention the CLI entry point.
That split is stated because a single coverage percentage would hide it.

Coverage is a weak signal, so the suite is also checked by mutation, and the
campaign is in the repository rather than in a report:

```
$ python3 scripts/mutants.py
baseline (unmutated copy): GREEN
  M1   KILLED   the DNS-rebinding origin check is deleted
  ...
17/17 killed by pytest alone
```

**Seventeen** deliberate defects — the origin check deleted, `DELETE` made a
no-op, session ids replaced by a counter, `PROTOCOL_VERSION` downgraded, the
arXiv namespace broken, the OpenAlex abstract reconstruction reversed, the
argument validator removed, the mute guard weakened, the USGS topic gate
removed, the body-size guard removed, the date-driven replay reverted, and one
for each of the six defects the second audit found — are **all** caught by
`pytest` alone, each by a named test. Six of the first eleven previously
survived; four survived both suites.

Each mutant is applied to a copy under `/tmp`, never to the working tree, and
the unmutated copy is run first and has to be green. That guard is not
decoration: without it a bad command line makes every mutant look killed and
reports a perfect kill rate for a suite that killed nothing.

## The dedup mechanism, stated accurately

Two layers, deliberately not presented as equals.

**Layer 1 — canonical fingerprint. This is the guarantee.** An exact key built
from a stable identifier where one exists (arXiv id, DOI, OpenAlex id, HN item
id, Wikipedia page id), with the decorations stripped, so the *same paper*
reached through arXiv and through OpenAlex collapses to one key. No threshold,
no tuning. This handles the dominant real case: an item resurfacing on a later
day.

**Layer 2 — lexical near-duplicate. This is a conservative assist.** IDF-weighted
containment over title and summary, with the IDF built from *this user's own*
history. It is tuned for precision, not recall, because the two errors are not
symmetric: a missed rewording costs a mild repeat, while a false positive means
the user is **never told something new at all** and has no way to find out.

Measured on the labelled corpus in `tests/test_novelty.py`: **zero false
positives, and a minority of rewordings caught.** Short titles do not carry
enough signal for a lexical method to do better, and an earlier simhash
implementation was cut from this repository after measurement showed its
same-claim and different-claim distances overlapped. Do not read this layer as
paraphrase detection. It is not.

## Sources

Five, all **public and credential-free** — a hard constraint, so that anyone
who clones this repo can run it with no key to obtain and no account to create.

| source | what it is for |
|---|---|
| arXiv | new preprints |
| OpenAlex | the same literature, indexed differently — included *deliberately* as an overlap, because it is what proves Layer 1 earns its keep |
| Hacker News | the practitioner counterpart. It queries Hacker News's own search index (`hn.algolia.com`, public, no key) newest-first. It used to filter the ~30-story front page by topic word, which an audit measured at **zero items for all three recorded topics** — a source that is queried, succeeds and returns nothing advertises breadth the user never gets. |
| USGS | a fast-moving feed with hard identifiers, where a repeat is actively harmful. It is a **global** feed with no topic parameter, so unlike the others it decides for itself whether the question is one it can answer. It answers only when the topic carries a geoscience word outright (`seismic`, `tsunami`, `aftershock`) or an ambiguous one in a geoscience collocation (`fault line`, `natural disaster`); a bare ambiguous word is refused, so `fault tolerance`, `magnitude estimation in astronomy` and `hazard ratio survival analysis` get nothing from it. Both of those were real defects, found by audit and fixed in that order: first it accepted the topic and ignored it, putting six off-topic earthquakes in every briefing on every subject; then its keyword list treated `fault` and `magnitude` as sufficient. |
| Wikipedia | background for an unfamiliar topic |

Adding one is a small, safe change: adapters only fetch and parse, and every
judgement about what the user hears happens above them.

A briefing **says which sources failed**. Depending on five public services
means one of them will be down, and presenting a thinner answer as though it
were complete would be the wrong kind of quiet.

## Running it

Python 3.11 or newer. No runtime dependencies.

```bash
git clone https://github.com/jianwang-ntu/cairn-mcp
cd cairn-mcp
python3 -m cairn                      # http://127.0.0.1:8765/mcp
```

Point any MCP client at `http://127.0.0.1:8765/mcp` over Streamable HTTP.

Or watch it do the thing it is for, with no network and no client to install:

```bash
python3 scripts/demo.py
```

That spawns the server against the committed cassette and drives it over real
HTTP — `initialize`, `tools/list`, a briefing, the same question again (three
items suppressed), `DELETE`, a **new** `initialize`, and then *"tell me more
about the second one"* answered from a session that no longer exists. It is
the script the demonstration video records.

A friction log from building this against the MCP 2025-11-25 spec is in
[`FRICTION_LOG.md`](FRICTION_LOG.md).

```bash
python3 -m cairn --help               # host, port, store path, weather opener
python3 -m cairn --weather 1.29,103.85,Asia/Singapore
```

Development:

```bash
pip install -e '.[dev]'
python3 -m pytest -q                  # 187 tests, no network needed
python3 scripts/conformance.py --spawn
python3 scripts/measure_impact.py --numbers data/measured.json
python3 scripts/check_readme_numbers.py
python3 scripts/sensitivity.py
python3 scripts/mutants.py
python3 scripts/measure_impact.py --record   # refresh the corpus (needs network)
python3 scripts/record_cassette.py           # refresh the cassette (needs network)
```

## The tools

| tool | the utterance it is for |
|---|---|
| `brief` | *"what's new"*, *"catch me up"* |
| `follow_up` | *"tell me more about the second one"* — works across sessions |
| `track` | *"follow X"*, *"never mention Y again"* |
| `mark_heard` | *"I already knew that"* |
| `place_marker` | *"remember where I am"* / *"where was I?"* |
| `recap` | *"what have you told me"* |

Every tool returns a `spoken` string alongside its structured payload: URLs
stripped, `et al.` expanded, length capped, ordinal spoken first so the user
has a short handle to reply with.

Three MCP resources expose the durable state directly — `cairn://profile`,
`cairn://ledger/recent`, `cairn://markers` — because a memory you cannot
inspect is one you cannot trust.

## Security

The transport spec's requirements are implemented, not described:

- the `Origin` header is validated, and a present-and-invalid one gets **403**
  (DNS-rebinding defence);
- the listener binds **127.0.0.1** by default;
- session ids come from `secrets.token_urlsafe`, and are visible ASCII only.

Cairn holds a record of what one person has been read. Treat the SQLite file
as personal data. There is no authentication layer yet: run it on loopback, or
put one in front of it. That is a real limitation and it is stated here rather
than left to be discovered.

## What it does not do

- No paraphrase detection worth the name — see Layer 2 above.
- No authentication.
- Description-based follow-up resolves 80% of the time on partial recall, not 100% — the figure in the table above, not a different one. (This line said 60% for two commits while the table said 80%. The README now quotes `data/measured.json`, which `scripts/measure_impact.py` writes, and `scripts/check_readme_numbers.py` reads the tables above **positionally** — row label, then cell, then the numbers in that cell in order — so corrupting any checked figure fails it. The first version of that checker tested only that each figure appeared *somewhere* in the document, which an audit showed could not fail on a wrong number: 57.3% → 57.4% passed, and a spoken-slot count of 55 → 5 passed. `tests/test_claims.py` now corrupts each checked cell in a copy of this file and asserts the checker rejects it.)
- Weather is exempt from the ledger, because it changes. That exemption is
  narrow and deliberate, and it is the only one.

## Licence

Apache-2.0. See `LICENSE`. The vendored MCP schema keeps its own attribution
in `spec/NOTICE`.