merlin

The memory and search layer for a files-canonical Claude Code working partner. The partner keeps its record as plain markdown in git: a daily journal of what was actually seen, host records, runbooks, short lessons and plans. merlin indexes that record for hybrid search (semantic plus full-text) and host history, and serves both to Claude Code as a read-only MCP server.

merlin grew out of the memory engine of Anna, a personal assistant project.

Principle: the files are canonical, and the index is derived. Every row can be rebuilt from git at any time. Nothing is ever written only to the database: no notes, no edits, no state you would miss if you dropped it.

How it works

 git repo (markdown)        ingest (timer)                              Postgres 17 + pgvector
 journal/ hosts/      ──►  pull --ff-only → walk HEAD → skip unchanged  ──►  documents (path, type, date, blob sha)
 runbooks/ memory/          blobs → redact secrets → chunk by ##/###          chunks (heading, body, hosts[], date,
 planning/ skills/          → embed (reusing vectors by text sha)                     vector, tsvector)
                                                                                   │
 Claude Code ◄── stdio (over ssh) ── mcp_server: search · host_history ◄───────────┘
                                     get_document · list_documents · index_status

• Chunking (chunk.py). One chunk per ##/### section, never split inside a code fence. A section over about 1,500 tokens is split at paragraph boundaries. Each chunk's embedded text starts with a context header, [type] path > H1 > H2 > H3, so a section called "Verify" still says which runbook it belongs to.

  • Hosts: the host: front matter field, plus any mention of a known host, alias or IP. The known hosts come from the front matter of the host records.

  • Dates: the filename date for the journal, else a front matter date/updated/created, else the file's last commit date. A section takes the date from its own heading when it has one, such as ## 2030-01-16 (session 2) or ### Disk full (2030-01-15).

• Ingest (ingest.py) runs one transaction per document.

  • Documents whose blob sha is unchanged are skipped. Changed ones are re-chunked, and embeddings are reused by the sha256 of the exact embedded text, so editing one paragraph re-embeds only that chunk. Removed documents are deleted.

  • Secrets are redacted before anything is chunked or stored, and again when a document is served: tokens (including Telegram bot tokens), sk- keys, hex strings of 32+ characters, JWTs, private-key blocks, key=value secrets and credentials in URLs. See Redaction.

  • Problems that would silently hurt retrieval are printed to stderr as warning: <path>: … (see Ingest warnings). They never stop the run.

  • The run records head_sha, last_ingest_at and the counts in meta.

• Search (search.py) fuses a vector top-N (HNSW cosine) with Postgres full-text search (websearch_to_tsquery + ts_rank_cd, headings weighted above bodies) by reciprocal-rank fusion (k = 60).

  • Filters: type, host (an alias works too), and since/until, matched against the section's date.

  • The fused scores are then reranked by doc type, recency and stale markers, and grouped to one section per document, naming the document's other matching sections. See Ranking.

  • Filtered vector scans use hnsw.iterative_scan = relaxed_order with a raised ef_search, so a filter can't starve the result.

  • exact=true drops the embedding and fuses a literal ILIKE with full-text search. Use it for error strings, paths and IDs.

• Host history returns one host's journal sections and host-record notes newest first, by date rather than by relevance. Use it to read "what have we already done or hit on this host" before running a procedure there. With topic=, it keeps the best hybrid matches within that host and still returns them in date order. Undated host-record sections carry the file's commit date, so the current record sorts above the evidence.

Related MCP server: repository-memory

Doc types and how to weigh them

The shipped layout maps paths to types: journal/** is journal, hosts/*.md host, runbooks/** runbook, memory/*.md memory (memory/MEMORY.md is memory-index), planning/** planning, skills/** skill, and top-level *.md doc. _template.md and non-markdown files are skipped. The MCP tool descriptions tell the model how to weigh each type:

The tool descriptions also say that an empty result is not proof of absence.

To use a different layout, copy merlin.example.toml to merlin.toml, or point MERLIN_CONFIG at your copy. It holds the glob → type rules (first match wins), the per-type weighting text, which types define hosts, which types take their date from the filename, and which types feed host_history.

Several repos as one corpus

A corpus can span several git repos (for example one repo per layer). List them in the layout file:

[[repos]]
name = "notes"
dir = "/srv/merlin/git/notes.git"     # a bare mirror or a checkout
[[repos]]
name = "infra"
dir = "/srv/merlin/git/infra.git"

Each file is then indexed under the virtual path <name>/<path> (infra/hosts/db1.md). That is what the layer globs match ({ glob = "infra/hosts/*.md", type = "host" }), what search returns, and what get_document takes (or repo= plus the path inside it). search gains repos=[…], list_documents gains repo=, and index_status reports each repo's indexed and current HEAD. Host records in any repo tag mentions in all of them. Every repo is listed before anything is written, so a repo that cannot be read stops the run instead of having its documents deleted. Without [[repos]], REPO_DIR is the corpus and paths carry no prefix. Adding [[repos]] to an existing index changes every path, so every document is re-indexed once.

Index freshness

Every search, host_history and list_documents result starts with one verdict line, and index_status adds the per-repo detail (indexed vs current HEAD, last ingest, model, counts):

The grace period matters: warning on every normal gap between timer runs trains readers to ignore the warning. The HEAD compared is the last-fetched upstream when the checkout tracks one, else HEAD. A verdict that cannot read a repo says STALE, never fresh.

Redaction

Two layers, set in the layout's [redact] table:

• Secret hashes (secret_hashes = "<path>", or SECRET_HASHES in the environment, which wins). A file of salted sha256 hashes of the live secrets, generated where the secrets are kept, so only hashes reach the index host: line 1 salt <hex>, then <sha256(salt + value)>\t<name> per secret. Every whole token whose salted hash matches becomes [REDACTED:<name>] (tokens split on whitespace, quotes and =:,()<>[]{}|;; a trailing = run is kept so padded base64 matches; trailing sentence punctuation is also tried). This catches a live secret in any shape, including weak ones no pattern would. Once configured it is required: a missing, malformed or empty file stops ingest and the MCP server (fail closed), and a failed ingest is reported by index_status.

• Pattern families for well-known shapes: private_key, url_credentials, sk_key, vendor_token, jwt, bearer, key_value, hex32. disable = [...] turns families off where a shape is mostly evidence: a record full of checksums, digests and commit IDs is a reason to disable hex32 (and rely on the hashes for live secrets). Unknown names are refused.

Hashes run first, so a named redaction is not re-redacted by a pattern.

Ranking

search scores in four steps. Every number below is the shipped default, and all but the last step are set in the layout file (merlin.toml).

1. Fusion. The vector leg (or the literal ILIKE leg with exact=true) and the full-text leg each return up to max(50, 8k) chunks; RRF scores each chunk Σ 1/(60 + rank).

2. Rerank. Each fused score is multiplied by three factors:

   • Doc-type boost ([boost]): host, runbook, memory 1.3; journal 1.0; planning 0.9; doc 0.8; skill 0.7; memory-index 0.6. Unlisted types get 1.0.

   • Recency ([recency]): 1 + weight · 0.5^(age_days / half_life_days), with weight 0.05 and a 30-day half life, so a section dated today gets at most +5%. The age uses the section's date, else the document's. It is a tiebreak, not a sort.

   • Stale ([stale]): × 0.5 when the section marks itself superseded or obsolete, either in its own heading (heading regex, matched anywhere) or at the start of its first non-blank line (lead regex). See Corpus conventions. With inherit = ["journal"] (off by default), a chunk of such a doc whose heading chain is an ancestor of a matched stale section is penalized too: a journal's day index restates what the superseded section said.

3. Per-document grouping (per_document, on by default). One result per document, ordered by the document's best score. The section shown is normally the best one, but among the sections scoring within 10% of it (NEAR_TIE), an unmarked section beats a stale one, and then a newer-dated section beats an older one, because a later section usually supersedes an earlier one. The document keeps its best score. more counts the document's other candidate chunks, and also names the headings of up to two other sections (ALSO), which the MCP output shows as an also: § … · § … line under the heading, so the reader knows which section to open with get_document.

4. Cut to k results.

Why it is needed: RRF scores are nearly flat (roughly 0.016 to 0.03 for the whole result list), so without the rerank, a journal entry or index file that merely mentions a topic ties with, and often beats, the host record, runbook or lesson that is the canonical answer. On the reference corpus, the boost and recency step raised hit@1 from 57% to 70%, and section-level grouping raised it to 74% (see design notes).

Tuning. Copy merlin.example.toml to merlin.toml and edit [boost], [recency] and [stale]; set a factor to 1.0 or a weight to 0 to switch a step off. Measure every change with the evaluation before and after: try overrides with --boost and --recency first, then write the winner to merlin.toml. Prefer moderate values. On a small case set, a stronger boost that scores slightly higher is usually overfitting. NEAR_TIE and ALSO are constants in search.py, not layout keys. host_history does not rerank: it is ordered by date.

Corpus conventions that help retrieval

The index can only rank what the files say. These conventions make the right section win:

• Mark superseded sections at the top. Start the section's first line with (superseded YYYY-MM-DD: see § X), or put superseded or obsolete in its heading. The stale penalty only fires on those two positions: a passing mention of "superseded" further down the body doesn't count, deliberately. (Conversely, a heading such as "Why the old plan was superseded" does count.) Pointing at the replacement (see § X) lets the reader follow it.

• Keep front matter valid YAML. Quote any value that contains : (for example description: "fix: restart the agent"). Invalid front matter is ignored as a whole, so a host record silently loses its aliases: and ips: and its host tagging. Ingest now warns about it.

• Balance code fences. One unclosed fence inverts every later fence in the file. Ingest recovers (see below) and warns, but fix the file. Note that ``` and ~~~ both toggle the fence state.

• Put the summary at the top of a document. The first section of a host record or runbook is what a "what is X / how is X set up" question should land on; a short facts block there (role, address, OS, what runs on it) beats facts scattered through later sections. Generating that block from a source of truth works well.

• Give sections specific headings and, where relevant, a date (### Disk full (2030-01-15)): the heading is weighted above the body in full-text search, and the date feeds the recency factor and the near-tie pick.

Ingest warnings

Ingest prints these to stderr and indexes the document anyway:

Before the recovery, a single unclosed fence in a long journal file merged hundreds of lines into one chunk, with the wrong heading and the union of every host mentioned; that chunk matched almost every query.

Configuration (environment)

Voyage gets input_type=document|query. The OpenAI adapter sends dimensions for text-embedding-3-* models.

Model-identity guard. The first ingest stores embed_model_id (provider:model:dim) in meta. Every later ingest and every query compares it with the configuration and refuses to run on a mismatch: index was built with X; configured Y; run ingest --reembed-all. Vectors from different models are never mixed. With TEI, the server's /info model id is also checked against EMBED_MODEL.

Split document and query backends. For bulk embedding on a GPU box with queries served by a local CPU TEI, set EMBED_URL to the GPU instance and EMBED_URL_QUERY to the local one. Both must serve the same model with the same weights. TEI on CPU and on GPU with identical weights agree to a cosine of about 1.0. A different model would silently ruin retrieval; the /info check catches that for TEI. If the document backend is unreachable, ingest fails cleanly: documents are atomic, nothing falls back to another model, and the error is recorded in meta.last_ingest_error, which index_status reports.

Running

bin/ingest --pull          # incremental; safe to run from a timer every few minutes
bin/ingest --reembed-all   # after changing the embedding model: drops every vector and rebuilds
bin/mcp                    # MCP server on stdio

With Docker, add an app service to the compose project that runs Postgres and TEI. The example below assumes those services are called db and embed:

  app:
    build: ./merlin                 # this repo
    environment:
      DATABASE_URL: postgresql://merlin@db:5432/merlin
      PGPASSWORD: ${POSTGRES_PASSWORD}
      EMBED_URL: http://embed:80
      REPO_DIR: /repo
      TZ: ${TZ:-UTC}
    volumes:
      - /srv/merlin/git/corpus:/repo    # writable by uid 10001 if ingest uses --pull
    depends_on: [db, embed]
    profiles: [tools]                   # run on demand only: `docker compose run --rm app ...`

docker compose -f /opt/merlin/compose.yaml run --rm app bin/ingest --pull

Connecting Claude Code

The MCP server runs over stdio through SSH from the workstation:

claude mcp add merlin -- ssh user@index-host \
  docker compose -f /opt/merlin/compose.yaml run --rm -T app python -m merlin.mcp_server

-T matters: without it, compose allocates a TTY, which corrupts the stdio JSON-RPC stream. Each Claude Code session starts a short-lived container. Call index_status first in a session, and before trusting that something is absent.

Evaluation

merlin/evaluate.py runs a fixed question set through search() and reports hit@1, hit@3 and MRR (mean reciprocal rank of the first acceptable result, 0 when it isn't in the top k). Run it after every ranking change and after large corpus edits. A ranking change made without measurement can easily make some query worse.

The cases belong to the corpus repo, not this one, because what counts as the right answer is a fact about the corpus:

cases:
  - q: how do I restore the database from last night's dump?      # phrased the way you'd actually ask
    expect:
      - path: runbooks/postgres/restore.md                          # any listed answer counts
      - path: hosts/db1.md
        heading: backups                                            # optional: substring of the shown section heading, case-insensitive
  - q: why does the backup job fail after a kernel update?
    expect: [{path: memory/backup-kernel-module.md}]

python -m merlin.evaluate <cases.yaml> [-k 10] [--json] [--boost JSON] [--recency F]
# with Docker (the corpus checkout is mounted at /repo):
docker compose run --rm -T app python -m merlin.evaluate /repo/<path/to/eval.yaml>

• -k: results fetched per question (default 10).

• --json: summary plus per-case rank and top 3, for diffing runs.

• --boost '{"host": 1.5, "runbook": 1.5}': replaces the whole [boost] table for this run (unlisted types become 1.0); --boost '{}' turns boosting off.

• --recency 0: overrides the recency weight (0 = off). There is no override for [stale]; edit a copy of the layout and point MERLIN_CONFIG at it.

Each case prints ✅ (rank 1), ⚠ (rank 2–3) or ⛔ (lower or missing), and every miss lists the top three path § heading results, which usually shows whether the fix belongs in the ranking or in the corpus. The last line is the summary: cases 26 · hit@1 81% · hit@3 100% · MRR 0.885.

Self-quote guard. Real questions end up pasted into journals and retrieval logs, and a document that quotes a question verbatim can rank for it. The eval flags every top-3 hit whose document contains the case's question verbatim (⚑ #2 <path> quotes the question verbatim — not an answer; clean rank N) and prints a second line with hit@1 / hit@3 / MRR without such documents (answers that happen to quote it are kept). A gap between the two lines means the score is inflated by the record quoting itself. Keep retrieval logs out of the corpus, and prefer vector fusion: full-text rewards an exact quote.

Growing the set. Add a case whenever a search in real use returns the wrong thing, phrased as it was asked, with every document that would be an acceptable answer. Use heading only when the document is long and only one section answers. Keep cases you already pass: they guard against regressions. With a few dozen cases, one case is several percentage points, so prefer fixes that are principled (a convention, a general ranking rule) over weights tuned to the last miss.

Tests

python3.12 -m venv .venv && .venv/bin/pip install -r requirements-dev.txt
.venv/bin/pytest -q

The unit tests run against the synthetic corpus in tests/fixtures/corpus/, which has fictional hosts alpha and beta. The other tests depend on the environment:

• tests/test_integration.py needs DATABASE_URL. It creates and drops a throwaway schema, and uses a deterministic fake embedder. It refuses (exit code 3) a database that already holds merlin index tables, so point it at a throwaway database, never a deployment's.

• test_real_embedder also needs EMBED_URL.

• CORPUS_DIR=/path/to/repo additionally checks the chunker against a real corpus.

License

Apache License 2.0. See NOTICE.