# mdctx

[![npm version](https://img.shields.io/npm/v/mdctx.svg)](https://www.npmjs.com/package/mdctx)
[![license](https://img.shields.io/npm/l/mdctx.svg)](https://github.com/zachkepe/mdctx/blob/main/LICENSE)

A zero-ML-dependency keyword index for markdown context files. Built for
large workspaces where dumping every doc into an LLM's context window
wastes tokens. `mdctx` indexes your markdown once, then answers "which
files are relevant to X" in milliseconds, from a flat JSON file you can
commit and diff like code.

## Why this exists

There are already several MCP servers that index markdown notes with
hybrid keyword+semantic search (qmd, dotmd, foam-notes-mcp, and others).
`mdctx` is deliberately narrower than those:

- **No SQLite, no vector DB, no embedding model to download.** The index
  is a single flat JSON file you can read, diff, and review in a PR.
- **Keyword-only by design**, using RAKE-style extraction and BM25
  ranking. Fully deterministic, fully offline, no cost per index update.
- **Cross-model on purpose.** Ships as both a CLI (usable from any script
  or agent that can shell out) and an MCP server (usable by Claude,
  ChatGPT, Cursor, and anything else that speaks MCP).

If you need semantic/fuzzy matching across large free-text notes, one of
the heavier tools above is likely a better fit. If you have a workspace
of structured docs and want a fast, transparent, git-friendly index,
that's what this is for.

## Install

```bash
npm install -g mdctx
```

Or via Homebrew (not yet in homebrew-core, so tap this repo first):

```bash
brew tap zachkepe/mdctx
brew install mdctx
```

## CLI usage

```bash
# Index every .md/.mdx file under a directory (incremental, only
# changed files are re-processed)
mdctx build ./docs

# Search the index
mdctx search "auth flow"
mdctx search "auth flow" --json --limit 3
```

## MCP usage

Add to your MCP client config (e.g. Claude Desktop's
`claude_desktop_config.json`):

```json
{
  "mcpServers": {
    "mdctx": {
      "command": "mdctx-mcp",
      "env": {
        "MDCTX_ROOT": "/absolute/path/to/your/docs"
      }
    }
  }
}
```

This exposes three tools: `search_context`, `refresh_index`, and
`list_context`. `search_context` auto-heals: if the index is missing,
corrupt, or was written by an older mdctx version, it gets rebuilt
transparently on the next call rather than erroring out.

## Keeping the index up to date automatically

Run this once per repo:

```bash
mdctx init ./docs
```

This wires up everything needed so you never have to remember to run
`mdctx build` by hand:

- **`.mdctx.json`**: records which directory holds your docs and where
  the index lives, so the hooks and CI job below don't need arguments.
- **A pre-commit hook**: rebuilds the index and stages it before every
  commit, so `context-index.json` always reflects what you're about to
  commit. If nothing changed, the rebuild is a no-op and nothing gets
  added to the commit.
- **post-merge / post-checkout hooks**: rebuild the index after a pull
  or a branch switch, so local search results match whatever's on disk.
  These are advisory, if the rebuild fails for any reason they print a
  warning to stderr rather than blocking the merge or checkout.
- **`.github/workflows/mdctx-index.yml`**: a CI check that rebuilds the
  index on every push/PR and fails the build if it doesn't match what's
  committed. This is the backstop for commits made with `--no-verify`,
  from a machine that never ran `mdctx init`, or by a bot.

Re-running `mdctx init` is safe. It won't overwrite a hook it didn't
install unless you pass `--force`, so it won't clobber pre-existing
hooks from another tool.

If you'd rather not touch git hooks, the CI workflow alone is enough to
catch staleness; just don't run `mdctx init` and use `mdctx build`
manually or in whatever pipeline already touches your docs.

## How it works

1. `mdctx build` walks the target directory for `.md`/`.mdx` files,
   hashes each one, and skips any file whose hash hasn't changed since
   the last run.
2. For new/changed files, it extracts a title (from frontmatter or the
   first heading) and a set of keywords using a RAKE-style scoring
   algorithm. No LLM call, no network request. The number of keywords
   per file is auto-sized to that file's word count (roughly one keyword
   per 30 words, clamped to 5-25) rather than a flat count for every
   file, so a two-paragraph doc doesn't get padded with low-signal
   phrases and a long doc doesn't lose relevant terms to an arbitrarily
   small cutoff. Pass `-k <n>` to `mdctx build` to pin a fixed count for
   every file instead. Because of the incremental hash cache, this only
   affects files that are new or whose content changed on that run. An
   already-indexed file keeps its existing keyword count until it's
   re-processed. Delete `context-index.json` and rebuild if you want to
   re-score everything under the new sizing immediately.

   Words that appear in a markdown heading or a bold/emphasis span get a
   score multiplier, on the reasoning that an author calling something
   out explicitly is a stronger relevance signal than RAKE's raw
   co-occurrence statistics. This matters most for short, specific terms
   (a product name, a technology) that would otherwise lose to longer,
   more interconnected prose phrases even in a long document. It's not a
   complete fix for every case: `*`/`_` are also phrase delimiters (they
   have to be, so raw asterisks don't leak into extracted phrase text),
   so a single word wrapped in `**word**` gets isolated as its own
   one-word phrase before the boost applies, and a short isolated phrase
   can still lose to a longer unboosted one. Bolding a multi-word span
   doesn't have this problem, only single words do.
3. Everything is written to `context-index.json`, one JSON object per
   file, human-readable and git-diffable. A rebuild that finds no content
   changes writes back the exact same bytes (the `root` path is stored
   relative to the index file, and the `generatedAt` timestamp only moves
   when something actually changed), so running `mdctx build` repeatedly
   never produces spurious diffs.
4. `mdctx search` loads that JSON, builds a BM25 index in memory over
   each file's title+keywords, and returns ranked results.

## Benchmark

To check whether keyword-based retrieval actually saves tokens in practice,
we ran a controlled comparison: two identical research agents given the same
task against the same real, mid-size production codebase (a security
software platform with a job queue, worker pool, and container-based task
execution, roughly 30 markdown docs plus a large multi-service backend).
Same prompt, same target repo, same task ("research how queuing and
concurrency work, identify the parallelization control points"). The only
variable was retrieval method: one agent ran `mdctx search` first and read
only what it surfaced; the other explored from scratch with grep/find/direct
reads, no index.

| Metric | mdctx | Baseline | Improvement |
|---|---|---|---|
| Tokens used | 68,748 | 86,314 | **20.3% fewer** |
| Tool calls | 32 | 45 | **28.9% fewer** |
| Wall-clock time | ~167s | ~206s | **~19% faster** |

Both agents independently converged on the same core architectural
findings: the primary concurrency limiter, the split between the two
container-execution models, and the coupling between worker replica count
and the concurrency cap, using a fifth fewer tokens and nearly a third
fewer tool calls with mdctx. That's the expected result of pointing an
agent at the handful of files that actually matter instead of letting it
rediscover the codebase's shape from scratch: less redundant exploration,
the same answer, faster and cheaper. For deeper audits where exhaustive
coverage matters more than speed, pair `mdctx search` with a broader
follow-up pass over the next few ranked results rather than stopping at
the top hit. The index is there to narrow where you start, not to replace
judgment about how far to go.

## Development

```bash
npm install
npm run build
npm test
```

## License

MIT