# rag-mcp

Search over a folder of Markdown files. The `.md` files stay the source of truth and
remain ordinary files you can edit in Obsidian or any editor. Next to them sits
`index.db`, a SQLite database holding a full-text index (FTS5) and embedding
vectors. Delete it and it rebuilds from the files.

Two ways in: the `markdown-rag` command line tool, and an MCP server so an
MCP-capable client can search the vault instead of reading the whole folder.

The repository is named `rag-mcp`; the Python package is `markdown-rag`. Names
inside still say "knowledge" — config `knowledge.toml`, `KNOWLEDGE_*` variables,
MCP server `knowledge`, tools `knowledge_search` and so on. Same thing: the
`markdown-rag` tool serves a knowledge base.

One dependency: `numpy`. There is no vector database. Twelve thousand chunks are
scanned by multiplying matrices in tens of milliseconds.

## Requirements

- Python 3.11 or newer (`tomllib`)
- `numpy`
- An OpenAI-compatible embeddings API — optional, but without it only keyword
  search works

## Install

```bash
python -m pip install -e .
markdown-rag init          # writes knowledge.toml
```

Or run without installing, from the repository root:

```bash
python -m markdown_rag.cli status
```

## Configure

One file, `knowledge.toml`. It is looked up in this order: `--config`,
`KNOWLEDGE_CONFIG`, the current directory, `~/.config/knowledge/`.

```toml
[vault]
path = "~/Obsidian/MyVault"     # required
db = "index.db"                  # relative to the config file

[tiers]
names = ["CONTEXT.md", "CONTEXT-*.md", "MEMORY.md", "INDEX.md"]
dirs = ["Memory", "Notes"]

[projects]
root = "Projects"

[embed]
base_url = "https://api.openai.com/v1"
model = "text-embedding-3-small"
dim = 1536
batch = 32
timeout = 300
api_key_env = ["OPENAI_API_KEY"]
credentials_file = "~/.secrets.yaml"
```

Only `vault.path` is required. `tiers`, `projects` and `embed` all have defaults.
Without `vault.path` the tool reports what is missing and exits with code 2.

`api_key_env` lists environment variables in priority order; `credentials_file` is
the fallback, a plain file of `KEY=value` or `KEY: value` lines. The key is only
needed for embeddings.

Any value can be overridden by an environment variable: `KNOWLEDGE_VAULT`,
`KNOWLEDGE_DB`, `KNOWLEDGE_BASE_URL`, `KNOWLEDGE_EMBED_MODEL`, `KNOWLEDGE_EMBED_DIM`,
`KNOWLEDGE_BATCH`, `KNOWLEDGE_TIMEOUT`, `KNOWLEDGE_CREDENTIALS`,
`KNOWLEDGE_TIER_NAMES`, `KNOWLEDGE_TIER_DIRS`, `KNOWLEDGE_API_KEY`. A `.env` next
to the config is loaded too and never overwrites what is already set.

## Commands

```bash
markdown-rag index                  # incremental reindex
markdown-rag index --full           # reread everything
markdown-rag embed                  # compute missing embeddings
markdown-rag embed --limit 500      # in batches
markdown-rag search "query"
markdown-rag search "query" --tier 1        # overviews and memory only
markdown-rag search "query" --mode fts      # no embeddings
markdown-rag search "query" --project p     # one project
markdown-rag read "Projects/p/CONTEXT.md" --start 1 --end 40
markdown-rag context p              # project entry point
markdown-rag projects               # list projects
markdown-rag status                 # index state
```

Global flags `--config`, `--vault`, `--db` go before the subcommand.

## How search works

Files are cut into chunks on headings and paragraphs — around 600 tokens, never
more than 1000. Small tails are glued to the previous chunk (`MIN_FLUSH = 200`):
30-token chunks ranked worse than nothing because they carried no context.

Every chunk goes into two searches at once: FTS5 over words (with `ё→е`
normalisation and prefix matching) and cosine similarity over vectors. Results are
merged with Reciprocal Rank Fusion.

`RRF_K = 10`, not the 60 from the original RRF paper. With `K=60` the weight
distribution is nearly flat — rank 1 is only 1.6× rank 40 — so an item ranked 40th
by FTS and 15th by vector scored `1/100 + 1/75 = 0.0233` and beat the genuine
leaders of both lists at `1/61 = 0.0164`. Mediocre in two lists beat a leader in
one. With `K=10` the spread is 7.5×, leaders surface, and the bonus for two lists
agreeing remains. The candidate pool is 20, not 40: the tail of the list is noise.

The symptom that catches this: the top results are off-topic even though
`--mode fts` and `--mode vector` each return the correct leaders on their own.

### Tiers

| tier | what lands there | why |
|---|---|---|
| 1 | `CONTEXT*.md`, `MEMORY.md`, `INDEX.md`, `Memory/` and `Notes/` | entry points and overviews |
| 2 | everything else, meaning the conversation archive | details |

Tier 1 gets a 1.3 multiplier on equal score. Names and directories are set in
`[tiers]`, so a different vault layout needs no code change.

`[projects] root` defines where a project name comes from: `Projects/p/...` →
project `p`. `knowledge_projects` and `--project` use it.

## MCP

`markdown-rag-mcp` (or `python -m markdown_rag.mcp`) is a stdio server speaking
JSON-RPC 2.0, with no third-party MCP dependency. Tools:

```
knowledge_search(query, project?, tier?, limit?, mode?)   hybrid search
knowledge_read(path, start_line?, end_line?)              read plus hash
project_context(project)                                  entry point
knowledge_write(path, content, expected_hash?)            write with a guard
knowledge_append(path, text)                              append
knowledge_commit(title, body, project?)                   note
knowledge_status()                                        state
knowledge_projects()                                      list projects
knowledge_reindex(full?)                                  rebuild
```

Before every call the server catches the index up by mtime — an editor changes
files directly and that is picked up without a daemon or a file watcher. The
catch-up runs at most once every 10 seconds so it does not pay for `stat` on every
call.

A write indexes the file and computes its embeddings immediately; otherwise a fresh
note would be findable by words but not by meaning. An API failure does not break
the write — the chunk stays queued and goes out on the next call. The catch-up
computes at most 64 chunks per call so edits do not hang a response for minutes.

### Registering with a client

The server talks over stdio and starts as an ordinary process. The exact format
depends on the client; the meaning is always a command plus arguments:

```json
{
  "command": "markdown-rag-mcp",
  "args": ["--config", "/path/to/knowledge.toml"]
}
```

Without `--config` the server looks up the config the usual way. Clients start
servers from their own working directory, so passing the config path explicitly
avoids surprises.

CLI and MCP share one index, so `markdown-rag status` and `knowledge_status` report
the same state.

The `skills/` directory holds an optional hint for clients that support skills; the
server works the same without it.

## Tests

```bash
python -m unittest discover -s tests -t .
```

No network and no API: embeddings are replaced by a deterministic stub, and each
test gets a temporary database. Covered: chunker, schema and FTS, classification
and incremental indexing, RRF merging, reading and writing with `expected_hash`,
configuration, and the MCP layer.

There is a dedicated regression test for `RRF_K`: it asserts that `K=60` breaks the
ranking order and `K=10` does not. 157 tests run in under a second.

## Limitations

- Vector search is a full in-memory scan. Fine for 12 thousand chunks; not for tens
  of millions.
- An external embeddings API is required for semantic search.
- Tier classification and project names depend on the path layout: the `Memory/`,
  `Notes/` and `Projects/<name>/` rules work while the vault is organised that way.
- Relevance is tuned on one real vault. Expect to adjust `RRF_K`, `TIER_BOOST` and
  the tier lists for your own data.

## Layout

```
markdown_rag/
  store.py      schema, connection, FTS normalisation
  chunker.py    Markdown to chunks on headings and paragraphs
  indexer.py    incremental indexing, tier and project classification
  embed.py      OpenAI-compatible embeddings, batching, retries
  service.py    search, RRF, tiers, read and write
  mcp.py        MCP server over stdio
  cli.py        command line
mcp_server.py   entry point wrapper used when a client calls the file directly
skills/         optional client hint
tests/          157 tests, unittest, no network
```

Code comments and in-code docs are in Russian.

## What is missing

1. Reindexing on a schedule or an editor hook.
2. A question set for measuring retrieval quality.
3. Sources other than Markdown.