semantic-search-mcp

A tiny, self-contained RAG-lite retrieval engine: it indexes files on disk and answers "what's semantically relevant to this query" — nothing more. It does not call an LLM and does not generate answers. It hands back the most relevant text chunks (file, line, score) so that whatever consumes it — a human, a script, or an LLM through MCP — can decide what to do with them.

Everything runs locally and offline after the first run:

• Embeddings: @huggingface/transformers running Xenova/all-MiniLM-L6-v2 with int8-quantized weights on CPU. No GPU, no API key, no network calls at query time.

• Vector store: @lancedb/lancedb — an embedded, file-backed vector database. No server process, no Docker.

• Interfaces: a CLI and a stdio MCP server, so any MCP-aware agent (Claude Code, Cursor, Zed, …) can search your corpus directly.

Quickstart

npm install -g @adborroto/semantic-search-mcp

semantic-search add ~/code/my-project      # add a folder to the corpus
semantic-search index                      # embed it (incremental on later runs)
semantic-search search "how does the retry logic work"

That's the whole setup. There is no config file to write by hand — add creates and manages it for you. To try it without installing anything:

npx @adborroto/semantic-search-mcp add ~/code/my-project

Heads up on install size: ~950MB of dependencies, plus a ~25MB embedding model downloaded on first use. Almost all of it is native binaries you can't avoid at this layer — @lancedb/lancedb (~430MB including its platform binary) and the ONNX runtime (~300MB, which ships builds for every platform in one package). Both are cached once; everything after the first run is offline.

Requirements

• Node.js >= 22 (node:sqlite, used by the fallback backend, is only stable from 22).

• ~950MB disk for dependencies and ~25MB for the embedding model, plus roughly 1–3 KB per indexed chunk.

• No GPU, no external services, no database server.

Why "RAG-lite"

A full RAG pipeline is: retrieve chunks → feed them to an LLM → LLM writes an answer. This project stops at step one. That keeps it simple, fast, cheap to run, and easy to reason about — and it composes cleanly with whatever LLM or agent framework you're already using, instead of bundling its own opinionated generation layer.

Managing the corpus

semantic-search add ~/code/api ~/notes     # add one or more folders
semantic-search list                       # show what's configured
semantic-search remove api                 # by folder name...
semantic-search remove ~/notes             # ...or by path
semantic-search config                     # where config + index actually live

add validates that each path is a real directory, resolves it to an absolute path, and skips duplicates (including the same directory reached through a symlink). remove also purges that folder's chunks from the index, so its content stops appearing in results — pass --keep-index if you want to drop it from the corpus but keep it searchable.

Where things are stored

Config and index follow the XDG base directory spec, so they survive upgrades and are shared by every install method:

Override any of it with SS_CONFIG_PATH, SS_INDEX_DIR, SS_MODEL_CACHE_DIR, or the standard XDG_CONFIG_HOME / XDG_DATA_HOME. SS_STORE_BACKEND=sqlite forces the fallback backend.

The index contains verbatim text of everything you indexed. If you point this at private code, ~/.local/share/semantic-search/ holds that content in plaintext. Never commit it, and don't attach it to a bug report.

Every option is documented in src/config.js — chunk sizing, ignore patterns, model name, top-k, concurrency. Editing config.json directly still works for those; add/remove preserve any keys they don't own.

Usage

Index

semantic-search index                      # all configured folders
semantic-search index ~/code/one-project   # just this folder, ignoring config
semantic-search index --force              # reprocess everything

Indexing is incremental: unchanged files are skipped by modification time, files whose content didn't actually change (just touched) skip re-embedding, and files deleted from disk get pruned from the index. Only what actually changed gets reprocessed.

With several folders configured, index walks them in sequence with a per-folder header and a combined total:

[1/3] my-api  /home/me/code/my-api  ─────────────────────────────
  ↺ indexed   src/auth/middleware.js  (8 chunks)
  2 indexed  1,203 skipped  16 chunks  4.1s

[2/3] my-app  /home/me/code/my-app  ─────────────────────────────
  ...

──────────────────────────────────────────────────────────────
total  5 indexed  3,891 skipped  0 deleted  41 chunks  12.3s

Each index <path> call only prunes stale entries for files under that path, so indexing folder B never touches folder A's entries.

Useful flags: --max-files <n> stops after N new files (bounds memory on huge corpora), --concurrency <n> sets parallelism, --verbose logs each file to stderr.

Search

semantic-search search "how does the retry logic work" -k 5

Prints a table of file path, line number, score, and a text preview.

Retrieval is hybrid: the query goes to two independent arms — a vector search over the embeddings, and a BM25 full-text search over the same chunks — and the two rankings are fused with Reciprocal Rank Fusion. The arms fail differently: the vector arm misses exact identifiers, error strings and config keys it has no semantic handle on; the lexical arm misses paraphrases. Running both is a recall fix, and fusing on rank rather than score keeps an unbounded BM25 score from drowning out cosine similarity.

Set "hybridSearch": false in config.json for vector-only retrieval, and "rrfK" to tune RRF's rank-smoothing constant (default 60, from the paper).

What gets indexed

Point it at a folder and everything inside it is indexed, recursively. There is no allow-list of "supported" file extensions — .dart, .kt, .java, .tsx, .sql, .erb and anything else textual are all indexed as-is, with .pdf and .docx going through a parser first.

Four things are excluded:

1. Whatever git ignores, if the folder is a git repo. .gitignore is honored at any depth, along with .git/info/exclude, your global excludes file, and negation patterns (!keep.this). This is delegated to git ls-files rather than reimplemented, so it matches git exactly — which means generated and vendored output your project already ignores stays out of the index without you maintaining a second list.

2. Your .indexignore rules (see below), for content that is committed but shouldn't be searchable — fixtures, snapshots, a checked-in secrets template.

3. Binary files, by extension (images, archives, fonts, compiled objects, model weights) and by content — a NUL byte in the first 4KB means binary, the same heuristic grep -I uses. This is a safety measure to keep non-text bytes out of the tokenizer, not a judgement about what is worth indexing.

4. Files over 500,000 bytes (maxFileSizeBytes), which is the main guard against a generated single-line megabyte file exhausting memory.

Symlinks are skipped rather than followed, so a link planted inside a folder can't pull outside content into the index.

For folders that are not git repos there is no .gitignore to lean on, so a small built-in list (node_modules/, .git/, dist/, build/, coverage/, vendor/, …) still applies.

To exclude more, drop a gitignore-style .indexignore in either place:

• inside a folder you index — patterns are relative to that folder;

• next to your config (~/.config/semantic-search/.indexignore) — applies everywhere.

See .indexignore.example for a starting point covering iOS, Android, Flutter, Ruby, and JVM build artifacts.

MCP server

semantic-search mcp

Starts a stdio MCP server exposing six tools.

search(query, k?) — semantic search, returns raw JSON:

[{ filePath, text, score, offset, startLine }, ...]

gather(query, k?, contextLines?) — same search, returned as a single formatted markdown block ready to drop into a context window:

### [1/5]  my-api  ·  src/auth/session.js  ·  line 42  ·  score 0.923
```
...chunk text...
```

contextLines (default 0) reads N extra lines around each chunk from the source file — useful when a chunk boundary cuts off context you need.

list_folders() — every configured folder with its name and absolute path. A good first call so the agent knows what corpus exists.

cat_file(filePath, startLine?, endLine?) — read a file by absolute path, as returned by search/gather. Confined to the configured folders (see Security).

grep(pattern, folder?, fileGlob?, caseSensitive?, maxResults?) — literal or regex search across the corpus, for when you need exact matches rather than similarity. Filtered against the exact same file list the indexer would index, so gitignored and .indexignored files can't leak through exact-match search.

my-api  ·  src/auth/session.js:42  export function createSession(user) {

index(root?, force?, maxFiles?, concurrency?) — trigger an incremental reindex, so an agent can refresh the corpus without shelling out.

All search tools share the same ranking and file-resolution code as the CLI; neither reimplements it.

Registering with an MCP client

Claude Code:

claude mcp add --scope user semantic-search -- semantic-search mcp
claude mcp list   # should show "✔ Connected"

Any client that takes a JSON server definition:

{
  "mcpServers": {
    "semantic-search": {
      "command": "semantic-search",
      "args": ["mcp"]
    }
  }
}

Prefer a global install over npx here: a bare npx re-resolves the package every time the server launches, adding startup latency and picking up upgrades unannounced. If you do use npx, pin the version — npx -y @adborroto/semantic-search-mcp@0.1.0 mcp.

New MCP servers are usually only picked up when a session starts, so start a fresh session after registering.

Security

This is a local, single-user tool with a simple trust model: anything inside a configured folder is readable by any MCP client that can reach the server.

• cat_file refuses paths outside the configured folders, resolving symlinks first so a link planted inside a folder can't be used to escape it.

• grep is filtered against the same file list the indexer builds — git's ignore rules plus your .indexignore — so files deliberately excluded from indexing don't leak through exact-match search instead.

• Subprocesses are spawned with argv arrays (never a shell), so patterns can't inject commands.

Given that, don't point it at a corpus you wouldn't hand to your LLM provider — chunks are returned to whatever client asked for them. See SECURITY.md.

How it works

File discovery

The rule is "index everything under the folder", and the only interesting part is what not to index. Rather than reimplementing git's ignore semantics — nested .gitignore files, negations, info/exclude, the global excludes file — a git root is enumerated with:

git ls-files -z --cached --others --exclude-standard

Tracked files plus untracked-but-not-ignored ones, scoped to the directory it runs in. Anything git ignores is absent by construction. Non-git folders fall back to a plain recursive walk with the built-in pattern list.

The same function backs both the indexer and the MCP grep tool (src/ignoreRules.js). That's deliberate: grep shells out to real grep -r, which happily reports hits inside gitignored build output, so it filters its results against the indexer's own file list. If the two derived their rules separately they would drift, and the ignore list would stop being a boundary.

Hybrid retrieval

A query runs through two arms in parallel:

• Vector — embed the query, take the nearest neighbours by cosine distance, then reorder that list with a small lexical boost for chunks containing the literal query terms.

• Lexical — BM25 over the same chunk text, via a LanceDB full-text index (the sqlite fallback computes BM25 in JS, since node:sqlite isn't guaranteed to ship FTS5).

The two rankings are fused with RRF: each list contributes 1 / (60 + rank) to every chunk it returns, and the contributions are summed. Fusing on rank rather than score is the point — cosine sits in [-1, 1] while BM25 is unbounded above, so adding or averaging raw scores lets one arm silently overwhelm the other depending on corpus size.

Why two arms at all: a lexical boost applied to the vector arm's output can only reorder what the vector query already returned. A chunk whose sole signal is an exact term match — an error code, a symbol name, a config key with no semantic neighbourhood — was unreachable if it fell outside the vector pool. The lexical arm retrieves it independently. That's a recall fix, not a reranking one, and it is why search scores now look like 0.03 rather than 0.9: they are RRF sums, not cosine similarities. Only their ordering is meaningful.

The full-text index is rebuilt at the end of each indexing run, because an FTS index doesn't cover rows added after it was built — otherwise the chunks a run just wrote would be invisible to the lexical arm.

Chunking

Text is split into paragraphs, then packed greedily into chunks of about 200 tokens with ~35 tokens of overlap, counted with the embedding model's real tokenizer rather than a character-count approximation. This isn't arbitrary: all-MiniLM-L6-v2 has a 256 token window and silently truncates anything longer, so chunks are sized to fit inside it with margin for the [CLS]/[SEP] tokens. Overlap is additionally capped so that overlap plus the next paragraph can never breach that limit — otherwise a chunk's tail would be dropped at embed time while still being returned by search.

A single paragraph larger than the hard limit (a minified bundle, one giant log line) falls back to word-level packing with the same overlap logic, and any single "word" over 500 chars is sliced first, so nothing huge is ever handed to the tokenizer in one piece.

Token counts are computed once per paragraph/word and cached for reuse during overlap calculation. An earlier version re-tokenized on every overlap lookup, which was fine on small inputs but caused runaway CPU and multi-GB memory growth on large repositories. If you extend the chunker, preserve that property.

Incremental reindexing

There's no separate manifest — the vector store is the manifest. Every stored chunk carries its source file's mtimeMs and a sha256 content hash. On each run:

1. If a file's on-disk mtime matches what's stored, skip it without reading the file.

2. If mtime changed but the content hash is identical (a touch), skip re-embedding.

3. Otherwise, delete that file's old chunks and insert freshly embedded ones.

4. After the listing, any indexed path no longer on disk (and under the root being indexed) is pruned.

Storage backends

The default is LanceDB: embedded, file-backed, real vector search. A node:sqlite + brute-force cosine fallback (src/store/sqliteFallbackStore.js) implements the same interface (src/store/vectorStore.js) for environments where LanceDB's native binding doesn't load — sandboxed containers, unusual architectures. Switch with SS_STORE_BACKEND=sqlite.

The fallback does a full table scan per search: fine for tens of thousands of chunks, not beyond. LanceDB's default metric is L2, not cosine, so this project explicitly sets .distanceType('cosine') on every query, since embeddings are compared as normalized vectors.

Project structure

src/
  config.js            Defaults + config file resolution (XDG) — the only source of tunables
  configFile.js        Read/modify/write the config file (backs add/remove/list)
  embeddings.js        transformers.js pipeline + tokenizer (lazy singletons)
  chunker.js           Token-aware paragraph packing with overlap
  ignoreRules.js       What is indexable: git ignore rules + .indexignore + binary filter,
                       shared by the indexer and grep so they can't drift apart
  safePath.js          Path confinement for the MCP file-reading tools
  version.js           Version read from package.json
  extractors/          text (anything not binary), pdf (pdf-parse), docx (mammoth)
  store/
    vectorStore.js        Storage interface + backend selector
    lancedbStore.js       LanceDB implementation (default)
    sqliteFallbackStore.js node:sqlite + manual cosine fallback
  indexer.js           List + extract + chunk + embed + incremental upsert/prune
  search.js            Hybrid retrieval: vector + BM25 arms fused with RRF — shared by CLI and MCP
  mcp-server.js        MCP stdio server: the six tools above
  index.js             CLI entrypoint (commander)
scripts/index-all.sh   Batched indexing for very large corpora on constrained hosts (Linux)

Development

git clone https://github.com/adborroto/semantic-search-mcp.git
cd semantic-search-mcp
npm install
npm test              # unit + end-to-end (node:test, no framework)
npm run test:unit     # skip the slow end-to-end test
npm run lint

A config.json in the checkout root takes precedence over the XDG location, so you can develop against a scratch corpus without touching your real setup. Tests always write to temp directories. See CONTRIBUTING.md.

Out of scope (by design)

• Answer generation. This returns chunks, not answers. Feed them to an LLM yourself.

• Reranking with a second model. Hybrid retrieval plus RRF is dependency-free and gets most of the way there — but it is not a cross-encoder reranker.

• A web UI. CLI and MCP only.

• Massive-scale corpora. Built for a personal or team-sized corpus of docs and code — tens of thousands of chunks, not millions. Both backends assume that scale.

License

MIT