# mcp-server-bm25-code-search

**English** | [日本語](README.ja.md)

Fast, low-token BM25 local code search plugin & MCP server backed by SQLite FTS5 for AI coding agents (VS Code, Cursor, GitHub Copilot, ChatGPT & Codex, Kiro, Hermes Agent, OpenClaw, Grok Bot, NanoClaw, etc.), fully compliant with the **Agent Plugins** specification.

---

## ✨ Features

- 📦 **Zero External Dependencies (Python Standard Library Only)**  
  Built entirely on `sqlite3` (FTS5) and the Python standard library. Runs out-of-the-box without requiring third-party package installations (`pip install`).

- 🧩 **Agent Plugins (v1.0.0) Compliant**  
  Conforms to the [Agent Plugins](https://agent-plugins.org/compatible-clients) standard. In supported clients (VS Code, Cursor, GitHub Copilot, ChatGPT & Codex, Kiro, Hermes Agent, OpenClaw, Grok Bot, NanoClaw), simply pointing to this repository automatically discovers and loads both the MCP server (`mcp.json`) and the search guidance skill (`skills/`) with zero configuration.

- 🔤 **Code Identifier & Japanese/CJK Hybrid Tokenization**  
  Pre-processes code identifiers with subword splitting for `getUserProfile` (camelCase) and `session_token` (snake_case), combined with Python-side CJK 2-gram (bigram) tokenization for technical documentation and comments. Automatically applied to both FTS5 indexing and search queries.

- 📁 **File Path Boost (3.0x)**  
  Leverages FTS5 column weighting `bm25(code_fts, 3.0, 1.0)` to weight file path matches 3.0x higher than file content matches, pinpointing target files in fewer search iterations.

- ⚡ **Fast Incremental Indexing & Git Worktree Isolation**  
  Strict `.gitignore` compliance using `git ls-files` with ultra-fast incremental updates (0.1–0.5s during standard editing) via `git diff` / HEAD hash tracking. The index database `.bm25_index.db` is stored locally within the worktree and automatically ignored by `.gitignore`.

- 🌿 **Follows Branch Switches Automatically**  
  No rebuild or manual step after `git checkout`. On the next search, only the files deleted, modified, renamed or added since the previously indexed HEAD (via `git diff`) are swapped in, so results always match the branch you are on. Uncommitted changes are picked up too, and the index still converges to the current tree even if the previous HEAD was rebased away (covered by integration tests).

- 🔌 **MCP 2026-07-28 & Hermes Native Support**  
  - **MCP Native**: Stateless stdio JSON-RPC server adhering to the MCP 2026-07-28 specification, with deterministic tool sorting for prompt cache optimization.
  - **Hermes Agent**: Includes a lightweight Function Calling adapter layer (`hermes_adapter.py`) for environments without native MCP support.

- 🛡️ **Context Overflow Protection & Fallback Guidance**  
  Safe byte-length truncation (`--max-bytes`) preserving UTF-8 multi-byte character boundaries. Returns structured fallback messages prompting agents to switch to `grep`/`glob` when zero results are found.

---

## 📁 Directory Structure

```text
mcp-server-bm25-code-search/
├── plugin.json            # Agent Plugins v1.0.0 manifest
├── mcp.json               # Agent Plugins v1.0.0 MCP configuration
├── skills/                # Agent Skills (agent search guidelines & prompt)
│   └── bm25-search/
│       └── SKILL.md
├── bm25_search/
│   ├── db.py              # SQLite FTS5 v2 schema (chunks / code_fts / triggers)
│   ├── tokenizer.py       # Pre-tokenizer (camelCase / snake_case / CJK 2-gram)
│   ├── indexer.py         # Indexer (git ls-files, 80/20 chunking, incremental sync)
│   ├── search.py          # Search engine & CLI interface
│   ├── mcp_server.py      # MCP 2026-07-28 stateless stdio server
│   └── hermes_adapter.py  # Function Calling adapter for Hermes Agent
├── bin/
│   └── cli.js             # Node.js CLI / npx runner wrapper
├── docs/
│   ├── specification.md   # Detailed specification
│   └── plans/             # Design documents
└── tests/                 # pytest test suite
```

---

## 🚀 Getting Started

### 1. Installation via Agent Plugins (Recommended / Zero-Config)

In [Agent Plugins compatible clients](https://agent-plugins.org/compatible-clients) (**VS Code, Cursor, GitHub Copilot, ChatGPT & Codex, Kiro, Hermes Agent, OpenClaw, Grok Bot, NanoClaw**), simply adding this repository directory or loading it as a plugin automatically recognizes **both the MCP server (`mcp.json`) and the search guidance skill (`skills/`)**:

Official setup documentation for supported clients:
- **VS Code**: [Agent Plugins in VS Code](https://code.visualstudio.com/docs/agent-customization/agent-plugins)
- **Cursor**: [Cursor Plugins](https://cursor.com/docs/plugins)
- **GitHub Copilot**: [Copilot Agent Plugins](https://docs.github.com/en/copilot/concepts/agents/about-plugins)
- **ChatGPT & Codex**: [OpenAI Plugin Developers](https://developers.openai.com/plugins)
- **Kiro**: [Kiro Powers](https://kiro.dev/docs/powers/)
- **Hermes Agent**: [Hermes Portable Plugins](https://hermes-agent.nousresearch.com/docs/developer-guide/plugins#portable-agent-plugins-v1-packages)
- **OpenClaw**: [OpenClaw Plugin Bundles](https://docs.openclaw.ai/plugins/bundles)
- **Grok Bot**: [Grok Bot Automations](https://docs.x.ai/grok-bot/skills-routines-and-automations)
- **NanoClaw**: [NanoClaw Templates](https://github.com/nanocoai/nanoclaw/blob/main/docs/templates.md)

### 2. Running Search via CLI

```bash
python bm25_search/search.py "<search query>" --top-k 5 --format markdown --max-bytes 4000
```

**Key Options:**
- `<query>`: Search query (supports Japanese, CJK, camelCase, snake_case)
- `--top-k`: Maximum number of search results to return (default: `5`)
- `--format`: Output format, `markdown` or `json` (default: `markdown`)
- `--max-bytes`: Maximum output bytes; safely truncated preserving multibyte characters (default: `4000`)
- `--mode`: Token conjunction mode, `OR` or `AND` (default: `OR`)
- `--db`: Path to SQLite index DB file (default: `.bm25_index.db`)

### 3. Running as a Standalone MCP Server (uvx / npx / Manual)

The server communicates via Stdio and automatically indexes the project codebase. You can launch it instantly with `uvx` or `npx`.
If arguments are omitted, the server automatically detects the current working directory as the project root and synchronizes the incremental index (`.bm25_index.db`).

#### CLI Options
| Option | Description | Default |
|---|---|---|
| `--root`, `-r` | Target project root directory to index and search | `.` (current directory) |
| `--db` | Path to SQLite FTS5 index DB file | `<root>/.bm25_index.db` |
| `--no-auto-sync` | Disable automatic index synchronization on tool calls | Disabled (auto-sync active) |
| `--stdio` | Run stdio JSON-RPC transport loop | Enabled |

> **💡 Automatic Project Root Detection with `--db`:**  
> When specifying `--db <path>` (e.g. `--db /path/to/project/.bm25_index.db`) without an explicit `--root`, **the parent directory of the DB file is automatically detected as the project root**.  
> This allows global or shared agent configurations to easily target specific projects while maintaining seamless Auto Sync and search functionality (specifying `--root` explicitly will take precedence).

> **🔄 When Auto Sync Runs (Branch Switching):**  
> The index is updated only when the `search` tool is called (there are no git hooks or file watchers). Switching branches with `git checkout` does nothing by itself; **the index is diff-synced right before the next `search` call**. Instead of a full rebuild, only changed files are re-indexed, based on `git diff` against the previously indexed HEAD plus mtime / hash checks (uncommitted changes included). No sync happens with `--no-auto-sync`.

#### ① Using `uvx` (uv / Python)
```json
{
  "mcpServers": {
    "bm25-code-search": {
      "command": "uvx",
      "args": ["mcp-server-bm25-code-search"],
      "alwaysAllow": ["search"]
    }
  }
}
```

#### ② Using `npx` (Node.js / npm)
```json
{
  "mcpServers": {
    "bm25-code-search": {
      "command": "npx",
      "args": ["-y", "mcp-server-bm25-code-search"],
      "alwaysAllow": ["search"]
    }
  }
}
```

#### ③ Using Local Python Directly
```json
{
  "mcpServers": {
    "bm25-code-search": {
      "command": "python",
      "args": [
        "/path/to/mcp-server-bm25-code-search/bm25_search/mcp_server.py",
        "--stdio"
      ],
      "alwaysAllow": [
        "search"
      ]
    }
  }
}
```

#### ④ Specifying a Target Project DB via `--db`
```json
{
  "mcpServers": {
    "bm25-code-search": {
      "command": "uvx",
      "args": [
        "mcp-server-bm25-code-search",
        "--db",
        "/path/to/my-project/.bm25_index.db"
      ],
      "alwaysAllow": ["search"]
    }
  }
}
```
*Note: The parent directory `/path/to/my-project` is automatically recognized as the project root for indexing and synchronization.*

#### 💡 Agent Instruction Guideline (`AGENTS.md` / `CLAUDE.md`)

To prevent AI agents from repeatedly spamming `grep` and wasting context tokens, adding the following instruction to your project's `AGENTS.md`, `CLAUDE.md`, or system prompt is strongly recommended:

```markdown
## Code Search Policy
- When exploring code or investigating features across the codebase, always prioritize the MCP tool `search` (BM25 Code Search) first.
- Only fall back to `grep_search` or `glob` if `search` returns zero results or when exact literal matches for a specific symbol are required.
```

### 4. Setup in Claude Code (Manual Configuration)

Because [Claude Code](https://docs.claude.com/en/docs/claude-code) does not natively support the [Agent Plugins](https://agent-plugins.org/) standard, automatic manifest discovery (`plugin.json` / `mcp.json`) is not available. Configure it using Claude Code's MCP server registration:

> **💡 Note on Deferred Tools in Claude Code:**  
> Claude Code treats MCP tools as "Deferred Tools" to conserve context. The agent will load the tool schema before execution. Adding the instruction above to `CLAUDE.md` or placing the skill in `.claude/skills/bm25-search/SKILL.md` ensures consistent tool selection.

#### Method A: `claude mcp add` CLI Command (Recommended)

Run one of the following commands in your project root (`--scope project` generates a `.mcp.json` file to commit to git; `--scope user` registers it globally):

```bash
# Using uvx (uv / Python)
claude mcp add bm25-code-search --scope project -- uvx mcp-server-bm25-code-search

# Using npx (Node.js / npm)
claude mcp add bm25-code-search --scope project -- npx -y mcp-server-bm25-code-search

# Using local Python directly
claude mcp add bm25-code-search --scope project -e PYTHONUTF8=1 -- python /path/to/mcp-server-bm25-code-search/bm25_search/mcp_server.py --stdio
```

Verify the configuration with `claude mcp list` or the `/mcp` command inside a Claude Code session.

#### Method B: Direct `.mcp.json` File

Place a `.mcp.json` file in your project root (identical schema to the MCP configuration above):

```json
{
  "mcpServers": {
    "bm25-code-search": {
      "command": "uvx",
      "args": ["mcp-server-bm25-code-search"]
    }
  }
}
```

### 5. Using Hermes Agent Adapter

For Hermes Agent environments without MCP support, use the `bm25_search.hermes_adapter` module:

```python
from bm25_search.hermes_adapter import hermes_function_schema, run_hermes_tool

# Get Hermes Tool Schema
schema = hermes_function_schema()

# Execute Function Call from Hermes
response = run_hermes_tool({
    "name": "bm25_search",
    "arguments": {
        "query": "getUserProfile",
        "top_k": 5
    }
})
```

---

## 🧪 Running Tests

Run the unit and integration test suite using `pytest`:

```bash
pytest tests/
```

---

## 📄 Documentation

- [Specification (docs/specification.md)](docs/specification.md)
- [Design Specification (docs/plans/bm25-multi-agent-search-skill-design.md)](docs/plans/bm25-multi-agent-search-skill-design.md)

---

## 📚 References & Links

- **Agent Plugins Specification**: [Agent Plugins Specification (agentplugins/agent-plugins-spec)](https://github.com/agentplugins/agent-plugins-spec) / [agent-plugins.org](https://agent-plugins.org/)
- **Agent Skills Specification**: [Agent Skills Specification](https://agentskills.io/specification)
- **Paper**: Wang et al., *"BM25 Wins at Scale: Evaluating Agentic Search over Enterprise Corpora"* (2026)  
  [https://arxiv.org/abs/2607.26497](https://arxiv.org/abs/2607.26497)
- **Article**: Hidetoshi Sudo (KnowledgeSense, Inc.), *"Using BM25 to reduce Codex token consumption by 30%"* (Zenn, 2026)  
  [https://zenn.dev/knowledgesense/articles/9e55a3bb67729c](https://zenn.dev/knowledgesense/articles/9e55a3bb67729c)

---

## ⚖️ License

This project is licensed under the [MIT License](LICENSE).