mcp-markdown-ragdocs

A local-first documentation and git-history search tool with a daemon-backed MCP/CLI control plane.

What it is

This repository provides hybrid search over local Markdown and plain-text documentation plus semantic search over git history. The MCP server and CLI act as thin clients that attach to a shared local daemon for indexing, search, and operational status.

Why it exists

Technical documentation, personal notes, and project wikis are typically stored as Markdown files. Searching these collections manually or with grep is inefficient. Ragdocs provides daemon-backed search over that corpus while keeping indices synchronized with file changes and preserving a local-first deployment model.

Existing RAG solutions require manual database setup, explicit indexing steps, and ongoing maintenance. Ragdocs keeps the operational model local and simple: one daemon, daemon-backed thin clients, automatic file watching, durable task execution, and built-in index versioning.

Features

• Hybrid search combining semantic embeddings (FAISS), keyword search (Whoosh), and graph traversal (NetworkX)

• Community-based boosting: Louvain clustering detects document communities; co-community results receive score boost

• Score-aware dynamic fusion: Adjusts vector/keyword weights based on score variance per query

• HyDE (Hypothetical Document Embeddings): search_with_hypothesis tool for vague queries

• Cross-encoder re-ranking for improved precision (optional, ~50ms latency)

• Query expansion via concept vocabulary for better recall

• Git history search: Semantic search over commit history with metadata and delta context (parallel indexing for 2-4x speedup)

• Project-aware operation: project detection and overrides exist today; fully soft project metadata/ranking is still planned work

• Daemon-backed thin clients for MCP and CLI operations

• CLI daemon management and admin inspection (daemon status, queue status, index stats)

• CLI query and git-history search commands with JSON output for scripting

• Automatic file watching with debounced incremental indexing

• Huey-backed durable indexing and git refresh tasks owned by the daemon

• Zero-configuration operation with sensible defaults

• Index versioning with automatic rebuild on configuration changes

• Pluggable parser architecture: Markdown and plain text (.txt) support out-of-the-box

• Rich Markdown parsing: frontmatter, wikilinks, tags, transclusions

• Reciprocal Rank Fusion for multi-strategy result merging

• Recency bias for recently modified documents

• Local-first architecture with no external dependencies

Installation

Requires Python 3.13+.

git clone https://github.com/yourusername/mcp-markdown-ragdocs.git
cd mcp-markdown-ragdocs
uv sync

Quick Start

For VS Code / MCP Clients (Recommended)

Start the stdio-based MCP server for use with VS Code or other MCP clients:

uv run mcp-markdown-ragdocs mcp

The MCP process is now a thin client. It forwards tool discovery and tool calls to the local Ragdocs daemon over the daemon transport. On first use, the thin client will attach to an existing daemon or start one if needed.

See MCP Integration below for VS Code configuration.

For HTTP API / Development

Start the HTTP server on default port 8000:

uv run mcp-markdown-ragdocs run

The HTTP server currently runs its own in-process ApplicationContext. The daemon-backed thin-client model applies to MCP and CLI first; the HTTP path is still a separate interface.

See API Endpoints below for HTTP usage.

Basic Usage

Configuration

Create .mcp-markdown-ragdocs/config.toml in your project directory or at ~/.config/mcp-markdown-ragdocs/config.toml:

[server]
host = "127.0.0.1"
port = 8000

[indexing]
documents_path = "~/Documents/Notes"  # Path to your Markdown files
index_path = ".index_data/"           # Where to store indices

[parsers]
"**/*.md" = "MarkdownParser"          # Markdown files
"**/*.txt" = "PlainTextParser"        # Plain text files

[search]
semantic_weight = 1.0      # Weight for semantic search results
keyword_weight = 1.0       # Weight for keyword search results
recency_bias = 0.5         # Boost for recently modified documents
rrf_k_constant = 60        # Reciprocal Rank Fusion constant
min_confidence = 0.3       # Score threshold (default: 0.3)
max_chunks_per_doc = 2     # Per-document limit (default: 2)
dedup_enabled = true       # Semantic deduplication (default: true)

The server searches for configuration files in this order:

1. .mcp-markdown-ragdocs/config.toml in current directory

2. .mcp-markdown-ragdocs/config.toml in parent directories (walks up to root)

3. ~/.config/mcp-markdown-ragdocs/config.toml (global fallback)

This supports monorepo workflows where you can place a shared configuration in the repository root.

If no configuration file exists, the server uses these defaults:

• Documents path: . (current directory)

• Server: 127.0.0.1:8000

• Index storage: .index_data/

CLI Commands

Start MCP Server (stdio)

uv run mcp-markdown-ragdocs mcp

Starts the stdio-based MCP thin client for VS Code and compatible MCP clients. The MCP process forwards tool discovery and tool calls to the local daemon.

Start HTTP Server

uv run mcp-markdown-ragdocs run

Starts HTTP API server on port 8000 (default). Override with:

uv run mcp-markdown-ragdocs run --host 0.0.0.0 --port 8080

Query Documents (CLI)

Query documents directly from command line:

uv run mcp-markdown-ragdocs query "How do I configure authentication?"

With options:

# JSON output for scripting
uv run mcp-markdown-ragdocs query "authentication" --json

# Limit number of results
uv run mcp-markdown-ragdocs query "authentication" --top-n 3

# Specify project context
uv run mcp-markdown-ragdocs query "authentication" --project my-project

# Enable debug mode to see search internals
uv run mcp-markdown-ragdocs query "authentication" --debug

query and search-commits are daemon-backed thin-client commands. They will attach to the local daemon and may start it automatically if needed.

Daemon & Admin Commands

# Start the shared local daemon explicitly
uv run mcp-markdown-ragdocs daemon start

# Inspect daemon state
uv run mcp-markdown-ragdocs daemon status

# Inspect the Huey queue
uv run mcp-markdown-ragdocs queue status --json

# Inspect index/database state
uv run mcp-markdown-ragdocs index stats --json

queue status and index stats are strict daemon-backed admin commands. If the daemon is not running, start it first with ragdocs daemon start.

Debug Mode (--debug):

Displays two formatted tables showing search internals:

1. Search Strategy Results: Result counts from each search strategy

   • Vector (Semantic): FAISS embedding search results

   • Keyword (BM25): Whoosh keyword search results

   • Graph (PageRank): NetworkX graph traversal results

   • Code: Code-specific search results (if enabled)

   • Tag Expansion: Query expansion results (if enabled)

2. Compression Pipeline: Filtering stages with counts and items removed

   • Original (RRF Fusion): Combined results before filtering

   • Confidence Filter: Low-score results removed (threshold: min_confidence)

   • Content Dedup: Exact duplicate content removed

   • N-gram Dedup: Near-duplicate content removed (character n-grams)

   • Semantic Dedup: Semantically similar results removed (cosine similarity)

   • Doc Limit: Per-document chunk limit applied (max_chunks_per_doc)

Example output:

┌─ Search Strategy Results ─────────────┐
│ Strategy          │ Count            │
├───────────────────┼──────────────────┤
│ Vector (Semantic) │ 15               │
│ Keyword (BM25)    │ 8                │
│ Graph (PageRank)  │ 3                │
└───────────────────┴──────────────────┘

┌─ Compression Pipeline ─────────────────────┐
│ Stage                      │ Count │ Removed│
├────────────────────────────┼───────┼────────┤
│ Original (RRF Fusion)      │ 26    │ -      │
│ After Confidence Filter    │ 20    │ 6      │
│ After Content Dedup        │ 18    │ 2      │
│ After N-gram Dedup         │ 16    │ 2      │
│ After Semantic Dedup       │ 12    │ 4      │
│ After Doc Limit            │ 5     │ 7      │
└────────────────────────────┴───────┴────────┘

Use debug mode to:

• Understand why certain results appear or are filtered

• Tune search configuration (weights, thresholds, dedup settings)

• Diagnose low-quality results (check if semantic or keyword search dominates)

• Identify over-aggressive deduplication (high removal in dedup stages)

• Optimize performance (balance precision vs. recall via thresholds)

Configuration Management

Check your configuration:

uv run mcp-markdown-ragdocs check-config

Force a full index rebuild:

uv run mcp-markdown-ragdocs rebuild-index

MCP Integration

VS Code Configuration

Configure the MCP server in VS Code user settings or workspace settings.

File: .vscode/settings.json or ~/.config/Code/User/mcp.json

{
  "mcpServers": {
    "markdown-docs": {
      "command": "uv",
      "args": [
        "--directory",
        "/absolute/path/to/mcp-markdown-ragdocs",
        "run",
        "mcp-markdown-ragdocs",
        "mcp"
      ],
      "type": "stdio"
    }
  }
}

With project override:

{
  "mcpServers": {
    "markdown-docs": {
      "command": "uv",
      "args": [
        "--directory",
        "/absolute/path/to/mcp-markdown-ragdocs",
        "run",
        "mcp-markdown-ragdocs",
        "mcp",
        "--project",
        "my-project"
      ],
      "type": "stdio"
    }
  }
}

Claude Desktop Configuration

File: ~/Library/Application Support/Claude/claude_desktop_config.json (macOS)

{
  "mcpServers": {
    "markdown-docs": {
      "command": "uv",
      "args": [
        "--directory",
        "/absolute/path/to/mcp-markdown-ragdocs",
        "run",
        "mcp-markdown-ragdocs",
        "mcp"
      ]
    }
  }
}

Available Tools

The server exposes three MCP tools:

query_documents: Search indexed documents using hybrid search and return ranked document chunks.

search_git_history: Search git commit history using natural language queries. Returns relevant commits with metadata, message, and diff context.

search_with_hypothesis: Search using a hypothesis about expected documentation content. Embeds the hypothesis directly for semantic search (HyDE technique). Useful for vague queries where describing expected content yields better results than the query itself.

Parameters:

• query (required): Natural language query or question

• top_n (optional): Maximum results to return (1-100, default: 5)

• uniqueness_mode (optional): Result uniqueness strategy - "off" (allow duplicates), "document" (one chunk per document), or "content" (semantic deduplication, default)

• min_score (optional): Minimum confidence threshold (0.0-1.0, default: 0.3)

• similarity_threshold (optional): Semantic deduplication threshold (0.5-1.0, default: 0.85)

• show_stats (optional): Show compression statistics (default: false)

Note: Compression is enabled by default (min_score=0.3, max_chunks_per_doc=2, dedup_enabled=true) to reduce token overhead by 40-60%. Results use compact format: [N] file § section (score)\ncontent

Usage Pattern:

1. Call query_documents to identify relevant sections

2. Review returned chunks to locate specific files and sections

3. Use file reading tools to access full document context

Example query from MCP client:

{
  "query": "How do I configure authentication in the API?",
  "top_n": 5,
  "min_score": 0.3
}

The server returns ranked document chunks with file paths, header hierarchies, and relevance scores.

search_git_history: Search git commit history using natural language queries.

Parameters:

• query (required): Natural language query describing commits to find

• top_n (optional): Maximum commits to return (1-100, default: 5)

• min_score (optional): Minimum relevance threshold (0.0-1.0, default: 0.0)

• file_pattern (optional): Glob pattern to filter by changed files (e.g., src/**/*.py)

• author (optional): Filter commits by author name or email

• after (optional): Unix timestamp to filter commits after this date

• before (optional): Unix timestamp to filter commits before this date

Note: Git history search indexes up to 200 lines of diff per commit. Indexing processes 60 commits/sec on average. Search latency averages 5ms for 10k commits.

Example query:

{
  "query": "fix authentication bug",
  "top_n": 5,
  "file_pattern": "src/auth/**",
  "after": 1704067200
}

The server returns ranked commits with hash, title, author, timestamp, message, files changed, and truncated diff.

API Endpoints

Health check:

curl http://127.0.0.1:8000/health

Server status (document count, queue size, failed files):

curl http://127.0.0.1:8000/status

Query endpoint (standard):

curl -X POST http://127.0.0.1:8000/query_documents \
  -H "Content-Type: application/json" \
  -d '{"query": "authentication configuration"}'

Query endpoint (streaming SSE):

curl -X POST http://127.0.0.1:8000/query_documents_stream \
  -H "Content-Type: application/json" \
  -d '{"query": "authentication configuration", "top_n": 3}' \
  -N

The streaming endpoint returns Server-Sent Events:

event: search_complete
data: {"count": 3}

event: token
data: {"token": "Authentication"}

event: token
data: {"token": " is"}

event: done
data: {"results": [{"content": "...", "file_path": "auth.md", "header_path": ["Configuration"], "score": 1.0}]}

Example response (standard endpoint):

{
  "results": [
    {
      "chunk_id": "authentication_0",
      "content": "Authentication is configured in the auth section...",
      "file_path": "docs/authentication.md",
      "header_path": ["Configuration", "Authentication"],
      "score": 0.92
    },
    {
      "chunk_id": "security_2",
      "content": "Security settings include authentication tokens...",
      "file_path": "docs/security.md",
      "header_path": ["Security", "API Keys"],
      "score": 0.78
    }
  ]
}

MCP Stdio Format (Compact):

For MCP clients (VS Code, Claude Desktop), results use compact format:

[1] docs/authentication.md § Configuration > Authentication (0.92)
Authentication is configured in the auth section...

[2] docs/security.md § Security > API Keys (0.78)
Security settings include authentication tokens...

Factual queries (e.g., "getUserById function", "configure auth") truncate content to 200 characters:

[1] docs/api.md § Functions > getUserById (0.92)
getUserById(id: string): User | null

Retrieves user by ID. Returns null if not found. Example:
  const user = getUserById("123");
  if (user) ...

Each result contains:
- `chunk_id`: Unique identifier for the document chunk
- `content`: The text from the matching document chunk
- `file_path`: Source file path relative to documents directory
- `header_path`: Document structure showing nested headers (semantic "breadcrumbs")
- `score`: Calibrated confidence score [0, 1] representing absolute match quality (>0.9 = excellent, 0.7-0.9 = good, 0.5-0.7 = moderate, <0.3 = noise)

## Configuration Details

See [docs/configuration.md](docs/configuration.md) for exhaustive configuration reference including all TOML options, defaults, and environment variable support.

## Documentation

- [Architecture](docs/architecture.md) - System design, component overview, data flow
- [Configuration](docs/configuration.md) - Complete configuration reference
- [Hybrid Search](docs/hybrid-search.md) - Search strategies and RRF fusion algorithm
- [Integration](docs/integration.md) - VS Code MCP setup and client integration
- [Troubleshooting](docs/troubleshooting.md) - Debug mode usage, tuning guide, common issues
- [Development](docs/development.md) - Development setup, testing, contributing

## License

MIT