Embeddings Searcher

Embeddings Searcher for Claude Code Documentation

A focused embeddings-based search system for navigating markdown documentation in code repositories.

Features

Semantic Search: Uses sentence transformers to find relevant documentation based on meaning, not just keywords
Markdown-Focused: Optimized for markdown documentation with intelligent chunking
Repository-Aware: Organizes and searches across multiple repositories
MCP Integration: Provides an MCP server for integration with Cursor/Claude
UV Package Management: Uses UV for fast dependency management

Quick Start for Claude Code

1. Clone and setup

git clone <this-repo>
cd kb
uv sync

2. Add your documentation

Place your documentation repositories in the repos/ directory.

3. Index your documentation

uv run python embeddings_searcher.py --index

4. (Optional) Convert to ONNX for faster inference

uv run python onnx_convert.py --convert --test

5. Add MCP server to Claude Code

claude mcp add -- documentation-searcher uv run --directory /absolute/path/to/kb python mcp_server.py

Replace /absolute/path/to/kb with the actual path to your project directory.

6. Use in Claude Code

Ask Claude Code questions like:

"Search for authentication patterns"
"Find API documentation"
"Look up configuration options"

The MCP server will automatically search through your indexed documentation and return relevant results.

Quick Start

1. Index Documentation

First, index all markdown documentation in your repositories:

uv run python embeddings_searcher.py --index

This will:

Find all .md, .markdown, and .txt files in the repos/ directory
Chunk them intelligently based on markdown structure
Generate embeddings using sentence transformers
Store everything in a SQLite database

2. Search Documentation

# Basic search
uv run python embeddings_searcher.py --query "API documentation"

# Search within a specific repository
uv run python embeddings_searcher.py --query "authentication" --repo "my-project.git"

# Limit results and set similarity threshold
uv run python embeddings_searcher.py --query "configuration" --max-results 5 --min-similarity 0.2

3. Get Statistics

# Show indexing statistics
uv run python embeddings_searcher.py --stats

# List indexed repositories
uv run python embeddings_searcher.py --list-repos

MCP Server Integration

The project includes an MCP server for integration with Cursor/Claude:

# Start the MCP server
uv run python mcp_server.py

MCP Tools Available

search_docs: Search through documentation using semantic similarity
list_repos: List all indexed repositories
get_stats: Get indexing statistics
get_document: Retrieve full document content by path

Project Structure

kb/
├── embeddings_searcher.py    # Main searcher implementation
├── mcp_server.py            # MCP server for Claude Code integration
├── onnx_convert.py          # ONNX model conversion utility
├── pyproject.toml           # UV project configuration
├── embeddings_docs.db       # SQLite database with embeddings
├── sentence_model.onnx      # ONNX model (generated)
├── model_config.json        # Model configuration (generated)
├── tokenizer/               # Tokenizer files (generated)
├── repos/                   # Your documentation repositories
│   ├── project1.git/
│   ├── project2.git/
│   └── documentation.git/
└── README.md               # This file

How It Works

Intelligent Chunking

The system chunks markdown documents based on:

Header structure (H1, H2, H3, etc.)
Content length (500 words per chunk)
Semantic boundaries

Embedding Generation

Uses all-MiniLM-L6-v2 sentence transformer model by default
Supports ONNX models for faster inference
Caches embeddings for efficient updates

Search Algorithm

Generates embedding for your query
Compares against all document chunks using cosine similarity
Returns ranked results with context and metadata
Supports repository-specific searches

CLI Options

embeddings_searcher.py

# Indexing
--index                    # Index all repositories
--force                    # Force reindex of all documents

# Search
--query "search terms"     # Search query
--repo "repo-name"        # Search within specific repository
--max-results 10          # Maximum results to return
--min-similarity 0.1      # Minimum similarity threshold

# Information
--stats                   # Show indexing statistics
--list-repos             # List indexed repositories

# Configuration
--kb-path /path/to/kb    # Path to knowledge base
--db-path embeddings.db  # Path to embeddings database
--model model-name       # Sentence transformer model
--ignore-dirs [DIRS...]  # Directories to ignore during indexing

mcp_server.py

--kb-path /path/to/kb         # Path to knowledge base
--docs-db-path embeddings.db  # Path to docs embeddings database
--model model-name            # Sentence transformer model

ONNX Model Conversion

For faster inference, you can convert the sentence transformer model to ONNX format:

# Convert model to ONNX
uv run python onnx_convert.py --convert

# Test ONNX model
uv run python onnx_convert.py --test

# Convert and test in one command
uv run python onnx_convert.py --convert --test

Example Usage

# Index documentation
uv run python embeddings_searcher.py --index

# Search for API documentation
uv run python embeddings_searcher.py --query "API endpoints"

# Search for authentication in specific repository
uv run python embeddings_searcher.py --query "user authentication" --repo "my-project.git"

# Get detailed statistics
uv run python embeddings_searcher.py --stats

Performance

Indexing: ~1400 documents in ~1 minute
Search: Sub-second response times
Storage: ~50MB for embeddings database with 6500+ chunks
Memory: ~500MB during indexing, ~200MB during search

Troubleshooting

Unicode Errors

Some files may have encoding issues. The system automatically falls back to latin-1 encoding for problematic files.

Large Files

Files larger than 1MB are automatically skipped to prevent memory issues.

Model Loading

If sentence-transformers is not available, the system will attempt to use ONNX models or fall back to dummy embeddings for testing.

Embeddings Searcher