Embeddings Searcher

# Embeddings Searcher for Claude Code Documentation A focused embeddings-based search system for navigating markdown documentation in code repositories. ## Features - **Semantic Search**: Uses sentence transformers to find relevant documentation based on meaning, not just keywords - **Markdown-Focused**: Optimized for markdown documentation with intelligent chunking - **Repository-Aware**: Organizes and searches across multiple repositories - **MCP Integration**: Provides an MCP server for integration with Cursor/Claude - **UV Package Management**: Uses UV for fast dependency management ## Quick Start for Claude Code ### 1. Clone and setup ```bash git clone <this-repo> cd kb uv sync ``` ### 2. Add your documentation Place your documentation repositories in the `repos/` directory. ### 3. Index your documentation ```bash uv run python embeddings_searcher.py --index ``` ### 4. (Optional) Convert to ONNX for faster inference ```bash uv run python onnx_convert.py --convert --test ``` ### 5. Add MCP server to Claude Code ```bash claude mcp add -- documentation-searcher uv run --directory /absolute/path/to/kb python mcp_server.py ``` *Replace `/absolute/path/to/kb` with the actual path to your project directory.* ### 6. Use in Claude Code Ask Claude Code questions like: - "Search for authentication patterns" - "Find API documentation" - "Look up configuration options" The MCP server will automatically search through your indexed documentation and return relevant results. ## Quick Start ### 1. Index Documentation First, index all markdown documentation in your repositories: ```bash uv run python embeddings_searcher.py --index ``` This will: - Find all `.md`, `.markdown`, and `.txt` files in the `repos/` directory - Chunk them intelligently based on markdown structure - Generate embeddings using sentence transformers - Store everything in a SQLite database ### 2. Search Documentation ```bash # Basic search uv run python embeddings_searcher.py --query "API documentation" # Search within a specific repository uv run python embeddings_searcher.py --query "authentication" --repo "my-project.git" # Limit results and set similarity threshold uv run python embeddings_searcher.py --query "configuration" --max-results 5 --min-similarity 0.2 ``` ### 3. Get Statistics ```bash # Show indexing statistics uv run python embeddings_searcher.py --stats # List indexed repositories uv run python embeddings_searcher.py --list-repos ``` ## MCP Server Integration The project includes an MCP server for integration with Cursor/Claude: ```bash # Start the MCP server uv run python mcp_server.py ``` ### MCP Tools Available 1. **search_docs**: Search through documentation using semantic similarity 2. **list_repos**: List all indexed repositories 3. **get_stats**: Get indexing statistics 4. **get_document**: Retrieve full document content by path ## Project Structure ``` kb/ ├── embeddings_searcher.py # Main searcher implementation ├── mcp_server.py # MCP server for Claude Code integration ├── onnx_convert.py # ONNX model conversion utility ├── pyproject.toml # UV project configuration ├── embeddings_docs.db # SQLite database with embeddings ├── sentence_model.onnx # ONNX model (generated) ├── model_config.json # Model configuration (generated) ├── tokenizer/ # Tokenizer files (generated) ├── repos/ # Your documentation repositories │ ├── project1.git/ │ ├── project2.git/ │ └── documentation.git/ └── README.md # This file ``` ## How It Works ### Intelligent Chunking The system chunks markdown documents based on: - Header structure (H1, H2, H3, etc.) - Content length (500 words per chunk) - Semantic boundaries ### Embedding Generation - Uses `all-MiniLM-L6-v2` sentence transformer model by default - Supports ONNX models for faster inference - Caches embeddings for efficient updates ### Search Algorithm 1. Generates embedding for your query 2. Compares against all document chunks using cosine similarity 3. Returns ranked results with context and metadata 4. Supports repository-specific searches ## CLI Options ### embeddings_searcher.py ```bash # Indexing --index # Index all repositories --force # Force reindex of all documents # Search --query "search terms" # Search query --repo "repo-name" # Search within specific repository --max-results 10 # Maximum results to return --min-similarity 0.1 # Minimum similarity threshold # Information --stats # Show indexing statistics --list-repos # List indexed repositories # Configuration --kb-path /path/to/kb # Path to knowledge base --db-path embeddings.db # Path to embeddings database --model model-name # Sentence transformer model --ignore-dirs [DIRS...] # Directories to ignore during indexing ``` ### mcp_server.py ```bash --kb-path /path/to/kb # Path to knowledge base --docs-db-path embeddings.db # Path to docs embeddings database --model model-name # Sentence transformer model ``` ## ONNX Model Conversion For faster inference, you can convert the sentence transformer model to ONNX format: ```bash # Convert model to ONNX uv run python onnx_convert.py --convert # Test ONNX model uv run python onnx_convert.py --test # Convert and test in one command uv run python onnx_convert.py --convert --test ``` ## Example Usage ```bash # Index documentation uv run python embeddings_searcher.py --index # Search for API documentation uv run python embeddings_searcher.py --query "API endpoints" # Search for authentication in specific repository uv run python embeddings_searcher.py --query "user authentication" --repo "my-project.git" # Get detailed statistics uv run python embeddings_searcher.py --stats ``` ## Performance - **Indexing**: ~1400 documents in ~1 minute - **Search**: Sub-second response times - **Storage**: ~50MB for embeddings database with 6500+ chunks - **Memory**: ~500MB during indexing, ~200MB during search ## Troubleshooting ### Unicode Errors Some files may have encoding issues. The system automatically falls back to latin-1 encoding for problematic files. ### Large Files Files larger than 1MB are automatically skipped to prevent memory issues. ### Model Loading If sentence-transformers is not available, the system will attempt to use ONNX models or fall back to dummy embeddings for testing.