Semantic Search MCP Server

A local Model Context Protocol (MCP) server that enables AI agents to perform semantic search over codebases using natural language queries. The server converts queries into efficient text search patterns (grep/ripgrep) and verifies relevance before returning results.

Quick Setup

Installation

Environment Variables

Set the following environment variables:

• REPO_PATH - Path to the repository to search (defaults to current directory)

• SEARCHER_TYPE - Searcher implementation to use (default: sgr_gemini_flash_lite)

API Keys (choose one based on your searcher type):

• For Claude-based searchers: CLAUDE_API_KEY or ANTHROPIC_API_KEY

• For Gemini-based searchers: GOOGLE_API_KEY, GEMINI_API_KEY, AI_STUDIO, or VERTEX_AI_API_KEY

• For OpenAI-based searchers: OPENAI_API_KEY

Available Searchers

SGR (Schema-Guided Reasoning) searchers - Production-ready implementations:

• sgr / sgr_gemini_flash_lite - Default, recommended (Gemini Flash Lite)

• sgr_gemini_flash - SGR with Gemini Flash

• sgr_gemini_pro - SGR with Gemini Pro

• sgr_gpt4o - SGR with GPT-4o

• sgr_gpt4o_mini - SGR with GPT-4o Mini

Note: Other searcher types (ripgrep_claude, agent_claude, agent_gemini_flash_lite, etc.) are experimental implementations from earlier development phases and are not recommended for production use.

Running the MCP Server

Important: The MCP server is not meant to be run directly in a terminal. It communicates via STDIO using JSON-RPC protocol and must be launched by an IDE or MCP client.

Cursor Configuration

Add to your cursor-mcp-config.json:

After configuring, restart Cursor. The server will be automatically launched when you use the semantic_search tool in Cursor's AI chat.

Note: If you see JSON parsing errors when running the command directly in terminal, this is expected - the server requires an MCP client (like Cursor) to communicate with it via JSON-RPC protocol.

Evaluation

Running Evaluation

Standard mode (single run per query):

Stability mode (10 runs per query to measure consistency):

Stability mode with custom runs (e.g., 20 runs per query):

Evaluate all searchers (compares different searcher implementations):

Additional options:

• --verbose / -v - Print detailed per-query statistics

• --single-dataset - Use only main dataset (exclude easy dataset)

• --output <path> - Export results to JSON file

Datasets

The evaluation uses two datasets:

1. Main dataset (data/dataset.jsonl) - 12 challenging examples across different codebases (Django, Gin, CodeQL, QGIS, etc.) with non-trivial queries where simple keyword matching fails.

2. Easy dataset (data/dataset_easy.jsonl) - 14 simpler examples designed for faster evaluation and testing. These queries are more straightforward but still require semantic understanding.

By default, both datasets are used together (26 queries total). Use --single-dataset to evaluate only the main dataset.

Metrics

For detailed metric definitions and mathematical proof of perfection, see METRICS_LOGIC.md.

Quick Summary:

• Precision@K = TP / (TP + FP) - Fraction of returned results that are relevant

• Recall@K = TP / (TP + FN) - Fraction of all relevant items that were returned

• F1@K = Harmonic mean of Precision and Recall

• File Discovery Rate = Files Found / Files Expected

• Substring Coverage = Substrings Found / Substrings Required

The Logic Test: If all metrics score 1.0, the solution is mathematically perfect (see proof in METRICS_LOGIC.md).

See eval/metrics.py for detailed implementations.

Performance Results

Evaluation results for sgr_gemini_flash_lite searcher (10 runs per query, 26 queries total):

Overall Performance

Dataset Breakdown

Easy Dataset (14 examples)

• Precision@10: 0.40 ± 0.44

• Recall@10: 0.46 ± 0.49

• F1@10: 0.42 ± 0.45

• File Discovery Rate: 0.92 ± 0.13 ✓ (Good stability)

• Avg Latency: 15.0s ± 4.8s

• Stability Score: 85.9% ✓ (Good stability)

Main Dataset (12 examples)

• Precision@10: 0.17 ± 0.25

• Recall@10: 0.13 ± 0.18

• F1@10: 0.14 ± 0.20

• File Discovery Rate: 0.26 ± 0.30

• Avg Latency: 27.2s ± 5.3s

• Stability Score: 60.0% ⚠ (Moderate stability)

Notes

• High variance in metrics is expected due to LLM non-determinism and the complexity of semantic search queries

• File Discovery Rate shows better stability, especially on easier queries (92% success rate)

• Latency varies significantly (9-38s) depending on query complexity and codebase size

• Results are evaluated on non-trivial queries where simple keyword matching fails

Project Structure

• src/ - Core MCP server and searcher implementations

• eval/ - Evaluation scripts and metrics

• data/ - Evaluation dataset and test repositories

• scripts/ - Utility scripts for testing and debugging

Documentation

• METRICS_LOGIC.md - Mathematical justification for metric selection and proof of perfection

• KNOWN_ISSUES.md - Current limitations, known problems, and workarounds

• FUTURE_ROADMAP.md - Planned improvements and mitigation strategies