# Semantic Search MCP Server

A local Model Context Protocol (MCP) server that enables AI agents to perform semantic search over codebases using natural language queries. The server converts queries into efficient text search patterns (grep/ripgrep) and verifies relevance before returning results.

## Quick Setup

### Installation

```bash
pip install -e .
```

### Environment Variables

Set the following environment variables:

- `REPO_PATH` - Path to the repository to search (defaults to current directory)
- `SEARCHER_TYPE` - Searcher implementation to use (default: `sgr_gemini_flash_lite`)

**API Keys** (choose one based on your searcher type):
- For Claude-based searchers: `CLAUDE_API_KEY` or `ANTHROPIC_API_KEY`
- For Gemini-based searchers: `GOOGLE_API_KEY`, `GEMINI_API_KEY`, `AI_STUDIO`, or `VERTEX_AI_API_KEY`
- For OpenAI-based searchers: `OPENAI_API_KEY`

### Available Searchers

**SGR (Schema-Guided Reasoning) searchers** - Production-ready implementations:
- `sgr` / `sgr_gemini_flash_lite` - Default, recommended (Gemini Flash Lite)
- `sgr_gemini_flash` - SGR with Gemini Flash
- `sgr_gemini_pro` - SGR with Gemini Pro
- `sgr_gpt4o` - SGR with GPT-4o
- `sgr_gpt4o_mini` - SGR with GPT-4o Mini

**Note:** Other searcher types (`ripgrep_claude`, `agent_claude`, `agent_gemini_flash_lite`, etc.) are experimental implementations from earlier development phases and are not recommended for production use.

## Running the MCP Server

**Important:** The MCP server is not meant to be run directly in a terminal. It communicates via STDIO using JSON-RPC protocol and must be launched by an IDE or MCP client.

### Cursor Configuration

Add to your `cursor-mcp-config.json`:

```json
{
  "mcpServers": {
    "qure-semantic-search": {
      "command": "/path/to/.venv/bin/qure-semantic-search-mcp",
      "env": {
        "REPO_PATH": "/path/to/your/repo"
      }
    }
  }
}
```

After configuring, restart Cursor. The server will be automatically launched when you use the `semantic_search` tool in Cursor's AI chat.

**Note:** If you see JSON parsing errors when running the command directly in terminal, this is expected - the server requires an MCP client (like Cursor) to communicate with it via JSON-RPC protocol.

## Evaluation

### Running Evaluation

**Standard mode** (single run per query):
```bash
python -m eval.run_eval
```

**Stability mode** (10 runs per query to measure consistency):
```bash
python -m eval.run_eval --stability
```

**Stability mode with custom runs** (e.g., 20 runs per query):
```bash
python -m eval.run_eval --stability --runs 20
```

**Evaluate all searchers** (compares different searcher implementations):
```bash
python -m eval.run_all_searchers --stability
```

**Additional options:**
- `--verbose` / `-v` - Print detailed per-query statistics
- `--single-dataset` - Use only main dataset (exclude easy dataset)
- `--output <path>` - Export results to JSON file

### Datasets

The evaluation uses two datasets:

1. **Main dataset** (`data/dataset.jsonl`) - 12 challenging examples across different codebases (Django, Gin, CodeQL, QGIS, etc.) with non-trivial queries where simple keyword matching fails.

2. **Easy dataset** (`data/dataset_easy.jsonl`) - 14 simpler examples designed for faster evaluation and testing. These queries are more straightforward but still require semantic understanding.

By default, both datasets are used together (26 queries total). Use `--single-dataset` to evaluate only the main dataset.

## Metrics

For detailed metric definitions and mathematical proof of perfection, see **[METRICS_LOGIC.md](METRICS_LOGIC.md)**.

**Quick Summary:**
- **Precision@K** = TP / (TP + FP) - Fraction of returned results that are relevant
- **Recall@K** = TP / (TP + FN) - Fraction of all relevant items that were returned
- **F1@K** = Harmonic mean of Precision and Recall
- **File Discovery Rate** = Files Found / Files Expected
- **Substring Coverage** = Substrings Found / Substrings Required

**The Logic Test:** If all metrics score 1.0, the solution is mathematically perfect (see proof in METRICS_LOGIC.md).

See `eval/metrics.py` for detailed implementations.

## Performance Results

Evaluation results for `sgr_gemini_flash_lite` searcher (10 runs per query, 26 queries total):

### Overall Performance

| Metric | Value | Stability |
|--------|-------|-----------|
| **Precision@10** | 0.30 ± 0.38 | ⚠ High variance (CV=127%) |
| **Recall@10** | 0.31 ± 0.41 | ⚠ High variance (CV=133%) |
| **F1@10** | 0.29 ± 0.38 | ⚠ High variance (CV=130%) |
| **Success Rate@10** | 0.40 ± 0.46 | ⚠ High variance (CV=114%) |
| **File Discovery Rate** | 0.61 ± 0.40 | ⚠ Moderate variance (CV=66%) |
| **Substring Coverage** | 0.35 ± 0.39 | ⚠ High variance (CV=111%) |
| **Avg Latency** | 20.6s ± 7.9s | Range: 9.6s - 38.3s |
| **Stability Score** | 73.9% | 16/26 stable queries (61.5%) |

### Dataset Breakdown

#### Easy Dataset (14 examples)
- **Precision@10**: 0.40 ± 0.44
- **Recall@10**: 0.46 ± 0.49
- **F1@10**: 0.42 ± 0.45
- **File Discovery Rate**: 0.92 ± 0.13 ✓ (Good stability)
- **Avg Latency**: 15.0s ± 4.8s
- **Stability Score**: 85.9% ✓ (Good stability)

#### Main Dataset (12 examples)
- **Precision@10**: 0.17 ± 0.25
- **Recall@10**: 0.13 ± 0.18
- **F1@10**: 0.14 ± 0.20
- **File Discovery Rate**: 0.26 ± 0.30
- **Avg Latency**: 27.2s ± 5.3s
- **Stability Score**: 60.0% ⚠ (Moderate stability)

### Notes

- **High variance** in metrics is expected due to LLM non-determinism and the complexity of semantic search queries
- **File Discovery Rate** shows better stability, especially on easier queries (92% success rate)
- **Latency** varies significantly (9-38s) depending on query complexity and codebase size
- Results are evaluated on non-trivial queries where simple keyword matching fails

## Project Structure

- `src/` - Core MCP server and searcher implementations
- `eval/` - Evaluation scripts and metrics
- `data/` - Evaluation dataset and test repositories
- `scripts/` - Utility scripts for testing and debugging

## Documentation

- **[METRICS_LOGIC.md](METRICS_LOGIC.md)** - Mathematical justification for metric selection and proof of perfection
- **[KNOWN_ISSUES.md](KNOWN_ISSUES.md)** - Current limitations, known problems, and workarounds
- **[FUTURE_ROADMAP.md](FUTURE_ROADMAP.md)** - Planned improvements and mitigation strategies