# hybrid-rag-mcp

A production-minded **Model Context Protocol (MCP) server** that gives any LLM agent
a high-quality `search_docs` tool backed by **hybrid retrieval** (BM25 + dense vectors,
fused with Reciprocal Rank Fusion) and **cross-encoder reranking** — with a real
**evaluation harness**, **prompt-injection guardrails**, and **OpenTelemetry tracing**
built in.

> One tool, exposed over MCP, that is *measurably* good and *demonstrably* safe.

---

## Documentation

Full documentation lives in **[`docs/`](docs/README.md)**. Quick links:

| | |
|---|---|
| [Overview](docs/overview.md) · [Installation](docs/installation.md) · [Usage](docs/usage.md) | Get started |
| [Architecture](docs/architecture.md) · [API Reference](docs/api-reference.md) · [Configuration](docs/configuration.md) | Reference |
| [Retrieval](docs/retrieval.md) · [Evaluation](docs/evaluation.md) · [Security](docs/security.md) · [Observability](docs/observability.md) | Deep dives |
| [Development](docs/development.md) · [FAQ](docs/faq.md) | Extend & troubleshoot |

---

## Why this project

Most retrieval demos stop at "embed the query, take cosine top-k." Real systems don't.
This repo shows the parts that actually matter in production:

| Capability | What it demonstrates |
|---|---|
| **MCP server** (`FastMCP`, stdio transport) | Tool-calling integration any MCP client (Claude Desktop, IDEs, custom agents) can use |
| **Hybrid retrieval** (BM25 + dense, RRF fusion) | You understand lexical vs. semantic search and how to combine them |
| **Cross-encoder reranking** | You can improve precision@k, not just recall — and *measure* it |
| **Eval harness** (recall@k, MRR) | You prove quality with numbers, before/after each stage |
| **Prompt-injection guardrail** | You treat tool inputs as untrusted (OWASP LLM01) |
| **OpenTelemetry tracing** | You can debug and observe an agent tool in production |

## Architecture

```mermaid
flowchart LR
    A[MCP Client] -- search_docs query --> B[FastMCP Server]
    B --> G{Injection guardrail}
    G -- flagged --> R[Reject + reason]
    G -- clean --> P[Retrieval pipeline]
    subgraph P [Retrieval pipeline]
        C[BM25 lexical] --> F[RRF fusion]
        D[Dense vector] --> F
        F --> E[Cross-encoder rerank]
    end
    P --> H[Top-k passages]
    H --> A
    B -. spans .-> T[(OpenTelemetry)]
```

## Quickstart

Requires **Python 3.10+** (tested on 3.13).

```bash
# 1. Create an isolated environment
python3.13 -m venv .venv && source .venv/bin/activate

# 2. Install the core (BM25 works immediately — no model downloads)
pip install -e .

# 3. Run the tests and the retrieval eval
make test
make eval

# 4. Start the MCP server (stdio)
make run
```

The server ships with a small sample corpus (`src/hybrid_rag_mcp/corpus/sample_docs.jsonl`)
so everything runs end-to-end on first clone. Swap in your own corpus to make it yours.

### Optional: enable dense vectors + reranking

The advanced retrieval stages activate automatically when their (heavier) dependencies
are installed; otherwise the pipeline gracefully degrades to BM25-only.

```bash
pip install -e ".[full]"   # sentence-transformers, flashrank, opentelemetry
```

### Connect it to an MCP client

Add this to your client's MCP config (example for Claude Desktop
`claude_desktop_config.json`):

```json
{
  "mcpServers": {
    "hybrid-rag": {
      "command": "/absolute/path/to/.venv/bin/python",
      "args": ["-m", "hybrid_rag_mcp.server"]
    }
  }
}
```

## Evaluation

`make eval` scores retrieval quality on `evals/qa_dataset.jsonl` and prints a table so you
can see the contribution of each stage. Reranking should lift precision — prove it:

| Config | Recall@5 | MRR |
|---|---|---|
| BM25 only | _run `make eval`_ | _._ |
| + dense (RRF fusion) | _._ | _._ |
| + cross-encoder rerank | _._ | _._ |

> Fill this table with your real numbers and screenshot it in your write-up. Numbers win interviews.

## Security / red-teaming

`make redteam` runs a battery of prompt-injection payloads (`redteam/injection_payloads.jsonl`)
through the input guardrail and reports how many were caught. Extend the payload set and the
detector rules — closing the gap between them is the interesting part.

## Observability

Every tool call is wrapped in an OpenTelemetry span (query, stage latencies, result count).
By default spans print to the console; point `OTEL_EXPORTER_OTLP_ENDPOINT` at a collector
(Jaeger, Grafana Tempo, Langfuse) to visualize traces.

## Repository layout

```
hybrid-rag-mcp/
├── src/hybrid_rag_mcp/
│   ├── server.py            # FastMCP entrypoint, exposes search_docs
│   ├── pipeline.py          # composes guardrail -> retrieve -> fuse -> rerank
│   ├── retrieval/           # bm25 / vector / hybrid (RRF) / rerank
│   ├── security/            # prompt-injection guardrail
│   ├── observability/       # OpenTelemetry tracing helpers
│   └── corpus/              # sample corpus (jsonl)
├── evals/                   # recall@k + MRR harness, QA dataset, promptfoo config
├── redteam/                 # injection payloads + runner
└── tests/                   # pytest
```

## Roadmap — make it yours

These are deliberately left for you to implement and defend in interviews:

- [ ] Replace the sample corpus with a real one (your notes, a docs site, arXiv abstracts).
- [ ] Add a second MCP tool (e.g., `fetch_document(id)` or `summarize(query)`).
- [ ] Swap the embedding model and benchmark quality vs. latency.
- [ ] Add caching for embeddings and rerank scores; measure cost/latency savings.
- [ ] Wire traces into Jaeger or Langfuse and add a screenshot to the README.
- [ ] Expand the red-team set and report your catch rate over time.
- [ ] Migrate to the MCP SDK 2.0 `MCPServer` API once it stabilizes (currently pinned to
      the stable 1.x `FastMCP` line for maximum tutorial/Claude Desktop compatibility).

## License

MIT — see [LICENSE](LICENSE).