# mcp-ai-workspace

**Give an LLM a tool that searches your own documents — and measure whether it retrieves the right ones.**

A small, from-scratch implementation of the pattern behind every serious
"chat with your data" product: an **MCP server** that exposes document retrieval
as a tool, a **vector-RAG** pipeline underneath it, and an **evals harness** that
scores retrieval quality. No framework magic — ~200 lines you can read in one
sitting.

---

## The problem it solves

An LLM on its own can't see your private documents, and when asked about them it
will confidently make things up. The fix is *retrieval*: look up the relevant
real passages first, then answer from them, with citations.

The interesting question is **how an agent gets access to that retrieval**. The
answer here is **MCP (the Model Context Protocol)** — the emerging standard for
giving models tools. This repo wires retrieval up as an MCP tool, so any
MCP-capable client (Claude Desktop, an agent framework, a self-hosted chat UI)
can call it and answer grounded in your documents instead of guessing.

## What it does

- Indexes a folder of markdown docs into a local vector store.
- Serves a single MCP tool, `search_knowledge_base`, that returns the passages
  most relevant to a question, each with its **source file** and a similarity
  score.
- Ships an **evals** harness that checks, over a set of known question→document
  pairs, whether retrieval actually surfaces the right source (`hit@k` + `MRR`).

The bundled corpus is a fictional company handbook (`corpus/`), so the whole
thing runs end-to-end with no setup beyond `pip install`.

## Architecture

```mermaid
flowchart LR
    subgraph Offline["Indexing (ingest.py)"]
        D[corpus/*.md] --> C[chunk into passages]
        C --> E1[embed]
        E1 --> Q[(Qdrant<br/>vector store)]
    end

    subgraph Online["Serving (server.py)"]
        U[MCP client / LLM] -->|calls tool| T[search_knowledge_base]
        T --> E2[embed query]
        E2 --> Q
        Q -->|top-k passages + sources| T
        T -->|grounded context| U
    end

    EV[evals/run_evals.py] -.->|same retrieval path| Q
```

The model on the left never talks to the vector store directly. It calls the
**tool**; the tool does the retrieval. That indirection is the whole point of MCP.

## How the agent knows what it can do

An MCP tool is defined by three things, and the model reads all three to decide
when and how to call it:

| Part | In this repo | What it's for |
|------|--------------|---------------|
| **name** | `search_knowledge_base` | how the model refers to the tool |
| **description** | the tool's docstring in `server.py` | the model reads this to decide *when* to call it |
| **input schema** | the typed arguments (`query: str`, `top_k: int`) | tells the model *how* to call it |

That contract — name + description + schema — is the entire interface between the
model and your code. Get the description right and the model uses the tool well;
that's most of the "prompt engineering" in an agentic system.

## Run it

```bash
make install      # create a venv, install deps
make ingest       # build the vector index from corpus/
make evals        # score retrieval quality
make serve        # run the MCP server (stdio)
# or just:
make demo         # ingest + evals, end to end
```

To use it from an MCP client (e.g. Claude Desktop), register the server with the
example in `mcp-client-config.example.json` (fix the absolute path), restart the
client, and the model gains a `search_knowledge_base` tool.

## Evals

`make evals` runs `evals/evalset.json` — questions whose correct source document
is known — and reports:

- **hit@k** — fraction of questions where the right document is in the top-k
- **MRR** — mean reciprocal rank, which rewards ranking the right doc *first*

The run exits non-zero if `hit@k` falls below the threshold, so it can gate CI.
"I built RAG" is cheap; "I measure RAG, and here's the number" is the point.

## Stack

| Layer | Choice | Why |
|-------|--------|-----|
| Tool protocol | **MCP** (`mcp` SDK, FastMCP) | the standard way to expose tools to an LLM |
| Vector store | **Qdrant** (local, on-disk) | a real vector DB API with no service to run |
| Embeddings | **fastembed** (`BAAI/bge-small-en-v1.5`) | ONNX, CPU-only, no torch, no API key |

Every choice is swappable: a stronger embedding model, a hosted Qdrant, or a
synthesis step that calls an LLM to write the final answer from the retrieved
passages.

## What I'd add next

- An `answer` tool that calls an LLM to synthesise a cited answer from the
  retrieved passages (kept out of the core so the repo runs with no API key).
- Chunking by semantics rather than character budget.
- A reranker, and reporting precision/recall per document, not just hit@k.

## Acknowledgements

The architecture here — a self-hosted LLM fronted by MCP tools and a vector-RAG
layer — follows the pattern I learned from my DevOps professor, **Oriol Rius**,
whose course stack first showed me how these pieces fit together. This repo is my
own from-scratch, minimal re-implementation, written to internalise the concepts
and demonstrate them honestly in code I wrote myself.

## License

MIT — see [LICENSE](LICENSE).