# FAQ RAG MCP Server

A deliberately small Retrieval-Augmented Generation (RAG) application for the
Glean Solutions Engineering technical exercise. It indexes the supplied FAQ
Markdown files, retrieves relevant passages with cosine similarity, generates a
grounded answer through an LLM, and exposes the result as one local MCP tool:
`ask_faq`.

The project is fully cross-platform: every setup and run command uses
[`uv`](https://docs.astral.sh/uv/) and is identical on Windows, macOS, and
Linux. **Giving this to a Windows user with Claude Code?** Start with
[START_HERE_WINDOWS.md](START_HERE_WINDOWS.md). The repository includes a
`CLAUDE.md` setup runbook that Claude Code reads automatically and a portable
project-scoped `.mcp.json` definition for the `faq-rag` server.

## Thirty-second explanation

At process startup, Python reads the FAQ files, splits them into roughly
200-character chunks, creates embeddings, normalizes them, and caches the index
in memory. For each question, it embeds the question, ranks chunks with cosine
similarity, sends the best four text chunks to the configured LLM, and returns
only a finished answer and source filenames.

```mermaid
flowchart LR
  A[FAQ Markdown files] --> B[~200-character chunks]
  B --> C[Document embeddings cached in RAM]
  Q[Question] --> D[Query embedding]
  C --> E[Cosine similarity]
  D --> E
  E --> F[Top 4 text chunks]
  F --> G[Grounded LLM generation]
  G --> H[answer + sources]
  H --> I[MCP client]
```

The embeddings are used only to locate passages. The LLM receives the original
question and retrieved **text**, not raw embedding vectors.

## Exact MCP contract

Tool: `ask_faq`

Input:

```json
{
  "question": "How do I reset my password?",
  "top_k": 4
}
```

Output—no additional keys:

```json
{
  "answer": "Use the reset link on the login page [faq_auth.md].",
  "sources": ["faq_auth.md", "faq_sso.md"]
}
```

`top_k` accepts integers from 1 through 10 and defaults to 4.

## Why MCP rather than the supplied HTTP option?

The RAG core would be identical behind either wrapper. MCP was selected because
an AI client can discover the tool schema, decide when to call it, start the
local Python process, and receive structured results without a custom HTTP
client, port, URL, or health endpoint. MCP improves interoperability; it does
not improve retrieval quality by itself.

This implementation uses the assignment's required `stdio` transport. The MCP
client launches `mcp_server.py` as a local child process and exchanges MCP
messages through the process's standard input and output. The server writes no
ordinary logs to stdout because that channel is reserved for protocol traffic.

## Setup (any OS: Windows, macOS, Linux)

Requirements:

- Git
- [`uv`](https://docs.astral.sh/uv/getting-started/installation/) — it
  downloads a compatible Python automatically, so no separate Python install
  is needed. Windows: `winget install -e --id astral-sh.uv`; macOS:
  `brew install uv`.
- An OpenAI API key with available API credits
- An MCP client such as Claude Code or Cursor

The commands are identical in PowerShell, zsh, and bash:

```bash
git clone https://github.com/cq2wgwtzb5-lgtm/glean-faq-rag-mcp.git
cd glean-faq-rag-mcp
uv sync
```

Create `.env.local` by copying `.env.example`, then add the API key to it in
your editor:

```dotenv
OPENAI_API_KEY=your_key_here
```

`.env.local` is ignored by Git. Never commit or share it.

Run the deterministic tests (no API calls):

```bash
uv run pytest -q
```

Run a direct end-to-end smoke test before adding MCP:

```bash
uv run rag_core.py
```

Claude Code discovers the checked-in `.mcp.json` automatically when a session
starts in this folder. Follow
[docs/WINDOWS_MCP_SETUP.md](docs/WINDOWS_MCP_SETUP.md) to approve, verify, and
invoke it (the steps apply to every OS). Windows users can alternatively run
`setup_windows.ps1`, which wraps the same `uv` commands.

## Use it from any chat thread on a machine

The project-scoped `.mcp.json` only loads in sessions started inside this
folder. To make `ask_faq` available in **every** Claude Code session on a
machine, register the server once at user scope with the absolute path to the
clone (same command on every OS):

```bash
claude mcp add --scope user faq-rag -- uv run --directory "<absolute path to this repo>" mcp_server.py
```

Sessions inside the repository keep using the project-scoped entry; every
other session uses the user-scoped one. Remove it with
`claude mcp remove --scope user faq-rag`.

## Evaluation

Unit tests use deterministic fake embeddings and make no model calls:

```bash
uv run pytest -q
```

The live evaluator runs five representative questions against the actual model
APIs and checks expected sources, required facts, and abstention behavior:

```bash
uv run evaluate.py --output eval-results.json
```

`eval-results.json` is intentionally ignored because model output and account
configuration vary. Capture or screen-share the report during the interview.

## Important design decisions

### In-memory NumPy index

The supplied corpus creates only a few chunks. A vector database would add
deployment and review complexity without improving this result. Normalized
NumPy vectors make cosine similarity a simple matrix-vector product.

### Boundary-aware chunking

The target remains approximately 200 characters, as required. The implementation
prefers paragraph, line, sentence, and word boundaries so text is not cut in an
arbitrary place merely to hit an exact number.

### One startup embedding pass

Document embeddings are generated once when the process starts and cached in
RAM. Each question receives a fresh query embedding. The cache is shared corpus
data—not conversation or user-session memory. When the process exits, the cache
disappears and is rebuilt on the next launch.

### Grounded generation and citations

The generation prompt restricts the model to retrieved FAQ context, requires
exact filename citations, and instructs it to say when the FAQs do not answer a
question. The response's `sources` list preserves retrieval order and contains
only filenames from retrieved chunks.

### Explicit failure behavior

The application fails immediately when `OPENAI_API_KEY` is missing, rejects
blank questions and invalid `top_k`, uses a 30-second model timeout, and permits
two SDK retries. Errors remain MCP errors rather than invented FAQ answers.

## Known limitations and production evolution

This exercise intentionally omits a persistent index, incremental ingestion,
access controls, hybrid lexical retrieval, reranking, freshness and authority
signals, audit logs, and per-user personalization.

In an enterprise system, permissions must be enforced **before retrieval** so
unauthorized text never enters the model context. Search quality would also use
lexical, semantic, freshness, authority, and graph signals rather than cosine
similarity alone. Those are central production concerns, but implementing them
for three local files would violate the exercise's request for a lightweight
solution.

## Repository guide

- `rag_core.py` — ingestion, chunking, embeddings, retrieval, and generation
- `mcp_server.py` — one `ask_faq` MCP tool over stdio
- `faqs/` — supplied FAQ corpus
- `tests/` — deterministic unit and configuration tests
- `evals/cases.json` — five live evaluation cases
- `evaluate.py` — live evaluation runner
- `pyproject.toml` / `uv.lock` — pinned cross-platform environment (`uv sync`)
- `setup_windows.ps1` — Windows convenience wrapper around the same `uv` steps
- `CLAUDE.md` — automatic setup and teaching instructions for Claude Code
- `.mcp.json` — portable project-scoped Claude Code MCP configuration
- `START_HERE_WINDOWS.md` — one-prompt handoff for the Windows user
- `docs/WINDOWS_MCP_SETUP.md` — Claude Code connection steps
- `docs/TALK_TRACK.md` — interview presentation and anticipated questions
- `docs/REQUIREMENTS_TRACEABILITY.md` — assignment-to-code evidence map
- `docs/VALIDATION.md` — passed checks and the remaining live-test boundary

## Security

Do not commit API keys. Review MCP servers before enabling them; a local stdio
server runs with the permissions of the user who launched the client. This
server reads only its configured FAQ directory and calls the configured OpenAI
models.

## Interview preparation

Use [docs/TALK_TRACK.md](docs/TALK_TRACK.md). It explains the architecture,
why each choice was made, how MCP differs from HTTP, and how this small exercise
maps to Glean's enterprise search and grounded-answer problem.