FAQ RAG MCP Server

A deliberately small Retrieval-Augmented Generation (RAG) application for the Glean Solutions Engineering technical exercise. It indexes the supplied FAQ Markdown files, retrieves relevant passages with cosine similarity, generates a grounded answer through an LLM, and exposes the result as one local MCP tool: ask_faq.

The project is fully cross-platform: every setup and run command uses uv and is identical on Windows, macOS, and Linux. Giving this to a Windows user with Claude Code? Start with START_HERE_WINDOWS.md. The repository includes a CLAUDE.md setup runbook that Claude Code reads automatically and a portable project-scoped .mcp.json definition for the faq-rag server.

Thirty-second explanation

At process startup, Python reads the FAQ files, splits them into roughly 200-character chunks, creates embeddings, normalizes them, and caches the index in memory. For each question, it embeds the question, ranks chunks with cosine similarity, sends the best four text chunks to the configured LLM, and returns only a finished answer and source filenames.

flowchart LR
  A[FAQ Markdown files] --> B[~200-character chunks]
  B --> C[Document embeddings cached in RAM]
  Q[Question] --> D[Query embedding]
  C --> E[Cosine similarity]
  D --> E
  E --> F[Top 4 text chunks]
  F --> G[Grounded LLM generation]
  G --> H[answer + sources]
  H --> I[MCP client]

The embeddings are used only to locate passages. The LLM receives the original question and retrieved text, not raw embedding vectors.

Related MCP server: Inkdex

Exact MCP contract

Tool: ask_faq

Input:

{
  "question": "How do I reset my password?",
  "top_k": 4
}

Output—no additional keys:

{
  "answer": "Use the reset link on the login page [faq_auth.md].",
  "sources": ["faq_auth.md", "faq_sso.md"]
}

top_k accepts integers from 1 through 10 and defaults to 4.

Why MCP rather than the supplied HTTP option?

The RAG core would be identical behind either wrapper. MCP was selected because an AI client can discover the tool schema, decide when to call it, start the local Python process, and receive structured results without a custom HTTP client, port, URL, or health endpoint. MCP improves interoperability; it does not improve retrieval quality by itself.

This implementation uses the assignment's required stdio transport. The MCP client launches mcp_server.py as a local child process and exchanges MCP messages through the process's standard input and output. The server writes no ordinary logs to stdout because that channel is reserved for protocol traffic.

Setup (any OS: Windows, macOS, Linux)

Requirements:

• Git

• uv — it downloads a compatible Python automatically, so no separate Python install is needed. Windows: winget install -e --id astral-sh.uv; macOS: brew install uv.

• An OpenAI API key with available API credits

• An MCP client such as Claude Code or Cursor

The commands are identical in PowerShell, zsh, and bash:

git clone https://github.com/cq2wgwtzb5-lgtm/glean-faq-rag-mcp.git
cd glean-faq-rag-mcp
uv sync

Create .env.local by copying .env.example, then add the API key to it in your editor:

OPENAI_API_KEY=your_key_here

.env.local is ignored by Git. Never commit or share it.

Run the deterministic tests (no API calls):

uv run pytest -q

Run a direct end-to-end smoke test before adding MCP:

uv run rag_core.py

Claude Code discovers the checked-in .mcp.json automatically when a session starts in this folder. Follow docs/WINDOWS_MCP_SETUP.md to approve, verify, and invoke it (the steps apply to every OS). Windows users can alternatively run setup_windows.ps1, which wraps the same uv commands.

Use it from any chat thread on a machine

The project-scoped .mcp.json only loads in sessions started inside this folder. To make ask_faq available in every Claude Code session on a machine, register the server once at user scope with the absolute path to the clone (same command on every OS):

claude mcp add --scope user faq-rag -- uv run --directory "<absolute path to this repo>" mcp_server.py

Sessions inside the repository keep using the project-scoped entry; every other session uses the user-scoped one. Remove it with claude mcp remove --scope user faq-rag.

Evaluation

Unit tests use deterministic fake embeddings and make no model calls:

uv run pytest -q

The live evaluator runs five representative questions against the actual model APIs and checks expected sources, required facts, and abstention behavior:

uv run evaluate.py --output eval-results.json

eval-results.json is intentionally ignored because model output and account configuration vary. Capture or screen-share the report during the interview.

Important design decisions

In-memory NumPy index

The supplied corpus creates only a few chunks. A vector database would add deployment and review complexity without improving this result. Normalized NumPy vectors make cosine similarity a simple matrix-vector product.

Boundary-aware chunking

The target remains approximately 200 characters, as required. The implementation prefers paragraph, line, sentence, and word boundaries so text is not cut in an arbitrary place merely to hit an exact number.

One startup embedding pass

Document embeddings are generated once when the process starts and cached in RAM. Each question receives a fresh query embedding. The cache is shared corpus data—not conversation or user-session memory. When the process exits, the cache disappears and is rebuilt on the next launch.

Grounded generation and citations

The generation prompt restricts the model to retrieved FAQ context, requires exact filename citations, and instructs it to say when the FAQs do not answer a question. The response's sources list preserves retrieval order and contains only filenames from retrieved chunks.

Explicit failure behavior

The application fails immediately when OPENAI_API_KEY is missing, rejects blank questions and invalid top_k, uses a 30-second model timeout, and permits two SDK retries. Errors remain MCP errors rather than invented FAQ answers.

Known limitations and production evolution

This exercise intentionally omits a persistent index, incremental ingestion, access controls, hybrid lexical retrieval, reranking, freshness and authority signals, audit logs, and per-user personalization.

In an enterprise system, permissions must be enforced before retrieval so unauthorized text never enters the model context. Search quality would also use lexical, semantic, freshness, authority, and graph signals rather than cosine similarity alone. Those are central production concerns, but implementing them for three local files would violate the exercise's request for a lightweight solution.

Repository guide

• rag_core.py — ingestion, chunking, embeddings, retrieval, and generation

• mcp_server.py — one ask_faq MCP tool over stdio

• faqs/ — supplied FAQ corpus

• tests/ — deterministic unit and configuration tests

• evals/cases.json — five live evaluation cases

• evaluate.py — live evaluation runner

• pyproject.toml / uv.lock — pinned cross-platform environment (uv sync)

• setup_windows.ps1 — Windows convenience wrapper around the same uv steps

• CLAUDE.md — automatic setup and teaching instructions for Claude Code

• .mcp.json — portable project-scoped Claude Code MCP configuration

• START_HERE_WINDOWS.md — one-prompt handoff for the Windows user

• docs/WINDOWS_MCP_SETUP.md — Claude Code connection steps

• docs/TALK_TRACK.md — interview presentation and anticipated questions

• docs/REQUIREMENTS_TRACEABILITY.md — assignment-to-code evidence map

• docs/VALIDATION.md — passed checks and the remaining live-test boundary

Security

Do not commit API keys. Review MCP servers before enabling them; a local stdio server runs with the permissions of the user who launched the client. This server reads only its configured FAQ directory and calls the configured OpenAI models.

Interview preparation

Use docs/TALK_TRACK.md. It explains the architecture, why each choice was made, how MCP differs from HTTP, and how this small exercise maps to Glean's enterprise search and grounded-answer problem.