AI Support Agent - MCP + RAG + Evals + Automation

An end-to-end AI support agent built to demonstrate production-grade AI integration: a FastAPI backend exposed as an MCP (Model Context Protocol) server, backed by semantic search over a real knowledge base, with automated evaluation and observability, and triggered live from Slack via n8n.

This isn't a chatbot demo - it's the architecture behind what companies actually pay AI engineers to build: an agent that can safely look up real business data, answer questions grounded in real documentation, hand off to a human when it can't resolve something, and prove (with logged evals) that it isn't hallucinating.

What it does

A support request comes in - via Claude Desktop, or a real Slack message - and the agent:

1. Looks up order status from a real backend database (get_order_status)

2. Searches a knowledge base semantically - not keyword matching - to answer policy questions (search_docs)

3. Creates a support ticket when it can't resolve something automatically (create_ticket)

Every tool call is traced end-to-end (input, output, tokens, cost, latency), and a small automated eval suite verifies the agent is grounded — it says "I don't know" instead of guessing, and correctly matches paraphrased questions to the right policy.

Related MCP server: Customer Support MCP Server

Architecture

Slack message
     │
     ▼
n8n (Code node filters bot's own messages to prevent loops)
     │
     ▼
FastAPI wrapper endpoint (/agent/support-query)
     │
     ├─▶ Order lookup (SQLite via SQLAlchemy)
     │
     └─▶ Semantic search:
              OpenAI embeddings (text-embedding-3-small)
                     │
                     ▼
              Supabase (pgvector) similarity search
     │
     ▼
Response posted back to Slack

─────────────────────────────────────────

Claude Desktop
     │
     ▼
MCP server (stdio) ── same three tools, same backend
     │
     ▼
Langfuse (traces every call: input, output, tokens, cost)

Stack

Tools exposed

get_order_status(order_id: int)

Looks up an order's status and tracking number. Returns a clear "not found" message for invalid IDs rather than guessing.

search_docs(query: str)

Semantic search over a knowledge base (shipping policy, returns, order tracking docs). Matches paraphrased queries — e.g. "can I get my money back" correctly retrieves the Return Policy doc with zero keyword overlap — because it embeds and compares meaning, not text.

create_ticket(customer_name: str, issue: str)

Creates a support ticket for issues the agent can't resolve automatically, so nothing falls through the cracks.

Evals

A small automated eval suite (app/run_evals.py) runs 10 test cases across all three tools, including:

• Correct order lookups and a graceful "no order found" for invalid IDs

• Semantic search matching paraphrased queries with no literal keyword overlap to the source doc

• Correctly saying "no relevant documents found" for off-topic queries, instead of hallucinating an answer

• Ticket creation

Current pass rate: 10/10 (100%). Every eval run is also traced in Langfuse, so pass/fail results are backed by inspectable, real execution data — not just a claimed number.

Running it locally

1. Backend

python -m venv venv
venv\Scripts\activate          # Windows
pip install -r requirements.txt
uvicorn app.main:app --reload

2. Seed sample data

python -m app.seed

3. Set up RAG

• Create a Supabase project, enable the pgvector extension

• Run the SQL in sql/setup.sql to create the doc_embeddings table and match_docs function

• Add your keys to .env (see .env.example)

• Embed and upload the seeded docs:

  python -m app.embed_docs

4. Run the MCP server (for Claude Desktop)

python -m app.mcp_server

Add it to claude_desktop_config.json pointing at your venv's Python and this script.

5. Run evals

python -m app.run_evals

6. n8n + Slack automation

• docker run -d --name n8n -p 5678:5678 -v n8n_data:/home/node/.n8n docker.n8n.io/n8nio/n8n

• Import the workflow from n8n/support-agent-workflow.json

• Point a Slack app's Event Subscriptions at your n8n webhook (via ngrok for local dev)

Environment variables

OPENAI_API_KEY=
SUPABASE_URL=
SUPABASE_KEY=
LANGFUSE_PUBLIC_KEY=
LANGFUSE_SECRET_KEY=
LANGFUSE_HOST=

Why this project

This project demonstrates the parts of production AI work that matter most to a client evaluating an AI engineer:

• Real backend integration via MCP, not just prompt engineering

• Grounded answers via real semantic search, not keyword matching dressed up as "AI"

• Proof it's not hallucinating via automated evals and full observability - not just a demo that happened to work once

• Deployable into existing tools (Slack, and by extension any CRM/helpdesk) via automation, not confined to a chat window

The architecture here - FastAPI backend + MCP + RAG + evals + automation — is directly reusable: swap the orders/tickets/docs tables for a real client's CRM or helpdesk data model, and this becomes a client project rather than a from-scratch build.