# AI Support Agent - MCP + RAG + Evals + Automation

An end-to-end AI support agent built to demonstrate production-grade AI integration: a FastAPI backend exposed as an MCP (Model Context Protocol) server, backed by semantic search over a real knowledge base, with automated evaluation and observability, and triggered live from Slack via n8n.

This isn't a chatbot demo - it's the architecture behind what companies actually pay AI engineers to build: an agent that can safely look up real business data, answer questions grounded in real documentation, hand off to a human when it can't resolve something, and prove (with logged evals) that it isn't hallucinating.

## What it does

A support request comes in - via Claude Desktop, or a real Slack message - and the agent:

1. **Looks up order status** from a real backend database (`get_order_status`)
2. **Searches a knowledge base semantically** - not keyword matching - to answer policy questions (`search_docs`)
3. **Creates a support ticket** when it can't resolve something automatically (`create_ticket`)

Every tool call is traced end-to-end (input, output, tokens, cost, latency), and a small automated eval suite verifies the agent is grounded — it says "I don't know" instead of guessing, and correctly matches paraphrased questions to the right policy.

## Architecture

```
Slack message
     │
     ▼
n8n (Code node filters bot's own messages to prevent loops)
     │
     ▼
FastAPI wrapper endpoint (/agent/support-query)
     │
     ├─▶ Order lookup (SQLite via SQLAlchemy)
     │
     └─▶ Semantic search:
              OpenAI embeddings (text-embedding-3-small)
                     │
                     ▼
              Supabase (pgvector) similarity search
     │
     ▼
Response posted back to Slack

─────────────────────────────────────────

Claude Desktop
     │
     ▼
MCP server (stdio) ── same three tools, same backend
     │
     ▼
Langfuse (traces every call: input, output, tokens, cost)
```

## Stack

| Layer | Tech |
|---|---|
| Backend API | FastAPI, SQLAlchemy, SQLite |
| Agent protocol | MCP (Model Context Protocol) via `mcp` Python SDK |
| Semantic search / RAG | OpenAI embeddings (`text-embedding-3-small`) + Supabase/pgvector |
| Observability | Langfuse (traces, token usage, cost tracking) |
| Automation | n8n (self-hosted via Docker), Slack Events API |
| Local tunnel | ngrok (with traffic policy to bypass free-tier interstitial) |

## Tools exposed

### `get_order_status(order_id: int)`
Looks up an order's status and tracking number. Returns a clear "not found" message for invalid IDs rather than guessing.

### `search_docs(query: str)`
Semantic search over a knowledge base (shipping policy, returns, order tracking docs). Matches paraphrased queries — e.g. *"can I get my money back"* correctly retrieves the Return Policy doc with zero keyword overlap — because it embeds and compares meaning, not text.

### `create_ticket(customer_name: str, issue: str)`
Creates a support ticket for issues the agent can't resolve automatically, so nothing falls through the cracks.

## Evals

A small automated eval suite (`app/run_evals.py`) runs 10 test cases across all three tools, including:
- Correct order lookups and a graceful "no order found" for invalid IDs
- Semantic search matching paraphrased queries with **no literal keyword overlap** to the source doc
- Correctly saying "no relevant documents found" for off-topic queries, instead of hallucinating an answer
- Ticket creation

Current pass rate: **10/10 (100%)**. Every eval run is also traced in Langfuse, so pass/fail results are backed by inspectable, real execution data — not just a claimed number.

## Running it locally

### 1. Backend
```bash
python -m venv venv
venv\Scripts\activate          # Windows
pip install -r requirements.txt
uvicorn app.main:app --reload
```

### 2. Seed sample data
```bash
python -m app.seed
```

### 3. Set up RAG
- Create a Supabase project, enable the `pgvector` extension
- Run the SQL in `sql/setup.sql` to create the `doc_embeddings` table and `match_docs` function
- Add your keys to `.env` (see `.env.example`)
- Embed and upload the seeded docs:
  ```bash
  python -m app.embed_docs
  ```

### 4. Run the MCP server (for Claude Desktop)
```bash
python -m app.mcp_server
```
Add it to `claude_desktop_config.json` pointing at your venv's Python and this script.

### 5. Run evals
```bash
python -m app.run_evals
```

### 6. n8n + Slack automation
- `docker run -d --name n8n -p 5678:5678 -v n8n_data:/home/node/.n8n docker.n8n.io/n8nio/n8n`
- Import the workflow from `n8n/support-agent-workflow.json`
- Point a Slack app's Event Subscriptions at your n8n webhook (via ngrok for local dev)

## Environment variables

```
OPENAI_API_KEY=
SUPABASE_URL=
SUPABASE_KEY=
LANGFUSE_PUBLIC_KEY=
LANGFUSE_SECRET_KEY=
LANGFUSE_HOST=
```

## Why this project

This project demonstrates the parts of production AI work that matter most to a client evaluating an AI engineer:

- **Real backend integration** via MCP, not just prompt engineering
- **Grounded answers** via real semantic search, not keyword matching dressed up as "AI"
- **Proof it's not hallucinating** via automated evals and full observability - not just a demo that happened to work once
- **Deployable into existing tools** (Slack, and by extension any CRM/helpdesk) via automation, not confined to a chat window

The architecture here - FastAPI backend + MCP + RAG + evals + automation — is directly reusable: swap the `orders`/`tickets`/`docs` tables for a real client's CRM or helpdesk data model, and this becomes a client project rather than a from-scratch build.