Support Triage Agent with MCP

An AI triage desk for a home-loan partner-support team. It classifies each ticket by category, priority and sentiment, drafts a first reply, flags duplicates, and answers questions about the queue. The same tools are exposed two ways: as an MCP server (any MCP client) and as a web dashboard (FastAPI + plain HTML/CSS/JS).

How it works

 MCP client ──┐                                    ┌─ embeddings ─ vector store (FAISS or NumPy)
              ├─► server.py (tools) ─► retrieval ──┤   similar resolved tickets, duplicate check
 Browser ─► app.py (REST) ─┘              │        └─ tickets.json
                                          └─► triage.py ─► llm.py ─► Groq (openai/gpt-oss-20b) or Ollama

Classification retrieves the 3 most similar resolved, labeled tickets and shows them to the model as reference examples. Duplicate detection flags a ticket when it shares a loan/partner ID with related wording, comes from the same named sender with related wording, or is a near-copy from an anonymous sender. Two named people with different IDs are never duplicates, however similar the text.

Related MCP server: TicketDesk MCP

Run it

pip install -r requirements.txt            # core (hashing embeddings, NumPy store)
pip install -r requirements-ml.txt         # optional: real embeddings (fastembed) + FAISS

export GROQ_API_KEY=...                    # Windows PowerShell: $env:GROQ_API_KEY="..."
python app.py                              # dashboard on http://localhost:8000
python server.py                           # MCP server over stdio (e.g. for Claude Desktop)

GET /api/health (and the header plate in the UI) reports which LLM, embedding model and vector store are active, so you can tell whether you are on real embeddings or the fallback.

Evaluation

python evaluate.py classify --dataset synthetic            # 240 labeled home-loan tickets
python evaluate.py classify --dataset public               # 200 real bank queries
python evaluate.py classify --dataset public --mode zero-shot
python evaluate.py classify --dataset public --mode baseline   # no LLM: nearest-neighbour vote, free
python evaluate.py classify --dataset synthetic --judge --judge-model openai/gpt-oss-120b
python evaluate.py duplicates

Every report gives n, accuracy with a 95% Wilson interval, macro-F1, weakest categories and top confusions. LLM failures (rate limits, outages) are excluded and reported separately, so an outage cannot change the score. Unparseable model output counts as wrong.

Datasets

• synthetic (data/generate_data.py): 240 template-generated tickets. Subjects never contain the category name, about 15% are noisy (lower-cased, typos), about 10% mention a second problem, priorities depend on urgency cues, and 24 are labeled duplicates plus 16 same-loan/different-issue hard negatives. It is still template-based, so it is the easier of the two sets.

• public (data/fetch_public_dataset.py): 200 held-out test queries from Banking77 (PolyAI, CC BY 4.0; Casanueva et al. 2020), with a disjoint 616-query retrieval pool. Its 77 intents are grouped into 9 routing queues by me, so quote results as "Banking77 grouped into 9 queues", not as a Banking77 benchmark. It has no priority labels.

Reference numbers without an LLM (kNN baseline, hashing embedder; reproduced in CI on every push):

On the synthetic set retrieval alone is already at 94.6%, so a high LLM score there says little. Use public, and compare rag against zero-shot and baseline to see what the model and the retrieval each contribute.

LLM results (fill in after running; do not quote numbers you have not measured):

Calibrating duplicates. Similarity scores differ a lot between embedders, so the threshold must be tuned for the one you use. The hashing fallback was measured: threshold 0.25 gives recall 1.00 and precision 0.55 (it cannot tell "same loan, same issue, reworded" from "same loan, different issue"). The fastembed default of 0.88 is an uncalibrated guess: run python evaluate.py duplicates with requirements-ml.txt installed and set DUPLICATE_THRESHOLD to the value it recommends.

Deploy on Render

Build: pip install -r requirements.txt (add -r requirements-ml.txt for real embeddings). Start: uvicorn app:app --host 0.0.0.0 --port $PORT. Set GROQ_API_KEY as a secret.

• Memory. The free tier has 512 MB. fastembed plus ONNX runtime may not fit; if the service restarts or is killed, set EMBEDDING_BACKEND=hashing. Check /api/health after deploy to see which backend loaded.

• Storage is ephemeral. Cases filed in the UI and status changes live in data/tickets.json and reset on restart or redeploy. A persistent disk or a database is the next step if that matters.

• First start downloads the embedding model (about 130 MB) before the service becomes ready.

Tests and CI

pip install -r requirements.txt -r requirements-dev.txt
pytest          # 119 tests, no API key or model download needed
ruff check .

GitHub Actions runs lint and tests on Python 3.11 and 3.12 (with FAISS), then the LLM-free evaluations. Tests mock the LLM, so they check the code, not the model; model quality is what evaluate.py measures. Dataset tests also guard against label leakage in ticket subjects, and floor tests fail if retrieval quality drops.

Notes and limitations

• A FAISS index crashes the process if one thread adds while another searches, and the web server uses a thread pool. VectorStore takes a lock on every operation, and tests/test_index.py has a regression test for it.

• Seeded tickets keep their ground-truth labels: "Run AI triage" on them only shows a prediction. Tickets filed in the UI are the ones whose category and priority get set.

• Ticket text is untrusted: it is HTML-escaped in the UI and the prompts tell the model to ignore instructions inside it. This reduces, but does not eliminate, prompt-injection risk.

• Groq is a third-party API, so ticket text leaves your infrastructure. For data that must stay on-prem, use LLM_PROVIDER=ollama.

• The in-memory rate limiter is per process, which is fine for one instance only.

License

MIT. Banking77 data is CC BY 4.0 (see above).