https://github.com/vishwavardhan33-arch/Support-Triage-Agent-with-MCP/actions/workflows/ci.yml

Support Triage MCP

An MCP (Model Context Protocol) server that exposes a support-ticket inbox as tools an AI client can use to list, inspect, triage, and update tickets. The dataset here is mocked (styled after a home-loan partner-support inbox: loan status queries, document issues, disbursement delays, KYC, etc.) but the server is written so a real ticket source (Zendesk, email, a helpdesk API) could be swapped in later without changing the tool interface.

Status

Core plumbing (list_tickets, get_ticket, update_ticket_status) and LLM-based triage (classify_ticket, with RAG-style retrieval of similar past tickets) are both working. classify_ticket runs entirely on a local, open-source LLM via Ollama — no API key, no per-call cost. See Roadmap for what's next.

Setting up Ollama (needed for classify_ticket)

1. Install Ollama from ollama.com (Windows/Mac/Linux).

2. Pull a model (one-time download, then it runs fully offline):

   ollama pull llama3.2

   llama3.2 (3B) is small and fast enough for CPU-only laptops. For better classification quality if your machine can handle it, try ollama pull llama3.1 (8B) or ollama pull mistral — just set OLLAMA_MODEL to match (see below).

3. Ollama runs a local server automatically after install. If it's not running, start it with:

   ollama serve

By default the server talks to http://localhost:11434 and uses the llama3.2 model. Override either with environment variables if needed:

# macOS/Linux
export OLLAMA_MODEL="mistral"

# Windows (PowerShell)
$env:OLLAMA_MODEL="mistral"

list_tickets, get_ticket, and update_ticket_status don't need Ollama at all — only classify_ticket does.

Related MCP server: rebar-mcp

Evaluation

evaluate.py measures two things against the mock dataset:

1. Routing accuracy — how often classify_ticket's predicted category and priority match the dataset's ground-truth labels (baked in by generate_data.py).

2. LLM-as-judge — there's no ground truth for the quality of the drafted suggested_response, so a second local model scores it on relevance, professionalism, and actionability (1-5 each).

# Quick run: 10 tickets, routing accuracy + judge scores
python evaluate.py --limit 10

# Use a different, larger model as judge to reduce self-judging bias
python evaluate.py --limit 10 --judge-model mistral

# Routing accuracy only, no judge step (faster)
python evaluate.py --limit 10 --no-judge

Each ticket costs 1-2 local LLM calls, so on a CPU-only laptop start with --limit 5-10 before running the full 40-ticket dataset — otherwise it can take a while (see Status for rough per-call timing). Full per-ticket results are written to eval_results.json (git-ignored, since it's a run artifact, not source).

Note on self-judging bias: if the same model both classifies and judges, it tends to rate its own drafts generously. Passing --judge-model with a different (ideally larger) model gives a more honest signal.

Web UI

On top of the MCP server, app.py wraps the same tool functions as a small FastAPI REST API and serves a browser-based dashboard from static/ — so the project looks and works like a real internal support tool, not just a script.

python app.py

Then open http://localhost:8000. Filter tickets by status/category in the left rail, click a case to open its detail panel, update its status, or click Run AI triage to call classify_ticket (via Ollama) and see the predicted category, priority, sentiment, and a copyable drafted response. The dot next to the header shows whether Ollama is currently reachable.

This is a separate, optional layer — the MCP server and its tools work identically with or without the web UI running.

Project structure

support-triage-mcp/
├── .github/
│   └── workflows/
│       └── ci.yml         # GitHub Actions: lint + pytest on every push/PR
├── app.py                 # FastAPI backend for the web UI (wraps the same tool functions)
├── static/                # Web UI frontend (HTML/CSS/JS, served by app.py)
├── server.py              # MCP server + tool definitions
├── evaluate.py            # routing-accuracy + LLM-as-judge evaluation harness
├── retrieval.py           # TF-IDF similar-ticket retrieval (RAG-style)
├── data/
│   ├── tickets.json       # mock ticket dataset
│   └── generate_data.py   # regenerates tickets.json
├── tests/
│   └── test_server.py     # smoke tests for the tool functions
├── requirements.txt
├── LICENSE
└── README.md

Setup

git clone <your-repo-url>
cd support-triage-mcp
python3 -m venv .venv
source .venv/bin/activate        # Windows: .venv\Scripts\activate
pip install -r requirements.txt

Regenerate the mock dataset any time with:

python data/generate_data.py

Run the test suite:

pip install -r requirements-dev.txt
pytest tests/ -v

Lint (same check CI runs):

ruff check .

Running the server standalone

python server.py

This starts the MCP server over stdio, waiting for a client to connect.

Connecting to Claude Desktop

Add this to your Claude Desktop MCP config (claude_desktop_config.json — Settings → Developer → Edit Config):

{
  "mcpServers": {
    "support-triage": {
      "command": "python",
      "args": ["/absolute/path/to/support-triage-mcp/server.py"]
    }
  }
}

Restart Claude Desktop, then try prompts like:

• "List all open support tickets"

• "Show me ticket T-1021"

• "Mark T-1021 as in_progress"

• "Classify ticket T-1021 and draft a first response"

Available tools

Roadmap

• Swap TF-IDF retrieval for real embeddings (a local embedding model, e.g. via ollama pull nomic-embed-text)

• Swap the mock dataset for a real ticket source

• Try larger local models (llama3.1, mistral) and compare routing accuracy vs. llama3.2

Continuous Integration

Every push and PR to main runs .github/workflows/ci.yml, which lints with ruff and runs the full pytest suite on Python 3.11 and 3.12. No Ollama install is needed in CI — classify_ticket's LLM call is mocked in tests/test_classify.py, so the suite runs fully offline. You'll see a status badge/checkmark on each commit and PR once this is pushed.

Notes on the MCP SDK version

This project uses mcp v2.x, where the server class was renamed from FastMCP to MCPServer (mcp.server.mcpserver.MCPServer). The decorator API (@mcp.tool()) and mcp.run() are unchanged from v1. If you're following older MCP tutorials that import from mcp.server.fastmcp import FastMCP, either update the import as above or pin mcp<2.0.0.