AI Ticket Triage Agent — LangGraph + MCP

A production-shaped support workflow that classifies messy requests, extracts evidence from PDF attachments, calls two internal systems through MCP, drafts a grounded reply, and routes uncertain cases to a human instead of guessing.

Evaluation scorecard

These are reproducible results from the committed synthetic corpus, measured on a local Windows development machine. Latency varies by hardware; the evaluator reports every per-case result in artifacts/scorecard.json. The five hard cases cover missing evidence, conflicting identifiers, an unreadable attachment, an ambiguous request, and a record absent from the internal system. The artifact also records its generation time, corpus hash, Python version, commit identifier, and tool transport so stale results are visible.

Why this project exists

Most agent demos show only the happy path. This one makes abstention a tested behavior. The agent can return one of two bounded outcomes:

• drafted — required identifiers were extracted, both read-only MCP checks completed, and the supplied references were verified.

• escalated — confidence or evidence failed policy, so the agent emits a non-committal holding response, a human queue, the missing evidence, and an auditable reason.

That decision is not hidden in a prompt. It is an explicit conditional edge in the LangGraph state machine and a metric in CI.

What it does

Email + PDF
    │
    ▼
classify ──► extract ──► intake safety gate
                              │
                    unsafe ───┴─── safe
                       │              │
                       ▼              ▼
                  human queue    MCP tool 1: customer account
                                      │
                                 MCP tool 2: billing / incident
                                      │
                                post-tool safety gate
                                  │              │
                             unverified       verified
                                  │              │
                                  ▼              ▼
                             human queue   grounded draft

The two MCP tools are deliberately narrow and read-only:

1. lookup_customer_account performs an exact account/email match.

2. lookup_billing_or_incident checks billing, service incident, or bounded support context.

The graph always uses one transport-neutral MCP tool contract. Offline evaluation uses the fast in-process adapter; Docker Compose runs the portfolio UI against a persistent, real JSON-RPC-over-stdio MCP server. Both transports are integration-tested, so orchestration never depends on the deployment choice.

Run it locally

Prerequisites: Python 3.11–3.13 and uv.

git clone https://github.com/Mohemed-Amine-Chalhy/ai-ticket-triage.git
cd ai-ticket-triage
uv sync --extra dev --locked
uv run uvicorn ai_ticket_triage.web:app --reload

Open http://127.0.0.1:8000. The web UI includes all 20 labelled examples, a PDF uploader, the graph trace, extracted fields, MCP call evidence, the final decision, and the scorecard.

The command above uses the fast in-process adapter. To run the exact UI shown in the MCP demo, start the locked container instead; Compose enables the persistent stdio server by default:

docker compose up --build

Regenerate all four portfolio proof images from the current scorecard and an actual verbose test run:

make proof

No API key is required. All names, emails, accounts, invoices, services, and incidents are fake; emails use the reserved example.test domain.

CLI demo

Run an answerable fixture:

uv run ticket-triage triage --case billing_duplicate_charge

Run a failure case and inspect the human handoff:

uv run ticket-triage triage --case failure_unreadable_attachment

Run a real PDF:

uv run ticket-triage triage \
  --text "I was charged twice; details are attached." \
  --pdf data/sample_attachments/duplicate-charge.pdf

Exercise the actual stdio MCP boundary:

uv run ticket-triage triage \
  --case billing_duplicate_charge \
  --transport stdio

Reproduce the scorecard

uv run ticket-triage-eval \
  --output artifacts/scorecard.json \
  --markdown-output artifacts/scorecard.md \
  --fail-on-runtime-error \
  --enforce-portfolio-targets

The evaluator scores each stage independently: exact category match, micro field-level F1, declarative draft checks, semantic handoff-reason grounding, escalation precision/recall, false escalations, runtime failures, and p50/p95/max latency. See Evaluation methodology.

Use the MCP server independently

Start the bundled official-SDK server over stdio:

uv run ticket-triage-mcp

Example configuration for a local stdio MCP host:

{
  "mcpServers": {
    "ticket-triage-tools": {
      "command": "uv",
      "args": [
        "--directory",
        "/absolute/path/to/ai-ticket-triage",
        "run",
        "ticket-triage-mcp"
      ]
    }
  }
}

This is a transport-neutral tool boundary: another compatible agent or desktop host can use the same two contracts without importing the LangGraph application. For remote hosts, put the server behind an authenticated Streamable HTTP deployment; the portfolio demo intentionally exposes only local stdio and in-process transports.

Engineering choices

Deterministic by design

The default classifier, extractor, and draft composer are deterministic. That makes safety regressions reproducible, keeps the public demo credential-free, and separates workflow quality from model variance. A hosted model can replace those nodes behind the same typed contracts; in a real rollout, its candidate outputs should still pass through the same evidence and tool gates. This repository does not claim that a 20-case synthetic benchmark predicts live-data quality.

Repository map

src/ai_ticket_triage/
├── agent.py          # LangGraph state machine and tool orchestration
├── classifier.py     # deterministic category scoring with evidence
├── extractor.py      # PDF/text extraction and conflict detection
├── confidence.py     # bounded-failure policy gates
├── drafting.py       # grounded replies and safe holding responses
├── mcp_server.py     # official MCP server; exactly two tools
├── mcp_client.py     # in-process and real stdio MCP gateways
├── internal_api.py   # mock read-only service adapters
├── evaluation.py     # corpus runner and scorecard metrics
├── web.py            # FastAPI application
└── static/           # responsive portfolio UI
data/cases/           # 20 synthetic labelled fixtures
tests/                # unit, API, workflow, evaluator, and MCP integration tests
artifacts/            # committed scorecard and proof outputs
assets/               # portfolio-ready architecture and result images
docs/                 # architecture, evaluation, security, runbook, portfolio copy

Quality commands

uv run ruff check .
uv run ruff format --check .
uv run mypy src
uv run pytest --cov=ai_ticket_triage --cov-report=term-missing
uv run ticket-triage-eval --fail-on-runtime-error --enforce-portfolio-targets
docker compose up --build

Documentation

• Architecture and decision flow

• Evaluation methodology and case design

• Security and trust boundaries

• Operations runbook

• Portfolio listing, proof checklist, and 55-second script

Known limits

• Text-based PDFs only; scanned documents need OCR and a malware-scanning pipeline.

• Synthetic exact-match internal systems, not a live CRM or billing platform.

• English fixtures and a four-class taxonomy.

• No durable queue, authentication, rate limiting, or distributed tracing in this local demo.

• Deterministic language logic is a reliability baseline, not a substitute for evaluation on a representative, privacy-reviewed production dataset.

Those omissions are intentional weekend-project boundaries. The interfaces isolate each missing production concern so it can be added without rewriting the graph.

License

MIT