# AI Ticket Triage Agent — LangGraph + MCP

[![CI](https://github.com/Mohemed-Amine-Chalhy/ai-ticket-triage/actions/workflows/ci.yml/badge.svg)](https://github.com/Mohemed-Amine-Chalhy/ai-ticket-triage/actions/workflows/ci.yml)

A production-shaped support workflow that classifies messy requests, extracts evidence from
PDF attachments, calls two internal systems through MCP, drafts a grounded reply, and routes
uncertain cases to a human instead of guessing.

## Evaluation scorecard

| Stage | Result |
|---|---:|
| Classification accuracy | **100%** (20/20) |
| Field extraction F1 | **100%** |
| Draft policy checks | **100%** |
| Deliberately unanswerable cases escalated | **100%** (5/5) |
| Case-specific escalation reasons | **100%** (5/5) |
| False escalation rate | **0%** (0/15) |
| Runtime error rate | **0%** |
| Offline latency | **4.6 ms p50 / 6.7 ms p95** |

These are reproducible results from the committed synthetic corpus, measured on a local Windows
development machine. Latency varies by hardware; the evaluator reports every per-case result in
[`artifacts/scorecard.json`](artifacts/scorecard.json). The five hard cases cover missing evidence,
conflicting identifiers, an unreadable attachment, an ambiguous request, and a record absent from
the internal system. The artifact also records its generation time, corpus hash, Python version,
commit identifier, and tool transport so stale results are visible.

![System architecture: email and PDF enter a LangGraph workflow, two MCP systems provide evidence, and a confidence gate branches to either a draft or a human queue.](assets/architecture.svg)

## Why this project exists

Most agent demos show only the happy path. This one makes abstention a tested behavior.
The agent can return one of two bounded outcomes:

- `drafted` — required identifiers were extracted, both read-only MCP checks completed, and the
  supplied references were verified.
- `escalated` — confidence or evidence failed policy, so the agent emits a non-committal holding
  response, a human queue, the missing evidence, and an auditable reason.

That decision is not hidden in a prompt. It is an explicit conditional edge in the LangGraph
state machine and a metric in CI.

## What it does

```text
Email + PDF
    │
    ▼
classify ──► extract ──► intake safety gate
                              │
                    unsafe ───┴─── safe
                       │              │
                       ▼              ▼
                  human queue    MCP tool 1: customer account
                                      │
                                 MCP tool 2: billing / incident
                                      │
                                post-tool safety gate
                                  │              │
                             unverified       verified
                                  │              │
                                  ▼              ▼
                             human queue   grounded draft
```

The two MCP tools are deliberately narrow and read-only:

1. `lookup_customer_account` performs an exact account/email match.
2. `lookup_billing_or_incident` checks billing, service incident, or bounded support context.

The graph always uses one transport-neutral MCP tool contract. Offline evaluation uses the fast
in-process adapter; Docker Compose runs the portfolio UI against a persistent, real
JSON-RPC-over-stdio MCP server. Both transports are integration-tested, so orchestration never
depends on the deployment choice.

## Run it locally

Prerequisites: Python 3.11–3.13 and [uv](https://docs.astral.sh/uv/).

```bash
git clone https://github.com/Mohemed-Amine-Chalhy/ai-ticket-triage.git
cd ai-ticket-triage
uv sync --extra dev --locked
uv run uvicorn ai_ticket_triage.web:app --reload
```

Open <http://127.0.0.1:8000>. The web UI includes all 20 labelled examples, a PDF uploader,
the graph trace, extracted fields, MCP call evidence, the final decision, and the scorecard.

The command above uses the fast in-process adapter. To run the exact UI shown in the MCP demo,
start the locked container instead; Compose enables the persistent stdio server by default:

```bash
docker compose up --build
```

Regenerate all four portfolio proof images from the current scorecard and an actual verbose test
run:

```bash
make proof
```

No API key is required. All names, emails, accounts, invoices, services, and incidents are fake;
emails use the reserved `example.test` domain.

### CLI demo

Run an answerable fixture:

```bash
uv run ticket-triage triage --case billing_duplicate_charge
```

Run a failure case and inspect the human handoff:

```bash
uv run ticket-triage triage --case failure_unreadable_attachment
```

Run a real PDF:

```bash
uv run ticket-triage triage \
  --text "I was charged twice; details are attached." \
  --pdf data/sample_attachments/duplicate-charge.pdf
```

Exercise the actual stdio MCP boundary:

```bash
uv run ticket-triage triage \
  --case billing_duplicate_charge \
  --transport stdio
```

### Reproduce the scorecard

```bash
uv run ticket-triage-eval \
  --output artifacts/scorecard.json \
  --markdown-output artifacts/scorecard.md \
  --fail-on-runtime-error \
  --enforce-portfolio-targets
```

The evaluator scores each stage independently: exact category match, micro field-level F1,
declarative draft checks, semantic handoff-reason grounding, escalation precision/recall, false
escalations, runtime failures, and p50/p95/max latency. See [Evaluation methodology](docs/EVALUATION.md).

## Use the MCP server independently

Start the bundled official-SDK server over stdio:

```bash
uv run ticket-triage-mcp
```

Example configuration for a local stdio MCP host:

```json
{
  "mcpServers": {
    "ticket-triage-tools": {
      "command": "uv",
      "args": [
        "--directory",
        "/absolute/path/to/ai-ticket-triage",
        "run",
        "ticket-triage-mcp"
      ]
    }
  }
}
```

This is a transport-neutral tool boundary: another compatible agent or desktop host can use the
same two contracts without importing the LangGraph application. For remote hosts, put the server
behind an authenticated Streamable HTTP deployment; the portfolio demo intentionally exposes only
local stdio and in-process transports.

## Engineering choices

| Concern | Implementation |
|---|---|
| Orchestration | Compiled `StateGraph` with typed state and explicit conditional edges |
| Safety | Two policy gates; low confidence, conflicts, missing evidence, unreadable files, tool failures, and misses all escalate |
| Documents | `pypdf` extraction, strict PDF upload validation, size limits, and extraction warnings |
| Tool boundary | Official MCP Python SDK, exactly two read-only tools, normalized error envelopes, timeouts |
| Contracts | Pydantic models with forbidden extra fields and JSON-safe public results |
| Evaluation | 20 versioned JSON labels, per-stage metrics, case diagnostics, runtime-error capture |
| API | FastAPI, generated OpenAPI docs, upload limits, request IDs, safe error responses, security headers |
| Operations | Locked dependencies, Docker health check, structured logs, CI lint/type/test/coverage gates |
| Privacy | Synthetic fixtures only; raw PDF bytes are excluded from model serialization |

### Deterministic by design

The default classifier, extractor, and draft composer are deterministic. That makes safety
regressions reproducible, keeps the public demo credential-free, and separates workflow quality
from model variance. A hosted model can replace those nodes behind the same typed contracts; in a
real rollout, its candidate outputs should still pass through the same evidence and tool gates.
This repository does **not** claim that a 20-case synthetic benchmark predicts live-data quality.

## Repository map

```text
src/ai_ticket_triage/
├── agent.py          # LangGraph state machine and tool orchestration
├── classifier.py     # deterministic category scoring with evidence
├── extractor.py      # PDF/text extraction and conflict detection
├── confidence.py     # bounded-failure policy gates
├── drafting.py       # grounded replies and safe holding responses
├── mcp_server.py     # official MCP server; exactly two tools
├── mcp_client.py     # in-process and real stdio MCP gateways
├── internal_api.py   # mock read-only service adapters
├── evaluation.py     # corpus runner and scorecard metrics
├── web.py            # FastAPI application
└── static/           # responsive portfolio UI
data/cases/           # 20 synthetic labelled fixtures
tests/                # unit, API, workflow, evaluator, and MCP integration tests
artifacts/            # committed scorecard and proof outputs
assets/               # portfolio-ready architecture and result images
docs/                 # architecture, evaluation, security, runbook, portfolio copy
```

## Quality commands

```bash
uv run ruff check .
uv run ruff format --check .
uv run mypy src
uv run pytest --cov=ai_ticket_triage --cov-report=term-missing
uv run ticket-triage-eval --fail-on-runtime-error --enforce-portfolio-targets
docker compose up --build
```

## Documentation

- [Architecture and decision flow](docs/ARCHITECTURE.md)
- [Evaluation methodology and case design](docs/EVALUATION.md)
- [Security and trust boundaries](docs/SECURITY.md)
- [Operations runbook](docs/RUNBOOK.md)
- [Portfolio listing, proof checklist, and 55-second script](docs/PORTFOLIO.md)

## Known limits

- Text-based PDFs only; scanned documents need OCR and a malware-scanning pipeline.
- Synthetic exact-match internal systems, not a live CRM or billing platform.
- English fixtures and a four-class taxonomy.
- No durable queue, authentication, rate limiting, or distributed tracing in this local demo.
- Deterministic language logic is a reliability baseline, not a substitute for evaluation on a
  representative, privacy-reviewed production dataset.

Those omissions are intentional weekend-project boundaries. The interfaces isolate each missing
production concern so it can be added without rewriting the graph.

## License

[MIT](LICENSE)