Skip to main content
Glama
Mohemed-Amine-Chalhy

ticket-triage-mcp

AI Ticket Triage Agent — LangGraph + MCP

CI

A production-shaped support workflow that classifies messy requests, extracts evidence from PDF attachments, calls two internal systems through MCP, drafts a grounded reply, and routes uncertain cases to a human instead of guessing.

Evaluation scorecard

Stage

Result

Classification accuracy

100% (20/20)

Field extraction F1

100%

Draft policy checks

100%

Deliberately unanswerable cases escalated

100% (5/5)

Case-specific escalation reasons

100% (5/5)

False escalation rate

0% (0/15)

Runtime error rate

0%

Offline latency

4.6 ms p50 / 6.7 ms p95

These are reproducible results from the committed synthetic corpus, measured on a local Windows development machine. Latency varies by hardware; the evaluator reports every per-case result in artifacts/scorecard.json. The five hard cases cover missing evidence, conflicting identifiers, an unreadable attachment, an ambiguous request, and a record absent from the internal system. The artifact also records its generation time, corpus hash, Python version, commit identifier, and tool transport so stale results are visible.

System architecture: email and PDF enter a LangGraph workflow, two MCP systems provide evidence, and a confidence gate branches to either a draft or a human queue.

Related MCP server: reach-dispute-mcp

Why this project exists

Most agent demos show only the happy path. This one makes abstention a tested behavior. The agent can return one of two bounded outcomes:

  • drafted — required identifiers were extracted, both read-only MCP checks completed, and the supplied references were verified.

  • escalated — confidence or evidence failed policy, so the agent emits a non-committal holding response, a human queue, the missing evidence, and an auditable reason.

That decision is not hidden in a prompt. It is an explicit conditional edge in the LangGraph state machine and a metric in CI.

What it does

Email + PDF
    │
    ▼
classify ──► extract ──► intake safety gate
                              │
                    unsafe ───┴─── safe
                       │              │
                       ▼              ▼
                  human queue    MCP tool 1: customer account
                                      │
                                 MCP tool 2: billing / incident
                                      │
                                post-tool safety gate
                                  │              │
                             unverified       verified
                                  │              │
                                  ▼              ▼
                             human queue   grounded draft

The two MCP tools are deliberately narrow and read-only:

  1. lookup_customer_account performs an exact account/email match.

  2. lookup_billing_or_incident checks billing, service incident, or bounded support context.

The graph always uses one transport-neutral MCP tool contract. Offline evaluation uses the fast in-process adapter; Docker Compose runs the portfolio UI against a persistent, real JSON-RPC-over-stdio MCP server. Both transports are integration-tested, so orchestration never depends on the deployment choice.

Run it locally

Prerequisites: Python 3.11–3.13 and uv.

git clone https://github.com/Mohemed-Amine-Chalhy/ai-ticket-triage.git
cd ai-ticket-triage
uv sync --extra dev --locked
uv run uvicorn ai_ticket_triage.web:app --reload

Open http://127.0.0.1:8000. The web UI includes all 20 labelled examples, a PDF uploader, the graph trace, extracted fields, MCP call evidence, the final decision, and the scorecard.

The command above uses the fast in-process adapter. To run the exact UI shown in the MCP demo, start the locked container instead; Compose enables the persistent stdio server by default:

docker compose up --build

Regenerate all four portfolio proof images from the current scorecard and an actual verbose test run:

make proof

No API key is required. All names, emails, accounts, invoices, services, and incidents are fake; emails use the reserved example.test domain.

CLI demo

Run an answerable fixture:

uv run ticket-triage triage --case billing_duplicate_charge

Run a failure case and inspect the human handoff:

uv run ticket-triage triage --case failure_unreadable_attachment

Run a real PDF:

uv run ticket-triage triage \
  --text "I was charged twice; details are attached." \
  --pdf data/sample_attachments/duplicate-charge.pdf

Exercise the actual stdio MCP boundary:

uv run ticket-triage triage \
  --case billing_duplicate_charge \
  --transport stdio

Reproduce the scorecard

uv run ticket-triage-eval \
  --output artifacts/scorecard.json \
  --markdown-output artifacts/scorecard.md \
  --fail-on-runtime-error \
  --enforce-portfolio-targets

The evaluator scores each stage independently: exact category match, micro field-level F1, declarative draft checks, semantic handoff-reason grounding, escalation precision/recall, false escalations, runtime failures, and p50/p95/max latency. See Evaluation methodology.

Use the MCP server independently

Start the bundled official-SDK server over stdio:

uv run ticket-triage-mcp

Example configuration for a local stdio MCP host:

{
  "mcpServers": {
    "ticket-triage-tools": {
      "command": "uv",
      "args": [
        "--directory",
        "/absolute/path/to/ai-ticket-triage",
        "run",
        "ticket-triage-mcp"
      ]
    }
  }
}

This is a transport-neutral tool boundary: another compatible agent or desktop host can use the same two contracts without importing the LangGraph application. For remote hosts, put the server behind an authenticated Streamable HTTP deployment; the portfolio demo intentionally exposes only local stdio and in-process transports.

Engineering choices

Concern

Implementation

Orchestration

Compiled StateGraph with typed state and explicit conditional edges

Safety

Two policy gates; low confidence, conflicts, missing evidence, unreadable files, tool failures, and misses all escalate

Documents

pypdf extraction, strict PDF upload validation, size limits, and extraction warnings

Tool boundary

Official MCP Python SDK, exactly two read-only tools, normalized error envelopes, timeouts

Contracts

Pydantic models with forbidden extra fields and JSON-safe public results

Evaluation

20 versioned JSON labels, per-stage metrics, case diagnostics, runtime-error capture

API

FastAPI, generated OpenAPI docs, upload limits, request IDs, safe error responses, security headers

Operations

Locked dependencies, Docker health check, structured logs, CI lint/type/test/coverage gates

Privacy

Synthetic fixtures only; raw PDF bytes are excluded from model serialization

Deterministic by design

The default classifier, extractor, and draft composer are deterministic. That makes safety regressions reproducible, keeps the public demo credential-free, and separates workflow quality from model variance. A hosted model can replace those nodes behind the same typed contracts; in a real rollout, its candidate outputs should still pass through the same evidence and tool gates. This repository does not claim that a 20-case synthetic benchmark predicts live-data quality.

Repository map

src/ai_ticket_triage/
├── agent.py          # LangGraph state machine and tool orchestration
├── classifier.py     # deterministic category scoring with evidence
├── extractor.py      # PDF/text extraction and conflict detection
├── confidence.py     # bounded-failure policy gates
├── drafting.py       # grounded replies and safe holding responses
├── mcp_server.py     # official MCP server; exactly two tools
├── mcp_client.py     # in-process and real stdio MCP gateways
├── internal_api.py   # mock read-only service adapters
├── evaluation.py     # corpus runner and scorecard metrics
├── web.py            # FastAPI application
└── static/           # responsive portfolio UI
data/cases/           # 20 synthetic labelled fixtures
tests/                # unit, API, workflow, evaluator, and MCP integration tests
artifacts/            # committed scorecard and proof outputs
assets/               # portfolio-ready architecture and result images
docs/                 # architecture, evaluation, security, runbook, portfolio copy

Quality commands

uv run ruff check .
uv run ruff format --check .
uv run mypy src
uv run pytest --cov=ai_ticket_triage --cov-report=term-missing
uv run ticket-triage-eval --fail-on-runtime-error --enforce-portfolio-targets
docker compose up --build

Documentation

Known limits

  • Text-based PDFs only; scanned documents need OCR and a malware-scanning pipeline.

  • Synthetic exact-match internal systems, not a live CRM or billing platform.

  • English fixtures and a four-class taxonomy.

  • No durable queue, authentication, rate limiting, or distributed tracing in this local demo.

  • Deterministic language logic is a reliability baseline, not a substitute for evaluation on a representative, privacy-reviewed production dataset.

Those omissions are intentional weekend-project boundaries. The interfaces isolate each missing production concern so it can be added without rewriting the graph.

License

MIT

Available Tools

2 tools
lookup_billing_or_incidentC

Read-only billing or service-incident lookup over synthetic data. Also returns bounded workflow guidance for account/other requests.

ParametersJSON Schema
NameRequiredDescriptionDefault
productNo
categoryYes
order_idNo
ticket_idNo
account_idNo
error_codeNo
invoice_idNo
service_idNo
incident_idNo
account_numberNo
invoice_numberNo

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

C2.7/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden of behavioral disclosure. It declares 'Read-only' and 'over synthetic data', which is useful, but it does not explain error handling, limits, what 'bounded workflow guidance' means, or how the output varies with input combinations. The behavioral description is minimal and undetailed.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, concise sentence that front-loads the core purpose ('Read-only billing or service-incident lookup') and then adds the supplementary workflow guidance. It is well-structured and avoids unnecessary verbosity, though it could be more detailed without losing conciseness.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the complexity of the schema (11 optional parameters, only 1 required) and the absence of annotations and parameter descriptions, the description is heavily underspecified. It does not explain how to select parameters based on the category, what 'bounded workflow guidance' entails, or what the output schema contains. An agent would face significant ambiguity in correctly forming a request.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must compensate for missing parameter documentation. It does not explain the meaning or usage of any of the 11 parameters, nor does it clarify which parameters are relevant for 'billing' vs 'service-incident' or how the 'category' field drives the lookup. The parameter names are self-explanatory, but the description adds no additional semantics.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a clear verb ('lookup') and resource ('billing or service-incident'), and also mentions additional workflow guidance. It implicitly distinguishes from the sibling `lookup_customer_account` by domain, though without naming it explicitly. The purpose is specific and understandable.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description does not provide any explicit guidance on when to use this tool versus alternatives, nor does it mention exclusions or conditions. The phrase 'for account/other requests' is vague and does not clarify selection criteria. No alternatives are referenced.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

lookup_customer_accountC

Read-only exact lookup in the synthetic customer account system. Use returned misses or mismatches as escalation evidence.

ParametersJSON Schema
NameRequiredDescriptionDefault
emailNo
ticket_idNo
account_idNo
account_numberNo

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

C2.9/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description must carry the full burden. It discloses the read-only nature and that it returns misses or mismatches, which is useful. However, it does not explain how multiple parameters combine, whether authentication is needed, or any other behavioral details beyond these basics.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is two concise sentences with the key fact 'Read-only exact lookup' front-loaded. Every clause earns its place, and there is no redundancy or filler.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a tool with 4 optional parameters, zero required, and no parameter documentation, the description is inadequate. It does not explain how to construct a valid lookup, what happens if multiple params are provided, or what escalation evidence looks like. The output schema exists but the description offers no context on call semantics.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters1/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must compensate for the lack of parameter documentation. It does not mention email, ticket_id, account_id, or account_number at all, nor does it explain whether at least one is required, how they interplay, or which is preferred. The agent has no guidance on parameter usage.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a clear verb and resource ('Read-only exact lookup in the synthetic customer account system'), which distinguishes it from the sibling tool by resource. However, it doesn't explicitly name the sibling, so differentiation is implied rather than explicit.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description gives no explicit guidance on when to use this tool versus the sibling lookup_billing_or_incident. The only usage-related hint is 'Use returned misses or mismatches as escalation evidence,' which concerns result handling, not tool selection. It lacks any exclusions or alternatives.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

  1. 2 tool updatesv1.0.0
    • First observedlookup_billing_or_incident
    • First observedlookup_customer_account

TDQS

B3/5.0

Scored across 2 tools

Disambiguation4/5

The two tools target distinct data domains (customer accounts vs. billing/incidents), and their descriptions clearly differentiate them. However, since both are lookups and the second tool's name is broader, there is slight potential for an agent to confuse them when deciding which to use.

Naming Consistency5/5

Both tools follow the same 'lookup_*' verb-noun pattern, with the resource type clearly indicated. This is perfectly consistent and predictable.

Tool Count3/5

With only two tools, the server feels thin for a 'triage' purpose, but the scope might be intentionally limited to read-only lookups. This is borderline on the low end of the range.

Completeness2/5

The tool surface consists solely of lookups and does not include any actions to actually triage or resolve tickets (e.g., update status, assign, escalate). Even if the second tool returns guidance, the lack of write or workflow-management tools leaves significant gaps for a triage-focused server.

Maintenance

ActivityMaintained
ResponsivenessNo issues

Related MCP Connectors

Related MCP Servers

  • F
    license
    Not graded
    quality
    C
    maintenance
    Provides read-only MCP tools to diagnose customer billing disputes by analyzing billing records, identifying contradictions, citing evidence, and scoring confidence to auto-resolve or escalate.
    -
  • F
    license
    Not graded
    quality
    B
    maintenance
    Provides MCP tools that let an agent retrieve customer account, product usage, interaction, and support summaries, and create follow-up tasks after user approval.
    -