integration-triage

A support-engineer toolkit. Give it a customer ticket and the request logs, and it names the root cause, cites the failing request IDs, drafts the customer reply, writes the engineering escalation packet, and keeps a runbook for next time. Secrets are redacted on the way out. It also runs as an MCP server, so an agent can triage a ticket itself.

Unofficial. Not affiliated with Merge or any other vendor. It is built from public documentation and synthetic logs. No real customer data and no vendor internals are used.

flowchart LR
    T[ticket text] --> D[diagnose]
    L[request logs JSONL] --> D
    D --> R[customer reply]
    D --> E[escalation packet<br/>evidence table, severity,<br/>page on-call?]
    D --> B[runbook entry]
    R --> X[redact secrets]
    E --> X

Why this exists

A support engineer's loop is the same every time: find the failing calls, name the cause, tell the customer what to do, and hand engineering something they never have to ask a follow-up about. Most of the cost is in the details: "Your API is down!" turns out to be a rate limit, a 404 is really a key scoped to the wrong user, a customer pasted a live key into the ticket. This tool encodes that loop and tests it.

Related MCP server: reqlog

Quickstart (no keys, no network)

pip install -e ".[dev]"
pytest -q                 # 45 tests
triage eval --check       # 57 labelled incidents, gated against evals/baseline.json
make demo                 # diagnose examples/outage.jsonl end to end
triage runbooks           # regenerate docs/runbooks/

$ triage diagnose --logs examples/outage.jsonl --ticket-file examples/outage.ticket.txt
cause      : Third-party service outage [upstream_outage]  confidence 0.90
severity   : sev1   escalate: yes   page on-call: YES   users: 3
evidence   : req_wwxk8x, req_9cj357, req_x5sauv, req_yzfw76, req_vmn5te

Input is JSON lines. Only request_id and status are required; the more of ts, user_id, connector, key_prefix, upstream_status, retry_after, cursor, source and a latency breakdown you include, the more causes it can tell apart.

What it can diagnose

"Public docs" means the behaviour is quoted from Merge's public Agent Handler FAQ; the two exact strings used (billing_limit_reached and "Resource not in API key scope.") come from there. Every other error string in this repo is illustrative.

Design choices worth noting:

• It asks instead of guessing: no logs, or only healthy ones, produces a reply asking for a request ID and a UTC time, not an escalation.

• When two patterns overlap, the dominant one wins and the runner-ups are listed as alternatives.

• A low-confidence match escalates rather than sending a confident wrong answer.

• Outages and sustained latency scale severity with users affected, and sev1 says to page on-call.

Use it from an agent (MCP)

{ "mcpServers": { "integration-triage": { "command": "triage-mcp" } } }

Tools: diagnose_incident(logs_jsonl, ticket), get_runbook(cause_id), list_causes(). Output is already redacted and includes escalate and page_oncall flags. Tested over a real stdio connection (mcp 2.3.0).

How it is tested

57 labelled incidents in three groups:

Metrics: cause accuracy (overall, hard, wild), escalation correctness and recall, evidence validity (every cited request ID exists in the logs), no secret leak (planted keys must not appear in any reply or packet) and reply quality (cites IDs, gives numbered steps, no promises, no blame, short). triage eval --check fails CI if a score drops below evals/baseline.json, with zero tolerance for leaks and missed escalations.

The checks can fail. Turning redaction off drops the leak score from 1.00 to 0.53 (this is a test in the suite), and disabling the rate-limit matcher drops cause accuracy to 0.89.

Limits

• All 1.000 scores are on data I wrote. The standard cases come from my own generator and the matchers were built against the same patterns, so they mostly show the pipeline is consistent. The wild set is the fairer test: it scored 11/12 on its first run (a bare 401 Authentication failed was missed), I generalized one matcher, and it now scores 12/12. Treat it as a development set now, not an unseen benchmark.

• No real platform logs were used. Real logs will have shapes and failure modes this does not cover. Run it on your own logs before trusting any number here.

• Matching is rule-based on purpose: it is fast, free and explainable, and every conclusion cites its evidence. It has no LLM, so it cannot read free-text stack traces or customer prose beyond a few keywords.

• Thresholds (for example "3 or more calls", "500 ms") are judgment calls, written down in causes.py.

• Replies are templated. They are accurate and short but not warm; a person should read one before it is sent.

Layout

src/integration_triage/
  causes.py     cause catalog + matchers (what each cause is, how to confirm it, what to say)
  diagnose.py   scoring, dominant-cause selection, severity, escalation
  compose.py    customer reply, escalation packet, runbook entry
  checks.py     reply quality checks      redact.py   secret redaction
  evaluate.py   metrics + baseline gate   simulate.py synthetic incident generator
  mcp_server.py MCP tools                 cli.py      `triage` command
  data/         cases.jsonl (generated), wild_cases.jsonl (hand-written)
docs/runbooks/  one generated runbook per cause
examples/       sample logs and tickets

License

MIT