# integration-triage

A support-engineer toolkit. Give it a customer ticket and the request logs, and it names the root
cause, cites the failing request IDs, drafts the customer reply, writes the engineering escalation
packet, and keeps a runbook for next time. Secrets are redacted on the way out. It also runs as an
MCP server, so an agent can triage a ticket itself.

> **Unofficial.** Not affiliated with Merge or any other vendor. It is built from public
> documentation and **synthetic** logs. No real customer data and no vendor internals are used.

```mermaid
flowchart LR
    T[ticket text] --> D[diagnose]
    L[request logs JSONL] --> D
    D --> R[customer reply]
    D --> E[escalation packet<br/>evidence table, severity,<br/>page on-call?]
    D --> B[runbook entry]
    R --> X[redact secrets]
    E --> X
```

## Why this exists

A support engineer's loop is the same every time: find the failing calls, name the cause, tell the
customer what to do, and hand engineering something they never have to ask a follow-up about. Most of
the cost is in the details: "Your API is down!" turns out to be a rate limit, a 404 is really a key
scoped to the wrong user, a customer pasted a live key into the ticket. This tool encodes that loop
and tests it.

## Quickstart (no keys, no network)

```bash
pip install -e ".[dev]"
pytest -q                 # 45 tests
triage eval --check       # 57 labelled incidents, gated against evals/baseline.json
make demo                 # diagnose examples/outage.jsonl end to end
triage runbooks           # regenerate docs/runbooks/
```

```text
$ triage diagnose --logs examples/outage.jsonl --ticket-file examples/outage.ticket.txt
cause      : Third-party service outage [upstream_outage]  confidence 0.90
severity   : sev1   escalate: yes   page on-call: YES   users: 3
evidence   : req_wwxk8x, req_9cj357, req_x5sauv, req_yzfw76, req_vmn5te
```

Input is JSON lines. Only `request_id` and `status` are required; the more of `ts`, `user_id`,
`connector`, `key_prefix`, `upstream_status`, `retry_after`, `cursor`, `source` and a latency
`breakdown` you include, the more causes it can tell apart.

## What it can diagnose

| Cause | Severity | Engineer needed | Basis |
|---|---|---|---|
| Key not scoped to the requested user (404 "Resource not in API key scope.") | sev3 | no | public docs |
| Free-plan limit reached (402 `billing_limit_reached`) | sev3 | no | public docs |
| Per-user rate limit (429, Retry-After) | sev3 | no | public docs |
| Invalid, revoked or rotated API key (401) | sev3 | no | public docs |
| Call blocked by a security or guardrail rule | sev3 | no | public docs |
| Sustained platform latency (overhead above 500 ms) | sev2 / sev1 | yes | public docs |
| Slow third-party app (overhead normal) | sev3 | no | public docs |
| End-user authorization expired | sev3 | no | general practice |
| Connected account lacks a permission | sev3 | no | general practice |
| Third-party outage (5xx across several users) | sev2 / sev1 | yes | general practice |
| Client re-requests the same cursor | sev3 | no | general practice |
| Webhook signature verification failing | sev3 | no | general practice |
| Not enough information | sev3 | no | asks for request IDs |
| Failures with no known pattern | sev2 | yes | escalates |

"Public docs" means the behaviour is quoted from Merge's public
[Agent Handler FAQ](https://docs.merge.dev/merge-agent-handler/resources/faq); the two exact
strings used (`billing_limit_reached` and "Resource not in API key scope.") come from there. Every
other error string in this repo is illustrative.

Design choices worth noting:
- It **asks instead of guessing**: no logs, or only healthy ones, produces a reply asking for a
  request ID and a UTC time, not an escalation.
- When two patterns overlap, the **dominant one wins** and the runner-ups are listed as alternatives.
- A low-confidence match escalates rather than sending a confident wrong answer.
- Outages and sustained latency scale severity with users affected, and sev1 says to page on-call.

## Use it from an agent (MCP)

```json
{ "mcpServers": { "integration-triage": { "command": "triage-mcp" } } }
```

Tools: `diagnose_incident(logs_jsonl, ticket)`, `get_runbook(cause_id)`, `list_causes()`. Output is
already redacted and includes `escalate` and `page_oncall` flags. Tested over a real stdio
connection (mcp 2.3.0).

## How it is tested

57 labelled incidents in three groups:

| Group | Cases | Where they come from |
|---|---|---|
| standard | 45 | generated by `simulate.py` (`triage simulate`) |
| hard | 5 | hand-built conflicts: 429s mixed with 502s, an angry ticket blaming the wrong thing, one user with three 503s |
| wild | 12 | hand-written separately, with different log shapes (nested `error` objects, `Retry-After` headers, no timestamps) and different phrasing |

Metrics: cause accuracy (overall, hard, wild), escalation correctness and recall, evidence validity
(every cited request ID exists in the logs), **no secret leak** (planted keys must not appear in any
reply or packet) and reply quality (cites IDs, gives numbered steps, no promises, no blame, short).
`triage eval --check` fails CI if a score drops below `evals/baseline.json`, with zero tolerance for
leaks and missed escalations.

The checks can fail. Turning redaction off drops the leak score from 1.00 to 0.53 (this is a test in
the suite), and disabling the rate-limit matcher drops cause accuracy to 0.89.

<img width="1470" height="708" alt="image" src="https://github.com/user-attachments/assets/e87f08f9-c1e7-47bc-ac8c-c2dced65aae5" />


## Limits

- **All 1.000 scores are on data I wrote.** The standard cases come from my own generator and the
  matchers were built against the same patterns, so they mostly show the pipeline is consistent. The
  wild set is the fairer test: it scored 11/12 on its first run (a bare `401 Authentication failed`
  was missed), I generalized one matcher, and it now scores 12/12. Treat it as a development set
  now, not an unseen benchmark.
- No real platform logs were used. Real logs will have shapes and failure modes this does not cover.
  Run it on your own logs before trusting any number here.
- Matching is rule-based on purpose: it is fast, free and explainable, and every conclusion cites
  its evidence. It has no LLM, so it cannot read free-text stack traces or customer prose beyond a
  few keywords.
- Thresholds (for example "3 or more calls", "500 ms") are judgment calls, written down in
  `causes.py`.
- Replies are templated. They are accurate and short but not warm; a person should read one before it
  is sent.

## Layout

```
src/integration_triage/
  causes.py     cause catalog + matchers (what each cause is, how to confirm it, what to say)
  diagnose.py   scoring, dominant-cause selection, severity, escalation
  compose.py    customer reply, escalation packet, runbook entry
  checks.py     reply quality checks      redact.py   secret redaction
  evaluate.py   metrics + baseline gate   simulate.py synthetic incident generator
  mcp_server.py MCP tools                 cli.py      `triage` command
  data/         cases.jsonl (generated), wild_cases.jsonl (hand-written)
docs/runbooks/  one generated runbook per cause
examples/       sample logs and tickets
```

## License

MIT