integration-triage

A support-engineer toolkit. Give it a customer ticket and the request logs, and it names the root cause, cites the failing request IDs, drafts the customer reply, writes the engineering escalation packet, and keeps a runbook for next time. Secrets are redacted on the way out. It also runs as an MCP server, so an agent can triage a ticket itself.

Unofficial. Not affiliated with Merge or any other vendor. It is built from public documentation and synthetic logs. No real customer data and no vendor internals are used.

flowchart LR
    T[ticket text] --> D[diagnose]
    L[request logs JSONL] --> D
    D --> R[customer reply]
    D --> E[escalation packet<br/>evidence table, severity,<br/>page on-call?]
    D --> B[runbook entry]
    R --> X[redact secrets]
    E --> X

Why this exists

A support engineer's loop is the same every time: find the failing calls, name the cause, tell the customer what to do, and hand engineering something they never have to ask a follow-up about. Most of the cost is in the details: "Your API is down!" turns out to be a rate limit, a 404 is really a key scoped to the wrong user, a customer pasted a live key into the ticket. This tool encodes that loop and tests it.

Related MCP server: mcp-incident-triage-agent

Quickstart (no keys, no network)

pip install -e ".[dev]"
pytest -q                 # 83 tests
triage eval --check       # 58 labelled incidents, gated against evals/baseline.json
make demo                 # diagnose examples/outage.jsonl end to end
triage runbooks           # regenerate docs/runbooks/

$ triage diagnose --logs examples/outage.jsonl --ticket-file examples/outage.ticket.txt
cause      : Third-party service outage [upstream_outage]  confidence 0.90
severity   : sev1   escalate: yes   page on-call: YES   users: 3
evidence   : req_wwxk8x, req_9cj357, req_x5sauv, req_yzfw76, req_vmn5te

Bring your own logs

Input can be JSON lines, a JSON array, a JSON object wrapping a list, or CSV. Common field names from other platforms are recognised automatically (requestId, status_code, response.status, timestamp, durationMs, headers.Retry-After, nested error.code / error.message, ...). For anything else, pass a mapping file of field to dotted path:

triage diagnose --logs examples/rotated_key.export.csv --mapping examples/rotated_key.mapping.json

Only a status is strictly required; a missing request id is replaced by record-N. Timestamps from ISO strings with any offset, YYYY-MM-DD HH:MM:SS or epoch seconds/milliseconds are normalised to UTC so mixed sources sort correctly. Bad records are skipped and reported as warnings instead of failing the whole file. The more of ts, user_id, connector, key_prefix, upstream_status, retry_after, cursor, source and a latency breakdown you include, the more causes it can tell apart.

Run it as a service (free)

triage-serve                                   # http://localhost:8080, docs at /docs in dev
# or: cp .env.example .env && docker compose up -d

Production behaviour, each covered by tests:

• Secrets and PII never leave. Known key shapes (Stripe-style, GitHub, Slack, AWS, Google, LangSmith, JWTs, bearer tokens, api_key= query strings, private key blocks), emails, phone numbers and Luhn-valid cards are removed. A second layer catches key formats nobody listed: any long token mixing letters and digits with near-random characters. Request IDs cited as evidence are protected so replies still point at the right calls. Only redacted text is stored.

• Locked down in prod: TRIAGE_ENV=prod refuses to start without API keys of at least 16 characters, hides /docs, compares keys in constant time, and rate-limits per key.

• Bounded: request bodies over TRIAGE_MAX_BODY_MB get 413; more than TRIAGE_MAX_ENTRIES records gets a clear "filter to the incident window" error. About 3 s to parse 100k lines on a laptop; diagnosis itself is milliseconds.

• Observable: JSON logs with a request id; ticket text and logs are never logged.

Operations: docs/OPERATIONS.md. Validating on real platform errors: docs/REAL_LOGS.md.

What it can diagnose

"Public docs" means the behaviour is quoted from Merge's public Agent Handler FAQ; the two exact strings used (billing_limit_reached and "Resource not in API key scope.") come from there. Every other error string in this repo is illustrative.

Design choices worth noting:

• It asks instead of guessing: no logs, or only healthy ones, produces a reply asking for a request ID and a UTC time, not an escalation.

• When two patterns overlap, the dominant one wins and the runner-ups are listed as alternatives.

• A low-confidence match escalates rather than sending a confident wrong answer.

• Outages and sustained latency scale severity with users affected, and sev1 says to page on-call.

Use it from an agent (MCP)

{ "mcpServers": { "integration-triage": { "command": "triage-mcp" } } }

Tools: diagnose_incident(logs_jsonl, ticket), get_runbook(cause_id), list_causes(). Output is already redacted and includes escalate and page_oncall flags. Tested over a real stdio connection (mcp 2.3.0).

How it is tested

58 labelled incidents in three groups:

Metrics: cause accuracy (overall, hard, wild), escalation correctness and recall, evidence validity (every cited request ID exists in the logs), no secret leak (planted keys must not appear in any reply or packet) and reply quality (cites IDs, gives numbered steps, no promises, no blame, short). triage eval --check fails CI if a score drops below evals/baseline.json, with zero tolerance for leaks and missed escalations.

The checks can fail. Turning redaction off drops the leak score from 1.00 to 0.53 (this is a test in the suite), and disabling the rate-limit matcher drops cause accuracy to 0.89.

Limits

• All 1.000 scores are on data I wrote. The standard cases come from my own generator and the matchers were built against the same patterns, so they mostly show the pipeline is consistent. The wild set is the fairer test: it scored 11/12 on its first run (a bare 401 Authentication failed was missed), I generalized one matcher, and it now scores 12/12. Treat it as a development set now, not an unseen benchmark.

• Not yet run against a live platform. Agent Handler signup requires a company email, which I don't have as an individual, so the MCP behaviour was tested against a local mock built from Merge's public docs (MCP URL format, X-Request-ID, JSON-RPC invalid-params errors), not real responses. Building that mock is what exposed the JSON-RPC-inside-HTTP-200 gap, now fixed. Anyone with a test key can run scripts/capture_agent_handler.py in about two minutes and see how it does on real errors (docs/REAL_LOGS.md).

• Entropy-based redaction errs towards removing: a 40-character git commit hash, for example, is redacted. Request IDs cited as evidence are protected.

• Single instance: SQLite storage and an in-process rate limiter. No TLS; put a proxy in front.

• Matching is rule-based on purpose: it is fast, free and explainable, and every conclusion cites its evidence. It has no LLM, so it cannot read free-text stack traces or customer prose beyond a few keywords.

• Thresholds (for example "3 or more calls", "500 ms") are judgment calls, written down in causes.py.

• Replies are templated. They are accurate and short but not warm; a person should read one before it is sent.

Layout

src/integration_triage/
  causes.py     cause catalog + matchers (what each cause is, how to confirm it, what to say)
  diagnose.py   scoring, dominant-cause selection, severity, escalation
  compose.py    customer reply, escalation packet, runbook entry
  checks.py     reply quality checks      redact.py   secret redaction
  evaluate.py   metrics + baseline gate   simulate.py synthetic incident generator
  mcp_server.py MCP tools                 cli.py      `triage` command
  api.py        HTTP service             service.py  diagnose + store + resolutions
  logs.py       log loader (aliases, mapping, CSV, timestamps)
  store.py      SQLite incident history  config.py   settings, validated at startup
  data/         cases.jsonl (generated), wild_cases.jsonl (hand-written)
docs/runbooks/  one generated runbook per cause
examples/       sample logs and tickets

License

MIT