ci-triage-mcp

Ask Claude why your last CI run failed. Get a real answer built from parsed logs, not a guess from raw text.

Pipeline · Guardrail · Setup · Tools · Extractors · Dashboard

This is an MCP server for Claude Desktop and Claude Code. You ask, in conversation, why a GitHub Actions run failed; Claude fetches the run, and this server deterministically extracts the actual failure signal -- test name, class, assertion message, stack trace -- from JUnit/Surefire XML, ESLint, tsc, Prettier, or Maven console output. Claude never reads a raw log to guess what went wrong; it reasons over structured data a real parser produced.

No API key, no server to deploy, nothing running unattended. The "LLM" is whichever Claude client is already open when you ask -- same design as k6-loadtest-mcp.

Why this exists instead of just pasting a log into Claude

The LLM plans and explains. Code fetches and parses. The two never trade places.

Pasting a 2,000-line CI log into a chat works, but it wastes context on install/browser-download noise and leaves the model guessing at which lines actually matter. This server does the mechanical part in real parsers -- JUnit/Surefire XML via a proper XML parser, not regex on a raw log wherever structured output exists -- so what Claude sees is already reduced to the handful of lines that are the failure. It also remembers: a local history store means "this test has failed the same way 3 times this month" is something the tool tells you, not something you have to recall.

You:    Why did the last run of playwright-typescript-framework fail on main?

Claude: Fetched run 30298934612, extracted 14 failure signals across the run's failed
        shards, and cross-referenced them against local history...

        This run mixes one real bug with a batch of environment flakiness:

        🔴 Real bug -- axe-core flags a WCAG 2 AA color-contrast violation on the
           products, cart, and checkout pages (3 signals, one shared breadcrumb
           component: #6d68cf on #0a0a11, a 4.25:1 ratio against the 4.5:1 minimum).

        🟡 Not a bug -- 11 more signals (API timeouts, cancelled page loads) across
           unrelated specs all point at one shard losing connectivity to
           playground.krishanchawla.com mid-run, not broken tests.

        Fix the breadcrumb's text-accent token once; rerun the flaky shard once the
        target's back up.

Condensed, but the real shape of it -- this is an actual run this project's own live-testing was validated against (see Extractors), not a hypothetical.

Related MCP server: claude-rag-mcp

Pipeline

flowchart TD
    A["you: 'why did the last run of\nplaywright-typescript-framework fail?'"] --> B[fetch_pipeline_run]
    B --> C{artifact named\njunit/surefire?}
    C -->|yes| D[download + unzip artifact\nparse JUnit/Surefire XML]
    C -->|no, or expired| E[get_job_log\nparse ESLint / tsc / Prettier / Maven console]
    D --> F[extract_failure_signal\nstructured FailureSignal + signature]
    E --> F
    F --> G[find_similar_past_failures\nlocal history lookup]
    G --> H[Claude writes the explanation\nfrom structured data]
    H -. optional .-> I[record_triage_note\nlocal only, no confirmation needed]
    H -. optional, ask first .-> J[publish_triage\n→ shared dashboard]

    G1["Guardrail: allowedRepos,\nnot agent-editable"]
    B -. enforced before every fetch .-> G1

    style G1 fill:#6552D0,color:#fff,stroke:#333
    style J stroke-dasharray: 4 3

triage_pipeline_failure chains fetch → download/parse every relevant artifact and failed job's log → history lookup, in one call, and falls back to job-log parsing per artifact rather than aborting the whole run if one has expired. The granular tools exist for targeting one specific job.

Guardrail

Actions data (runs, job logs, artifacts) is only ever fetched for repos listed in allowedRepos in ~/.ci-triage-mcp/config.json (empty by default). The tools cannot add to this list themselves -- the same shape as k6-loadtest-mcp's host allowlist: an agent-authored call, legitimate or prompt-injected, doesn't get to expand its own blast radius. Add a repo yourself once you've confirmed you're authorized to read its Actions data:

{ "allowedRepos": ["krishanchawla/playwright-typescript-framework", "krishanchawla/selenium-java-framework"] }

Setup

Prerequisites: Node.js 18+, and a GitHub token available -- either the GITHUB_TOKEN env var, or the gh CLI already logged in (gh auth login); this server falls back to gh auth token automatically. This is your own local credential, used to call GitHub's API on your own behalf -- nothing is ever stored server-side or embedded in a deployed service, which is deliberate (see Why not just call an LLM API directly).

npm install
npm run build

Try the extractors locally first

npm run harness   # runs every parser against fixtures/ and checks the counts -- no GitHub calls

Try it against a real repo, without going through MCP

npm run live-check <owner> <repo> [branch]   # defaults to krishanchawla/playwright-typescript-framework main

Requires a real token (GITHUB_TOKEN or gh auth login). Exercises the same fetch → extract → history-match logic triage_pipeline_failure wires together, printed directly instead of over MCP transport -- useful for checking a parser against a real log before trusting it in conversation. This is how every bug documented in Extractors below was actually found.

Both playwright-typescript-framework and selenium-java-framework also have a standing demo branch (their main branches stay clean, ready-to-clone framework skeletons -- see each repo's own README) that exists specifically to give this project real, current CI failures to test against, instead of hoping main's last 30 runs happen to include one:

npm run live-check krishanchawla playwright-typescript-framework demo

Register with Claude Desktop / Claude Code

Claude Code, from a terminal:

claude mcp add ci-triage-mcp -- node /absolute/path/to/ci-triage-mcp/dist/index.js

If GITHUB_TOKEN isn't already in your shell environment and you're not relying on gh auth token, set it at registration time instead of in your current shell -- the server won't see a variable set afterward in some other terminal:

claude mcp add ci-triage-mcp -e GITHUB_TOKEN=<token> -- node /absolute/path/to/ci-triage-mcp/dist/index.js

Claude Desktop, edit claude_desktop_config.json:

{
  "mcpServers": {
    "ci-triage-mcp": {
      "command": "node",
      "args": ["/absolute/path/to/ci-triage-mcp/dist/index.js"]
    }
  }
}

Fully quit and restart Claude Desktop/Claude Code after registering or changing this -- it spawns the server once at startup and won't notice config/env changes made afterward, including a rebuilt dist/. This bites people (it bit me while building this) far more often than it should.

Then add the repos you want triaged to allowedRepos (see Guardrail), and ask, e.g.:

Why did the last run of playwright-typescript-framework's CI fail on main?

Tools

Extractors

selenium-java-framework has never had a failed CI run on its own -- there was nothing real to validate the Maven console parser against, so it shipped tested only against a hand-written fixture. To actually check it rather than leave that as a guess: a throwaway branch with one assertion deliberately flipped ($39.50 → $999.99), opened as a PR (triggers the same workflow, touches main's history not at all), triaged for real once it failed, then closed unmerged. The parser correctly pulled the AssertJ diff (expected:<"$[999.99]"> but was:<"$[39.50]">), test name, class, and line straight out of the real console output on the first try -- see PR #1 (closed) for the actual run this validated against.

Five bugs live-testing against real playwright-typescript-framework runs has actually surfaced, in the order they were found:

• GitHub prefixes every line of a raw job log with an ISO-8601 timestamp (stripped once at the source in github.ts, getJobLog). If you add a new text-based parser, write its regexes against already-stripped content -- every fixture in fixtures/ is pre-stripped for exactly this reason.

• A expect(x).toEqual(y) failure against a large object opens with a pretty-printed JSON dump before anything readable -- parsePlaywrightList prefers the annotated > N | expect(...) source line instead when the message would otherwise just be a bare Error: [.

• Playwright retries re-print the same failure block. A test that retries twice re-dumps the same (sometimes huge) error text three times into what parsePlaywrightList treats as one failure's block -- one accessibility assertion against a large violations object produced a ~60KB stackTrace on a single signal this way. src/extract/truncate.ts caps every extracted stackTrace at 4000 chars now, in every parser, not just this one.

• An artifact GitHub still lists (with a real file size) can still 410 on download once it's past its retention window -- listArtifacts doesn't reflect expiry, only the download attempt does. triage_pipeline_failure now catches each artifact's download individually and falls back to job-log parsing for that job instead of aborting the whole call.

• The bare-JSON-opener fix above only ever applied to parsePlaywrightList, not parseJUnitXml. Playwright's own JUnit reporter truncates a <failure message="..."> attribute the exact same way its list reporter's first line gets truncated -- so the preferred path (real XML) was producing a worse message ("[") than the fallback path (scraped console text) for the identical failure. The message-picking logic is now shared (src/extract/message.ts) so the two parsers can't drift on this again.

Why not just call an LLM API directly

Because that would mean an Anthropic API key living on a public-facing server, paid for per call and reachable if that server is ever compromised -- a materially different (and worse) risk than anything else in this project. This server never calls an LLM API at all: it's tool calls that the already-running Claude Desktop/Code session decides to make, under whatever plan you're already paying for. Nothing here would need to change if you're using Claude Free, Pro, or Max -- the server doesn't know or care.

Dashboard

dashboard/ is an optional Spring Boot + Thymeleaf app, sibling to k6-loadtest-mcp's own dashboard/, that publish_triage posts a triage result to -- gives it a real, shareable URL instead of living only in one Claude conversation. Same design as the load-test dashboard: a plain jar with its own embedded server, H2 file-backed storage, bearer-token-gated ingest separate from HTTP-Basic-gated (or public-demo, unauthenticated) viewing.

What it adds beyond just listing runs:

• Category breakdown chart across every extracted signal, not just each run's headline category -- a single run routinely mixes categories (the Example above is real: one CI run produced a genuine accessibility regression and an unrelated cluster of infra timeouts, and counting at the signal level is the only way that doesn't get hidden behind whichever one happened to run first).

• Recurrence tracking -- every signal is matched against prior triage runs for the same repo by its stable signature; the detail page shows "seen N× before" instead of treating every failure as novel, and the list page surfaces a standing "Recurring failures" panel.

• Narrative-first detail page -- the LLM's explanation and suggested fix are the headline content, with raw stack traces behind a <details> disclosure per signal, not the other way around.

• A 14-day run-volume sparkline per repo, so a rising or falling triage rate is visible at a glance, not just a bare run count.

Build and run it locally

cd dashboard
mvn -q package                       # -> target/ci-triage-dashboard.jar
DASHBOARD_API_TOKEN=<pick-a-token> java -jar target/ci-triage-dashboard.jar

Then point dashboardUrl in ~/.ci-triage-mcp/config.json at it (e.g. "http://localhost:8081" while testing locally) and set CI_TRIAGE_DASHBOARD_TOKEN to match, on the MCP server's own registration (see Setup for why it has to be set there, not a shell env var).

Deploying it

Same posture as k6-loadtest-mcp's dashboard -- a self-contained jar with its own embedded server (Spring Boot 4 / Jakarta EE, needs Tomcat 11+ if you ever did drop it into an external container, which there's no reason to). Run it via systemd with:

Roadmap

• Artifact upload for selenium-java-framework. The console parser works (see Extractors), but a target/surefire-reports/ upload-artifact step (mirroring what playwright-typescript-framework already does) would let it use the real JUnit XML parser instead -- structured XML over scraping console output whenever it's available at all.

• CI on this repo itself. npm run harness runs the extractor fixtures locally but nothing runs it on push -- a .github/workflows job that fails loudly on a broken parser would be a cheap, honest thing for a CI-triage tool to be missing.

• Actually deploy the dashboard to the VPS and wire dashboardUrl there -- built, verified locally, and now proven end-to-end against a real triage result (see the Dashboard screenshots above), just not yet live anywhere public.

• live-check.ts and triage_pipeline_failure reimplement the same fetch → extract pipeline independently. They drifted once already -- live-check.ts already caught per-artifact download failures individually, but triage_pipeline_failure didn't until a live test against the actual MCP tool caught the gap. Worth factoring into one shared function both call, so a fix to one can't silently miss the other again.