Skip to main content
Glama

Incident Triage Agent

A small Python project that shows an MCP server, a golden-dataset eval harness and Langfuse tracing working together. An agent triages synthetic incident alerts by calling a purpose-built, read-only MCP server (incident-mcp), and every run is scored and traced.

Status: repository tooling, CI, the incident fixtures and the incident-mcp server (three tools and the incident:// resource) are in place (issues #1–#4 and #9); the agent and evals are not built yet. See PRODUCT.md for what this is and the project plan for how it will be built. Progress is tracked in the GitHub issues and milestones.

This README is replaced with real usage, architecture and results as the milestones land. It will only state numbers that a run produced.

Development

Requires Python 3.11+ and uv.

uv sync                      # create .venv and install the locked dependencies and dev tools
uv run ruff check            # lint
uv run ruff format --check   # formatting
uv run pyright               # type check
uv run pytest                # tests

CI (.github/workflows/ci.yml) runs each of these as its own job on every pull request and push to main, plus a gitleaks scan of the full git history. None of it needs an API key. Secrets go in .env, which is git-ignored.

Pinned versions and the reasons for them are in ADR-001. On an Intel Mac the lockfile deliberately resolves cryptography to an older release; the ADR explains why.

Related MCP server: Athena Investigation MCP Server

Fixtures

The data the MCP server will serve lives in src/incident_triage/fixtures/data/: one directory per incident (incident.json, logs.jsonl, ticket.json) and shared runbooks. All of it is invented. IDs are type-prefixed and globally unique (INC-0001, LOG-0001-0042, TKT-0001, RB-DB-POOL). Fixtures never carry root-cause labels, because the agent can read them; labels belong in the golden set. load_fixtures() validates everything and reports problems as <path>:<line>: <id>: <problem>.

Running the server

incident-mcp is a read-only MCP server over the fixtures, with three tools and one resource.

uv run incident-mcp                                   # stdio (what an MCP host launches)
uv run incident-mcp --transport http --port 8000      # Streamable HTTP at http://127.0.0.1:8000/mcp

Name

Kind

What it returns

incident://{incident_id}

resource (JSON)

Where an investigation starts: the alert, severity, ticket_ids, runbook_ids, log_count, first and last log time, and the services in the logs. Listed per incident and as a template.

search_logs

tool

One page of an incident's logs. Takes an incident_id (required), query (space-separated terms, all must match), an optional minimum level and exact service, limit (1–50, default 20) and offset. Returns records in timestamp order with their LOG-… IDs, total_matches and next_offset.

get_ticket

tool

One ticket (TKT-…), with its body and comments.

get_runbook

tool

One runbook (RB-…): the symptoms it covers and its steps.

A model using the tools can't read resources itself; the host or agent code reads incident:// and gives it the overview. No match from search_logs is an empty result. An unknown but well-formed ID is an explicit error: an unknown incident names the known ones, and an unknown ticket or runbook points to the incident's incident:// overview. A malformed or wrong-kind ID (get_ticket("RB-DB-POOL")) is rejected by the tool's schema.

Every response is bounded because the fixtures are. A log message is at most 1,000 characters with 20 attributes; a ticket body at most 4,000 characters, with 20 comments of up to 2,000; a runbook at most 10 symptoms and 20 steps, each up to 500. An oversized fixture fails at load time. The limits are named constants in src/incident_triage/fixtures/models.py.

Logs are JSON lines on stderr, never stdout (stdout is the protocol channel under stdio): one server_started line with the transport and fixture counts, then one request line per request with a fresh request_id, its method, tool or uri, outcome and duration_ms. The outcome is ok; tool_error (a tool reported an anticipated error to the model); rejected (the request was wrong, for example an unknown resource, logged at INFO with its error_code); or raised (a server fault, logged at ERROR). So "outcome": "raised" always means a bug. Invalid fixtures stop the server with exit code 2 and a fixtures_invalid line naming the file and line.

Available Tools

3 tools
get_runbookA
Read-onlyIdempotent

Get one runbook: the symptoms it covers and its steps. Read-only.

    An incident's runbook IDs are listed in its incident://<incident_id> resource.
    

Inputs: runbook_id (required). Returns: id, steps, title, service, +1 more

get_ticketA
Read-onlyIdempotent

Get one ticket, with its body and comments. Read-only.

    An incident's ticket IDs are listed in its incident://<incident_id> resource.
    

Inputs: ticket_id (required). Returns: id, body, title, status, +4 more

search_logsA
Read-onlyIdempotent

Search one incident's logs. Read-only and paginated, in timestamp order.

    No match is not an error: it returns total_matches 0. Each record carries its LOG- ID.
    

Inputs: incident_id (required), level, limit, query, +2 more. Returns: offset, records, returned, incident_id, +2 more

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    A4.1/5.0

    Scored across 3 tools

    Disambiguation5/5

    Each tool targets a clearly distinct artifact: logs, a ticket, or a runbook. The read-only nature and ID sourcing from the incident resource further separate their roles, leaving no realistic overlap.

    Naming Consistency5/5

    All names follow a consistent snake_case verb_noun pattern: search_logs, get_ticket, get_runbook. The verb and plural/singular choices match the artifact semantics without introducing ambiguity.

    Tool Count5/5

    Three tools is lean but well within the appropriate range for a focused incident-investigation server. Each tool has a distinct, necessary role and none appears redundant.

    Completeness3/5

    The set covers read-only retrieval of logs, tickets, and runbooks, but lacks incident-level tools and any write operations such as updating tickets, commenting, or changing incident status. The incident resource helps anchor IDs, but the tool surface remains notably incomplete for full incident lifecycle work.

    Maintenance

    ActivityMaintained
    ResponsivenessNo recent community issues

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      B
      quality
      D
      maintenance
      Enables natural-language investigation of Datadog data including logs, metrics, monitors, traces, hosts, dashboards, events, and incidents, all through read-only API access.
      14
      1,905 npm
      MIT
    • A
      license
      Not graded
      quality
      D
      maintenance
      Enables governed, read-only investigations of Amazon Athena access logs by searching exact user, IP, or correlation indicators and summarizing activity by correlation ID, with parameterized queries, validation, and result limits.
      MIT
    • A
      license
      Not graded
      quality
      C
      maintenance
      Enables read-only investigation of service incidents by retrieving metrics, logs, and deployment history, and correlating evidence for root-cause analysis.
      MIT
    • A
      license
      Not graded
      quality
      C
      maintenance
      Enables MCP clients to query bundled synthetic logs, service metrics, and traces through three bounded read-only tools for evidence-first incident investigation.
      MIT