Agentic Automated Triage for Splunk

This is an agentic incident-triage pipeline for Splunk: a saved-search alert fires a webhook → an orchestrator starts a Claude tool-use conversation → Claude autonomously drives a chain of SPL investigation queries through three MCP tools (search → populate-if-empty → re-search → email) → an AI-written triage report — severity, root-cause hypothesis, breakdowns, and a clickable Splunk deep link — lands in your inbox.

This isn't a static dashboard or a fixed alert template. The agent decides, on each run, what to search, whether the result set needs enriching, and how to characterise the incident — the same three tools are exposed both to the orchestrator's tool-use loop and as a standalone MCP server, so any MCP-compatible client can drive the same investigation.

 Splunk saved-search alert  (index=triage_demo, error_count > 0, every 1 min)
        │  webhook action (HTTP POST JSON)
        ▼
 ┌───────────────────────────┐        ┌──────────────────────────────┐
 │ webhook listener (FastAPI)│  call  │ orchestrator (Anthropic loop) │
 │   POST /webhook  :5001    │ ─────► │   model = claude-sonnet-4-6   │
 └───────────────────────────┘        └───────────────┬──────────────┘
                                                       │ tool-use
            ┌──────────────────────────────────────────┼─────────────────────────┐
            ▼                                           ▼                          ▼
   search_splunk_logs                      populate_splunk_test_data        send_email
   (Splunk REST :8089)                     (Splunk HEC :8088)               (SMTP/mailpit :1025)
            └────────────── same triage.tools module also exposed by the MCP server ┘
                                  (triage.mcp_server, MCP/SSE :8050)

The three tools live in one module (triage/tools.py). That module is exposed two ways: as MCP tools by triage/mcp_server.py (the "one MCP server"), and called directly by the orchestrator's tool-use loop. One implementation, two surfaces.

Components

Related MCP server: MCP Server for Splunk

Prerequisites (this dev box)

These already run as standalone dev containers (Docker Desktop auto-starts them):

Check: docker ps should show both. Python 3.12 on the host is only needed for the scripts/ helpers (py on this machine — the bare python alias is the broken MS Store stub).

Quick start

cd agentic-automated-triage-for-splunk

# 1. Prepare Splunk: mint tokens, create index + HEC + the webhook alert.
#    Writes SPLUNK_API_TOKEN + SPLUNK_HEC_TOKEN into .env (created from .env.example).
py scripts\setup_splunk.py

# 2. (OPTIONAL) Add your Claude API key to .env  (>>> SUBSTITUTE <<<)
#    ANTHROPIC_API_KEY=sk-ant-...
#    Leave it blank to run the deterministic "scripted" mode (see Run modes below) —
#    that is how the boss demo is driven, no key required.

# 3. Bring up the pipeline (MCP server + webhook/orchestrator).
docker compose up --build -d

# 4a. Immediate end-to-end run (no waiting for Splunk's scheduler):
py scripts\trigger_alert.py --manual

# 4b. ...or the real path: inject errors and let the scheduled alert fire (~1 min):
py scripts\trigger_alert.py --count 30

# 5. Verify the email arrived.
py scripts\verify_email.py --subject "[Triage]"
#    ...or just open the mailbox: http://localhost:8025

Automated check of the whole chain:

py tests\test_e2e.py

How the agent behaves

On each alert the orchestrator (Claude) is instructed to:

1. search_splunk_logs for the alert's index over the last 15 minutes.

2. If that returns no/insufficient data → populate_splunk_test_data (realistic sample events via HEC, stamped now) → search_splunk_logs again to confirm.

3. Summarise: counts, top error codes / affected services, a P1–P4 severity, next actions.

4. send_email once — subject starts [Triage], body includes the alert name, findings, the exact SPL, the severity, and the Splunk deep link (in links).

The /test-triage endpoint seeds an empty result set on purpose, so it always exercises the populate-then-re-search branch.

Run modes (with or without an API key)

run_triage() reports its mode explicitly:

• agentic — ANTHROPIC_API_KEY is set. Claude drives the tools in a real tool-use loop and chooses the sequence itself.

• scripted — no key. A deterministic stand-in performs the identical documented procedure (search → populate-if-empty → re-search → stats breakdowns → email) with no model call, so the full pipeline — and the rich HTML report — is demonstrable offline. This is the mode the boss demo runs in. Flip to agentic any time by adding the key and docker compose up -d --force-recreate.

Either way the email is built by triage/report.py from real Splunk stats results, so the breakdowns (top error codes, affected services, regions, latency, severity) are genuine aggregates of the indexed events — not hard-coded.

Resetting the demo data

Injected events stay inside the 15-minute search window for ~15 min, so repeated fires within that window stack up. For a pristine single-incident screenshot, clear the index first (admin Bearer token, config-safe — no index/HEC teardown):

# deletes all events in triage_demo; the next fire re-populates a clean, skewed batch
curl.exe -sk -H "Authorization: Bearer $env:SPLUNK_API_TOKEN" `
  https://127.0.0.1:8089/services/search/jobs `
  --data-urlencode "search=search index=triage_demo | delete" `
  -d exec_mode=oneshot -d output_mode=json -d earliest_time=-24h -d latest_time=now

Verifying the email step

• Web UI: open http://localhost:8025 — the triage email appears at the top.

• CLI: py scripts\verify_email.py --subject "[Triage]" prints From/To/Subject/body and exits 0 on success.

• API: curl http://localhost:8025/api/v1/messages returns the JSON message list.

>>> SUBSTITUTE for your environment <<<

Everything is env-driven via .env (copied from .env.example). Flagged values:

Splunk Cloud note: the previous Victoria trial (prd-p-6oxft) was decommissioned, so this pipeline targets the local splunk-dev container. To repoint at Splunk Cloud, set SPLUNK_HOST to the stack host, supply an ACS/HEC token, and create the alert via the ACS API instead of setup_splunk.py's REST call. All runtime auth stays Bearer-token.

Auth model

Runtime auth is Bearer tokens only — no admin:password, no Basic header, no -u (matches the repo-wide rule). setup_splunk.py performs a single bootstrap form-login (/services/auth/login, a session key — not a Basic header) purely to mint the JWT auth token and HEC token; every subsequent call uses those tokens.

Guardrails

The agent composes its own SPL at runtime, so it is never trusted to behave — it is constrained:

• Read-only SPL guard (triage/spl_guard.py) — every agent-issued query passes through a deny-by-default gate at the single Splunk choke point (splunk_client.run_search) before it reaches the REST API. Commands that mutate state or exfiltrate data (delete, collect, outputlookup, sendemail, script, …) are blocked as pipeline commands — the literal word "delete" appearing in log text still searches fine. Unit-tested offline in tests/test_spl_guard.py (runs in CI).

• Bounded action surface — the agent has exactly three tools (search, populate test data, send email). It cannot touch Splunk config, users, or apps; the only outbound side effect is the triage email, and in the lab that lands in mailpit, not a real mailbox.

• Secrets stay in the environment — API keys and tokens come from .env (gitignored, .env.example is placeholders-only); nothing is hardcoded and nothing is echoed into the report.

• Deterministic fallback — with no ANTHROPIC_API_KEY set, the pipeline runs a scripted mode that exercises the identical tool chain, so the guardrails are testable without a live model.

The MCP server on its own

triage/mcp_server.py is a standalone MCP server you can point any MCP client at:

# SSE on :8050 (default)
py -m triage.mcp_server
# or stdio
$env:MCP_TRANSPORT="stdio"; py -m triage.mcp_server

It exposes exactly search_splunk_logs, populate_splunk_test_data, send_email.

Verification

See VERIFICATION.md for a full end-to-end verification record — the exact tool-call sequence observed, the rendered report contents, and the reproduction steps used to confirm the pipeline works as described.

License

MIT — see LICENSE.