This server exposes MCP tools that let an AI agent run and iteratively drive Splunk investigations — from local file exports or live SPL queries — reasoning over detector findings and producing reports.
Start an investigation from a JSON/CSV export file or a live SPL query, with optional repo path for code cross-referencing.
Submit a markdown report with confidence and follow-up SPL queries; the server executes them and returns next findings or done status.
Fetch the current findings for an active run mid-loop.
Pause an investigation after the current iteration completes.
Retrieve example SPL queries from past investigations to ground follow-up queries in proven field patterns.
Trace a function or symbol through a microservice call graph via LSP (callers/callees, depth, file/line lookup).
Inject analyst hints to steer the next iteration of the investigation.
Provides tools for investigating Splunk data by ingesting exports (JSON/CSV) or running live SPL queries, applying deterministic detectors, and managing an investigation loop with findings and reports.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type @ followed by the MCP server name and your instructions, e.g., "@Splunk Intelligence MCP Server Investigate cert_errors.json for anomalies"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
A local Splunk investigation stack that ingests exports (JSON/CSV) or runs live SPL queries, applies deterministic detectors, and drives a structured multi-iteration investigation loop via MCP tools exposed to AI agents (GitHub Copilot or Claude Code). Everything runs on-device — no data leaves the machine.
Splunk export (JSON/CSV) ──or── Splunk REST API
└─> parsers.py # Polars DataFrame: field extraction, timestamp normalisation
└─> detectors.py # rule-based: spikes, patterns, cert anomalies, correlations,
│ # severity, host rankings, slow queries, numeric anomalies
└─> connector.py # facade: loading, detection, run state — no HTTP, no server
│ # process; MCP tools, the TUI, and its own CLI
│ # (`python -m splunk`) all call into it directly
└─> mcp_server.py # FastMCP: exposes investigation tools to Copilot / Claude
└─> tui.py # terminal UI: run history + live progress, reads splunk.db directly
└─> reports/ # generated markdown reports
└─> logs/ # per-run JSONL structured logs (audit trail — every
│ # investigate/pause/hint/done action, not just the CLI pipeline)
└─> splunk.db # SQLite: events, reports, queries, active_runs, alerts per run_id
standalone/ # top-level dir: processes that run outside the MCP/agent loop
└─> agent.py # optional: standalone LangGraph ReAct loop (`python -m splunk agent`),
│ # via splunk/llm_backends.py — ollama / claude_cli / copilot_cli
└─> watcher.py # standalone process (python -m standalone.watcher) — polls Splunk on an
# interval, runs detectors, writes hits to splunk.db's alerts table;
# consumed via splunk__check_alerts / splunk__ack_alertThe investigation loop is self-contained — splunk__submit_report returns {status, findings, next} and the agent loops on its own without external hooks.
Copilot/Claude via MCP is the primary reasoning path — no Ollama or CLI subprocess required. For environments without either, standalone/agent.py provides an optional standalone LangGraph ReAct agent, enabled via uv run python -m splunk agent --source <file> (requires uv sync --extra llm). See Standalone agent below for backend options.
Related MCP server: Splunk MCP for SOC Operations
Python 3.12+
uv — brew install uv
Splunk instance URL (set SPLUNK_URL env var; required for live queries only)
uv sync --extra dev
uv run playwright install chromiumecho "SPLUNK_URL=https://your-splunk-instance:8089" > .envuv run python -m splunk.authThis opens a visible Chromium window via Playwright. Complete the SSO login manually. The session cookie is saved to ~/.splunk/auth.json and loaded automatically on every live query. Re-run when your session expires (Splunk uses SSO/SAML — password login is not available).
# From a local export file
uv run python -m splunk findings --source results/cert_errors.json
# Live SPL query
uv run python -m splunk findings --spl "index=pki sourcetype=ocsp_error" --earliest -6h
# Findings JSON on stdout instead of a markdown report
uv run python -m splunk findings --source results/cert_errors.json --jsonNo server process required — start the MCP tool server, and optionally the TUI:
# Terminal 1 — MCP tool server
uv run python -m splunk.mcp_server
# Terminal 2 (optional) — terminal UI for watching live investigation progress
uv run python -m splunk.tuiThen ask Copilot or Claude: "Start a Splunk investigation on results/cert_errors.json"
The agent calls splunk__investigate_start, reasons over findings, and loops via splunk__submit_report until confident. See docs/investigation-loop.md for the full loop protocol.
The TUI reads splunk.db directly for run history and the rendered report, and polls the active_runs table for live iteration/confidence/event-count every ~2s — no HTTP involved. Because every connector function writes to active_runs regardless of which process calls it, the TUI shows live per-iteration progress for both MCP/Claude-driven investigations and the standalone agent path — previously (before this design), MCP-driven progress was invisible to any other process since it only lived in an in-memory dict inside whichever process was running it.
Same investigation engine, no MCP tool-calling required:
uv run python -m splunk start --source results/cert_errors.json
uv run python -m splunk submit-report --run-id <id> --report "..." --queries "-- tls\nindex=pki ..."
uv run python -m splunk get-findings --run-id <id>
uv run python -m splunk pause --run-id <id>
uv run python -m splunk hint --run-id <id> --text "focus on web-01 after 14:30 UTC"python -m splunk agent)For environments without Copilot or Claude Code driving MCP tools directly, standalone/agent.py
runs its own LangGraph ReAct loop over the same detector findings and produces the same kind of
markdown report. It's a fallback, not the primary path — prefer the MCP flow above when available.
uv sync --extra llm # pulls in langgraph, langchain-core, langchain-ollama
uv run python -m splunk agent --source results/cert_errors.jsonThe chat backend driving the loop is selected via SPLUNK_AGENT_BACKEND (default ollama):
Backend | Requires | Notes |
|
| Model via |
|
| No API key needed — reuses your existing login. Model via |
|
| No API key needed. Model via |
# Ollama (default) — needs `ollama serve` running and the model pulled
ollama pull qwen2.5:14b
uv run python -m splunk agent --source results/cert_errors.json
# Claude CLI — no separate server process, reuses your `claude` login
SPLUNK_AGENT_BACKEND=claude_cli \
uv run python -m splunk agent --source results/cert_errors.json
# Copilot CLI
SPLUNK_AGENT_BACKEND=copilot_cli \
uv run python -m splunk agent --source results/cert_errors.jsonclaude_cli/copilot_cli shell out to the CLI non-interactively (claude -p / copilot -p)
with the CLI's own tool use disabled, bridging tool-calling by hand via a small JSON protocol —
see splunk/llm_backends.py and splunk/cli_tool_protocol.py for how. One CLI session is opened
per investigation and reused (--resume) across all ReAct iterations rather than starting cold
every turn.
SPLUNK_AGENT_MAX_ITER (default 10) caps ReAct loop iterations regardless of backend.
/splunk-investigate <input> — the investigation loop (splunk__investigate_start →
reason over findings → splunk__submit_report → repeat). <input> is a file path or an
SPL query — one skill handles both:
/splunk-investigate results/cert_errors.json or
/splunk-investigate "index=pki sourcetype=ocsp_error" --earliest -6h
If invoked with no argument, it asks.
Nine splunk__* tools drive the loop. The tool list, response shapes and done rules are in
docs/investigation-loop.md.
An interactive onboarding prompt is available for GitHub Copilot. In VS Code Copilot Chat, attach .github/prompts/onboard.prompt.md via the # file picker — Copilot will walk you through setup, auth, and running your first investigation.
uv run pytest tests/Tests are fully deterministic — no Splunk connection, no server required. Fixtures live in tests/fixtures/.
--live path locallylocal_splunk/ provides a throwaway single-instance Splunk container (Docker, based on
splunk/docker-splunk) for exercising --live
queries against a real Splunk REST API and real SPL execution — without production credentials
or SSO. See local_splunk/README.md for setup/teardown steps.
File | Purpose |
| All tunables — thresholds, paths, auth |
|
|
|
|
| Facade: loading, run state, standalone agent loop, the one CLI ( |
| FastMCP server — 9 investigation tools (thin wrappers over connector.py) |
| Terminal UI — |
| Splunk REST client (cookie-based, SSO-compatible) |
| Playwright SSO — opens Chromium, saves cookie |
| SQLite store: events, reports, queries, active_runs, alerts, per-sourcetype schema cache |
| Structured JSON-lines logging per run — audit trail for every connector action |
| Standalone |
| Standalone LangGraph ReAct agent ( |
| Pluggable chat backend for |
| Registry of investigation domains (prompt + SPL template) consumed by |
Variable | Default | Purpose |
| — | Splunk base URL (required for live queries) |
|
| Default index substituted into generated follow-up SPL (standalone agent path) |
| — | Comma-separated indexes relevant to your environment — reference only, surfaced to the user during the live-SPL preflight; doesn't affect |
|
| MCP-driven investigation loop's iteration cap (primary path — |
|
| Event-pair correlation window (seconds) for |
|
| Events/window to trigger a spike |
|
| Spike detection window (seconds) |
|
| Duration (ms) above which an event is flagged as a slow query |
|
| Rolling window size (events) for z-score anomaly detection |
|
| |z-score| above which an event is flagged as a numeric anomaly |
|
| Splunk session cookie name |
|
| Cookie persist path |
|
| Live REST job polling interval (seconds) |
|
| Live REST job poll timeout (seconds) |
|
| Max silent re-auth attempts on a 401 before failing |
|
| Log verbosity |
|
| Standalone agent ( |
|
| Ollama model (backend |
|
| Model passed to |
|
| Model passed to |
|
| Standalone agent ReAct loop cap |
| — | SPL query the watcher ( |
|
| Seconds between watcher poll cycles |
Put these in a .env file at the repo root (gitignored).
Epic planning, subtask creation, task grooming, task implementation using https://github.com/debaditya-mohankudo/Lite-Task-Framework
Claude Code — see CLAUDE.md for project conventions and task backlog
Onboarding — see .github/prompts/onboard.prompt.md
Get current findings from the active investigation session. Use this to inspect the latest detector output mid-loop.
Inputs: run_id (required). Returns: result
Inject an analyst hint into the investigation for the next iteration. The hint is included in the findings passed to the next reasoning step. Example: "focus on web-01 cert chain errors after 14:30 UTC"
Inputs: hint (required), run_id (required). Returns: result
Start a Splunk investigation. Loads events, runs detectors, returns structured findings for Claude to reason over.
Args: source: Path to a Splunk export file (JSON or CSV). Use this OR spl. spl: SPL query string for a live Splunk query. Requires SPLUNK_URL configured. earliest: Earliest time for live query (default: -24h). latest: Latest time for live query (default: now). repo_path: Optional path to the microservice source repo. When provided, the agent can call splunk__lsp_call_chain to trace error log sites back through the call graph. Leave empty to skip code cross-referencing.
Returns JSON with run_id and findings dict.
Inputs: spl, latest, source, earliest, +1 more. Returns: result
Trace a function or symbol through the microservice call graph using LSP. Use this during the Reason step to find which code path produced a log error.
Args: run_id: Active investigation run_id. symbol: Function or class name to look up (e.g. "validate_cert", "TLSHandler"). file_path: Optional absolute path to the file containing the symbol. Speeds up lookup. line: Optional 1-based line number of the symbol definition. direction: "callers" (who calls this?) or "callees" (what does this call?). Default: callers. depth: How many levels up/down to trace. Default: 3.
Returns JSON with the call chain and file locations, or an error if repo_path was not provided at investigate_start or if the symbol cannot be resolved.
Inputs: run_id (required), symbol (required), line, depth, +2 more. Returns: result
Pause the investigation after the current iteration completes.
Inputs: run_id (required). Returns: result
Return example SPL queries from past investigations stored in splunk.db. Use this to ground follow-up queries in field names and patterns that have actually worked against this Splunk environment.
Args: area: Filter by area label (e.g. "tls", "cert", "auth"). Empty = all areas. limit: Max number of examples to return (default 20).
Returns JSON list of {area, spl, result_rows, run_id, iteration} sorted by most recent first. result_rows is the event count the query returned, or null if it was never executed.
Inputs: area, limit. Returns: result
Submit your investigation report and follow-up SPL queries to the server. The server stores the report, executes the queries, builds new findings, and returns either next findings (status=continue) or completion (status=done).
Args: run_id: The run_id from splunk__investigate_start. report: Your markdown investigation report including Confidence: High/Medium/Low. queries: List of follow-up SPL query strings. Each starts with a '-- area' comment line.
Returns JSON with status=continue+findings or status=done+ui_url.
Inputs: report (required), run_id (required), queries. Returns: result
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
Scored across 7 tools
Tools are mostly distinct: investigate_start initiates, get_findings retrieves current findings, hint injects hints, lsp_call_chain traces code, pause halts, query_examples provides example queries, submit_report submits a report. There is slight overlap between get_findings and the findings returned by investigate_start, but descriptions clarify the different usage points.
All tools share the 'splunk__' prefix and use lowercase with underscores, but the verb-noun pattern is inconsistent: 'get_findings' (verb_noun), 'hint' (just verb), 'investigate_start' (verb_noun), 'lsp_call_chain' (noun_verb), 'pause' (verb), 'query_examples' (noun_verb), 'submit_report' (verb_noun). Some tools are single words, others have multiple parts, but overall readable.
Seven tools is a reasonable count for a focused investigation server. It covers the core workflow without being excessive. A few more could be added (e.g., listing past investigations), but the current count is appropriate.
The tool set covers the main investigation lifecycle: start, get findings, inject hints, trace code, pause, get examples, and submit report. Missing operations like direct arbitrary SPL querying are partially handled through investigate_start and submit_report queries. Overall, it's well-scoped with minor gaps.
Open-source agent that observes and fixes your application. Query logs, traces, metrics, incidents.
Analytics, tag manager, email inboxes and session replay an AI agent sets up itself. Open source.
Checks AI-written code for leaked keys and made-up packages, on your machine. Never uploads code.
The AI orchestration agent for modern software teams.
