Splunk Intelligence — Local LLM Analysis Stack

A local Splunk investigation stack that ingests exports (JSON/CSV) or runs live SPL queries, applies deterministic detectors, and drives a structured multi-iteration investigation loop via MCP tools exposed to AI agents (GitHub Copilot or Claude Code). Everything runs on-device — no data leaves the machine.

How it works

Splunk export (JSON/CSV)  ──or──  Splunk REST API
    └─> parsers.py        # Polars DataFrame: field extraction, timestamp normalisation
    └─> detectors.py      # rule-based: spikes, cert anomalies, host rankings, timeline
    └─> mcp_server.py     # FastMCP: exposes investigation tools to Copilot / Claude
    └─> runner.py         # CLI orchestrator
    └─> reports/          # generated markdown reports
    └─> logs/             # per-run JSONL structured logs
    └─> splunk.db         # SQLite: events, findings, reports, queries per run_id

The investigation loop is self-contained — splunk__submit_report returns {status, findings, next} and the agent loops on its own without external hooks.

Related MCP server: MCP Server for Splunk

Quick start

1. Install prerequisites

• Python 3.12+

• uv — brew install uv

• Splunk instance URL (set SPLUNK_URL env var; required for live queries only)

Ollama is not required. Copilot or Claude Code handles all reasoning via MCP tools. To use the optional standalone LangGraph/Ollama agent, install the llm extra (uv sync --extra llm) and run with --llm.

2. Install dependencies

uv sync --extra dev
uv run playwright install chromium

3. Configure Splunk URL (live queries only)

echo "SPLUNK_URL=https://your-splunk-instance:8089" > .env

4. Authenticate to Splunk (live queries only)

uv run python -m splunk.auth

This opens a visible Chromium window via Playwright. Complete the SSO login manually. The session cookie is saved to ~/.splunk/auth.json and loaded automatically on every live query. Re-run when your session expires (Splunk uses SSO/SAML — password login is not available).

5. Run an investigation

# From a local export file
uv run python -m splunk --input results/cert_errors.json

# Live SPL query
uv run python -m splunk --live --spl "index=pki sourcetype=ocsp_error" --earliest -6h

# Parsers + detectors only, no LLM
uv run python -m splunk --input results/cert_errors.json

# With standalone Ollama agent (requires uv sync --extra llm + Ollama running)
uv run python -m splunk --input results/cert_errors.json --llm

Via AI agent (MCP tools)

Run both servers — the FastAPI UI server and the MCP tool server:

# Terminal 1 — FastAPI UI (http://127.0.0.1:8765)
./serve.sh

# Terminal 2 — MCP tool server
uv run python -m splunk.mcp_server

The UI at http://127.0.0.1:8765/ui/runs/<run_id> shows live investigation progress, findings, and the final report. The MCP server exposes the investigation tools to the agent.

Then ask Copilot or Claude: "Start a Splunk investigation on results/cert_errors.json"

The agent calls splunk__investigate_start, reasons over findings, and loops via splunk__submit_report until confident. See AGENTS.md for the full loop protocol.

MCP Tools

Onboarding (new team members)

An interactive onboarding prompt is available for GitHub Copilot. In VS Code Copilot Chat, attach .github/prompts/onboard.prompt.md via the # file picker — Copilot will walk you through setup, auth, and running your first investigation.

Tests

uv run pytest tests/

Tests are fully deterministic — no Ollama, no Splunk connection, no server required. Fixtures live in tests/fixtures/.

Key files

Environment variables

Put these in a .env file at the repo root (gitignored).

Agent instructions

• GitHub Copilot — see AGENTS.md for loop rules, MCP tool reference, and report format

• Claude Code — see CLAUDE.md for project conventions and task backlog

• Onboarding — see .github/prompts/onboard.prompt.md