<p align="center"><img src="docs/assets/cover.png" alt="kassi: divines disaster, crafts the cure" width="400" /></p>

# kassi-CLI

> Divines disaster, crafts the cure.

<p align="center">
  <img src="https://img.shields.io/badge/license-Apache--2.0-blue" alt="License: Apache-2.0" />
  <img src="https://img.shields.io/badge/python-3.12%2B-blue" alt="Python 3.12+" />
  <img src="https://img.shields.io/badge/MCP-k6%20%2B%20Splunk-8A2BE2" alt="MCP: k6 + Splunk" />
</p>

**kassi is an AI agent that load-tests a code change, finds the regression in live Splunk, and
writes the fix, before it reaches production.** Point it at a git diff or a plain-language intent.
It drives real [k6](https://github.com/grafana/mcp-k6) load against the affected endpoints,
correlates the result with server-side telemetry from the official
[Splunk MCP Server](https://splunkbase.splunk.com/app/7931), names the root cause, and hands back a
validated remediation diff. For engineering and SRE teams who need to catch load-induced
regressions before a 2am page.

kassi runs as an audited [Burr](https://github.com/apache/burr) state machine over MCP, served by
[Theodosia](https://msradam.github.io/theodosia/): the driving agent sees one tool, illegal steps
are refused, and every step and refusal lands on a hash-chained ledger. Named for Kassandra, who
foresaw what others would not believe; the workflow is themed as a tarot draw, one Major Arcana
card per phase (`kassi arcana` lays out the spread).

Built for the Splunk Agentic Ops Hackathon (Observability track). See [`DEVPOST.md`](DEVPOST.md)
for the writeup and [`architecture_diagram.md`](architecture_diagram.md) for the design.

<p align="center"><img src="docs/assets/kassi-run.gif" alt="kassi walking its state machine from The Fool to Judgement, correlating k6 load with Splunk telemetry, and proposing a fix" width="860" /></p>

The demo above is a recorded end-to-end run: it prints the state machine, then drives the
whole workflow (script + analysis from the configured model, k6 docs + run, Splunk preflight
and correlation) against a live Splunk.

### Screenshots

| | |
| --- | --- |
| ![state machine](docs/assets/shot-render.png) | ![the major arcana](docs/assets/shot-arcana.png) |
| **`kassi render`**: the full state machine, legal edges only | **`kassi arcana`**: a card per phase |
| ![doctor](docs/assets/shot-doctor.png) | ![correlate](docs/assets/shot-correlate.png) |
| **`kassi doctor --runtime`**: graph + governance checks | **a full run**: k6 + Splunk correlated, every tool call logged |

## What it does

- **Diff or intent driven.** Reads the changed endpoints from a git diff, or scores an OpenAPI spec against a plain-language intent.
- **Real load.** Generates and runs an actual k6 test through the Grafana k6 MCP server, on top of a deterministic scaffold so a run never fails for lack of a model.
- **Server-side truth from Splunk.** Correlates the run with windowed SPL through the official Splunk MCP Server, then runs the Splunk AI Toolkit (`StateSpaceForecast` + `anomalydetection`) to locate the saturation onset statistically. Catches latency degradations that produce zero errors and would slip past any threshold alert.
- **Root cause and a fix.** Writes a cited analysis and a validated remediation diff that applies cleanly, screened by an independent auditor model before the verdict is sealed.
- **Audited by construction.** A governed state machine refuses illegal steps; every step and refusal is on a hash-chained ledger that `kassi verify` checks.
- **Hands-free guard.** `kassi watch` runs the whole workflow when a commit changes an endpoint, catching the regression at commit time.
- **Model-agnostic.** The same loop (drive, write, audit) runs on a local open 8B, on-prem and air-gapped, or on a frontier model, unchanged.
- **Self-observable.** Publishes its own state-machine walk back to Splunk, so the agent is visible in the system it observes.
- **Measured.** 0% false alarms on a live ground-truth benchmark; root cause in the top 3 100% of the time on RCAEval RE3 ([details](#benchmark)).

## Install

```bash
uv sync
```

kassi drives k6 through the [Grafana k6 MCP server](https://github.com/grafana/mcp-k6) and reads
Splunk through the [official Splunk MCP Server](https://splunkbase.splunk.com/app/7931). Full
setup: [`docs/SPLUNK_SETUP.md`](docs/SPLUNK_SETUP.md).

```bash
brew install k6 && kassi warm-k6   # install k6 2.0+; warms the extension cache on first use
# alternatives: standalone binary (KASSI_K6_CMD=mcp-k6) or Docker (KASSI_K6_DOCKER=1)
```

The Splunk step is optional: without `KASSI_SPLUNK_MCP_ENDPOINT` + `KASSI_SPLUNK_TOKEN`, kassi
skips correlation and runs k6-only. Model backend: see [Configuration](#configuration).

## Quickstart

The fastest first success needs no Splunk, k6, or model. It runs the whole state machine against
fakes:

```bash
uv run pytest        # the full workflow against Theodosia's FakeUpstream and a fake model
kassi render         # print the state machine
kassi arcana         # the tarot spread, one card per phase
```

For a full live run against a bundled demo app (starts the app, drives real k6, queries live
Splunk, writes the grounded analysis):

```bash
uv run python scripts/verify_scenario.py petclinic   # or storefront | feed | gateway | orders
```

## Usage

Inspect and serve the workflow:

```bash
kassi doctor --runtime     # validate the graph and runtime tool shape
kassi render               # print the state machine
kassi serve                # mount as an MCP server over stdio (both upstreams wired in)
```

Drive it locally, no cloud agent. `kassi pilot` lets a local open model drive the FSM step by step:
it reads the reachable actions and calls `step` for each phase itself, doing the per-phase work as it
goes (the `screen` phase hands off to an independent auditor). Driver, writer, and auditor are all
the local model:

```bash
kassi pilot --intent "load test the pet listing endpoint" \
  --repo-path examples/petstore --target-base-url http://localhost:8000 --splunk-index web
# or diff mode: kassi pilot --repo-path /path/to/repo --ref HEAD~1 --splunk-index web
```

Run it in the background, triggered on diff detection. `kassi watch` polls a repo's git HEAD and,
when a new commit changes an HTTP endpoint, drives the whole workflow in diff mode against that
change, then prints the verdict and a proposed fix and publishes the run to Splunk:

```bash
kassi watch --repo-path /path/to/repo --target-base-url http://localhost:8000 --splunk-index web
# one-shot for a post-commit hook or CI: kassi watch --once --repo-path . --target-base-url ...
```

Or drive it from Claude Code (or any MCP client) by registering the server:

```bash
claude mcp add --scope=user --transport=stdio kassi -- kassi serve
```

Then ask the agent to run the workflow with the `step` tool, for example:
"Use the kassi step tool. Load test the pet listing endpoint against
http://localhost:8000; the spec is under examples/petstore; correlate with Splunk
index web."

The entry inputs for `select_mode`:

- diff mode: `{"repo_path": "/path/to/repo", "ref": "HEAD~1", "target_base_url": "http://localhost:8000", "splunk_index": "web"}`
- intent mode: `{"repo_path": "/path/with/openapi.json", "intent": "load test the checkout endpoint", "target_base_url": "...", "splunk_index": "web"}`

Review recorded runs:

```bash
kassi sessions ls
kassi sessions show <app-id>
kassi logs <app-id> --refusals
kassi verify <app-id>        # confirm the ledger has not been tampered with
```

## Configuration

| Variable | Default | Purpose |
| --- | --- | --- |
| `KASSI_LLM` | `ollama` | model backend: `ollama` (local), `claude_agent` (Claude via the Claude Code session, no API key), or `anthropic` (Claude Messages API) |
| `KASSI_MODEL` | `granite4.1:8b` / `sonnet` | model tag: an Ollama tag, or a Claude model alias/id for the Claude backends |
| `OLLAMA_HOST` | `http://localhost:11434` | Ollama endpoint (point at the host running the local model, e.g. a LAN box) |
| `ANTHROPIC_API_KEY` | unset | Claude API key (only for `KASSI_LLM=anthropic`; `claude_agent` uses the logged-in Claude Code session instead) |
| `KASSI_K6_CMD` | `k6 x mcp` | command line for the k6 MCP server (set to `mcp-k6` for the standalone binary) |
| `KASSI_K6_DOCKER` | unset | if set, run the k6 MCP server via Docker |
| `KASSI_K6_IMAGE` | `grafana/mcp-k6:latest` | Docker image when `KASSI_K6_DOCKER` is set |
| `KASSI_SPLUNK_MCP_ENDPOINT` | unset | streamable-HTTP endpoint of the Splunk MCP Server (e.g. `https://localhost:8089/services/mcp`) |
| `KASSI_SPLUNK_TOKEN` | unset | encrypted MCP token (sent as `Authorization: Bearer`) |
| `KASSI_SPLUNK_MCP_CMD` | `npx` | stdio bridge command (runs `mcp-remote`) |
| `KASSI_SPLUNK_INSECURE` | unset | skip TLS verification in the bridge (local self-signed Splunk only) |
| `THEODOSIA_HOME` | `~/.kassi` | ledger / session store |

`kassi serve` loads these from a `.env` in the project root if present (see
`.env.example`); real environment variables take precedence. Keep `.env` out of git
(it is git-ignored) since the token is a credential.

When running the k6 server in Docker, a target on the host is reachable as
`http://host.docker.internal:<port>` from inside the container.

## How it works

```mermaid
stateDiagram-v2
    [*] --> Initialize

    state "Initialization" as Initialize {
        [*] --> LoadConfig
        LoadConfig --> ValidateEnvironment
        ValidateEnvironment --> DetectExecutionMode
        DetectExecutionMode --> Ready
        ValidateEnvironment --> FatalError: Invalid Config
    }

    Ready --> InputProcessing

    state "Input Processing" as InputProcessing {
        [*] --> SelectMode

        SelectMode --> ReadDiff : Diff Mode
        SelectMode --> ParseIntent : Intent Mode

        ReadDiff --> ValidateDiff
        ValidateDiff --> ExtractEndpoints
        ValidateDiff --> ParseFailure : Invalid Diff

        ParseIntent --> NLPAnalysis
        NLPAnalysis --> ExtractRequirements
        NLPAnalysis --> ParseFailure : Low Confidence

        ExtractEndpoints --> MergeContext
        ExtractRequirements --> MergeContext

        ParseFailure --> RetryParser
        RetryParser --> SelectMode : Retry
        RetryParser --> Abort : Max Retries
    }

    MergeContext --> KnowledgeDiscovery

    state "Knowledge Discovery" as KnowledgeDiscovery {
        [*] --> LookupDocs
        LookupDocs --> SearchExamples
        SearchExamples --> FetchSchemas
        FetchSchemas --> BuildContext

        LookupDocs --> MissingDocs : No Match
        MissingDocs --> AIInference
        AIInference --> BuildContext
    }

    BuildContext --> Planning

    state "Execution Planning" as Planning {
        [*] --> GeneratePlan
        GeneratePlan --> DependencyAnalysis
        DependencyAnalysis --> RiskAssessment
        RiskAssessment --> PlanValidation

        PlanValidation --> Scaffold
        PlanValidation --> Abort : Critical Risk
    }

    Scaffold --> CodeGeneration

    state "Generation Pipeline" as CodeGeneration {

        [*] --> GenerateScript
        GenerateScript --> StaticAnalysis

        StaticAnalysis --> AutoFix : Issues Found
        AutoFix --> StaticAnalysis

        StaticAnalysis --> SecurityScan
        SecurityScan --> Linting
        Linting --> SyntaxValidation

        SyntaxValidation --> CompileCheck
        CompileCheck --> GenerationSuccess

        CompileCheck --> Regenerate
        Regenerate --> GenerateScript

        Regenerate --> Abort : Max Attempts
    }

    GenerationSuccess --> TestExecution

    state "Execution & Validation" as TestExecution {

        [*] --> PrepareRuntime

        PrepareRuntime --> RunSmokeTests
        RunSmokeTests --> RunFunctionalTests
        RunFunctionalTests --> RunPerformanceTests

        RunPerformanceTests --> EvaluateResults

        EvaluateResults --> RetryExecution : Retryable Failure
        RetryExecution --> RunPerformanceTests

        EvaluateResults --> ValidationSuccess
        EvaluateResults --> ValidationFailure
    }

    ValidationSuccess --> Observability

    state "Observability Pipeline" as Observability {

        [*] --> SplunkPreflight

        SplunkPreflight --> CollectLogs
        CollectLogs --> CorrelateEvents
        CorrelateEvents --> DetectAnomalies

        DetectAnomalies --> RootCauseAnalysis
        RootCauseAnalysis --> MetricsAggregation
    }

    MetricsAggregation --> QualityGate

    state "Quality Gates" as QualityGate {

        [*] --> CoverageCheck
        CoverageCheck --> PerformanceThreshold
        PerformanceThreshold --> SecurityThreshold
        SecurityThreshold --> ReliabilityScore

        ReliabilityScore --> Pass
        ReliabilityScore --> NeedsImprovement
    }

    NeedsImprovement --> CodeGeneration

    Pass --> Reporting

    state "Reporting & Delivery" as Reporting {

        [*] --> GenerateSummary
        GenerateSummary --> GenerateHTMLReport
        GenerateHTMLReport --> ExportArtifacts
        ExportArtifacts --> NotifyUser
        NotifyUser --> Complete
    }

    ValidationFailure --> FailureAnalysis

    state "Failure Analysis" as FailureAnalysis {
        [*] --> CaptureDiagnostics
        CaptureDiagnostics --> RootCause
        RootCause --> SuggestFixes
        SuggestFixes --> GenerateFailureReport
    }

    GenerateFailureReport --> Complete
    Abort --> Complete
    FatalError --> Complete

    Complete --> [*]
```

- `doc_lookup` — consults the k6 MCP documentation tools and records version-grounded citations. Non-blocking; generation proceeds if the docs are unavailable.
- `scaffold` — composes a deterministic, self-contained k6 baseline from the OpenAPI schema (per-endpoint requests with sample bodies, baked base URL, load options). No model. This scaffold is the known-good fallback.
- `generate_script` — the model authors the final script on top of the scaffold, guided by k6's own `generate_script` MCP prompt and `best_practices` resource.
- `validate_script` — gates the script at `k6.validate_script`. Failures route to `fix_script`, which repairs the script from the real k6 error (stderr + the server's structured issues and suggestions) and loops back. Bounded by `MAX_FIX_ATTEMPTS`; gives up cleanly to the scaffold so an unvalidated script never reaches `run_test`.
- `run_test` — executes the validated script via `k6.run_script` and records the wall-clock test window.
- `splunk_preflight` — checks the target index exists, capturing event count, sourcetypes, and Splunk version before correlating. Catches the wrong-index failure early. Non-blocking.
- `correlate` — four windowed SPL queries (rollup, timeline, by-endpoint, dominant error) synthesize client-vs-server findings: which route degraded and why. Answers what the k6 summary alone cannot show.
- `detect_anomalies` — the AI Toolkit's `StateSpaceForecast` forecasts the latency band (falling back to `predict` when the add-on is absent) and `anomalydetection` flags outlying buckets. The saturation onset is found by Splunk's ML, not a fixed threshold. Non-blocking.
- `analyze` — the writer model produces a grounded analysis (root cause, evidence with source citations, recommendation) and, in diff mode, a proposed remediation diff that applies cleanly. Both fall back to deterministic text when no model is available.
- `screen` — an independent auditor model checks every claim in the analysis against the cited telemetry. The pass/fail is sealed to the report. Non-blocking.
- `report` — assembles the combined client-plus-server verdict, seals it to the ledger, and publishes the run and the agent's state-machine walk to Splunk over HEC.

## The Major Arcana

Each phase is a card the agent turns. Run `kassi arcana` for the full spread.

| Card | Phase | Omen |
| --- | --- | --- |
| The Fool (0) | `select_mode` | the querent sets out: diff or intent |
| The High Priestess (II) | `read_diff` | hidden knowledge read from the diff |
| The Emperor (IV) | `extract_endpoints` | order from change: the routes are named |
| The Empress (III) | `parse_intent` | intuition reads the intent into endpoints |
| The Hierophant (V) | `doc_lookup` | doctrine consulted: the k6 docs ground the rite |
| The Chariot (VII) | `scaffold` | the vehicle is assembled from the spec: a runnable scaffold takes shape |
| The Magician (I) | `generate_script` | as above, so below: the agent authors the script atop the scaffold |
| Justice (XI) | `validate_script` | the script is weighed; the unworthy is turned back |
| Temperance (XIV) | `fix_script` | the flawed draft is tempered against k6's judgment until it holds |
| The Tower (XVI) | `run_test` | load strikes the structure; what breaks is revealed |
| The Hermit (IX) | `splunk_preflight` | a lantern into the index before the reading |
| The Lovers (VI) | `correlate` | client and server joined over one window |
| The Star (XVII) | `detect_anomalies` | Splunk's own forecast is cast; where the load breaches the band is revealed |
| The Sun (XIX) | `analyze` | the reading is made plain: cause, evidence, and the cure laid bare |
| The Hanged Man (XII) | `screen` | seen again through another's eyes: the reading is judged grounded, or not |
| Judgement (XX) | `report` | the verdict is spoken and sealed to the ledger |
| The World (XXI) | the ledger | the cycle closes: an immutable, hash-chained record |
| The Devil (XV) | a refusal | you are bound: only the legal moves are permitted |

## Case study

Verified end-to-end against Splunk Enterprise 10.4.0 with the official Splunk MCP Server
(Splunkbase 7931, v1.2.0), called live at runtime. `scripts/verify_petclinic.py` drives the
whole FSM with nothing canned, in diff mode. A throwaway git repo holds a healthy petclinic
baseline plus a second commit that adds `POST /api/visits`, so kassi picks the changed
endpoint from the diff, runs real k6 through the k6 MCP server against it, and reads the
server-side regression back from Splunk through the four `correlate` queries. It also runs the
AI Toolkit's `StateSpaceForecast` (with `predict` as the fallback) and `anomalydetection` over
the same window in `detect_anomalies`, all on the official `splunk_run_query` tool.

```console
$ KASSI_LLM=claude_agent uv run python scripts/verify_petclinic.py
target app:  petclinic (flawed POST /api/visits) at http://127.0.0.1:8400
diff mode:   HEAD~1..HEAD adds POST /api/visits         # kassi tests exactly the changed endpoint
... extract_endpoints_ok count=1                        # one new route, read from the diff
... validation failed (attempt 0): Unexpected token ILLEGAL ... Missing k6 module imports
... fix_script_done attempt=1                           # repaired from the real k6 error
... run_test_ok exit_code=0 reqs=2937
verdict:         server-side regression: /api/visits p95 318.44ms, 59.4% 5xx, cause 'database is locked'
endpoints:       POST /api/visits
k6 client-side:  2937 reqs, p95 318.44 ms, 59.4% failed
worst endpoint:  /api/visits  59.4% errs  p95 318.44 ms
root cause:      database is locked  (1797x)
anomaly scan:    splunk StateSpaceForecast + anomalydetection over 13 buckets, 1 anomalous bucket
mcp tool calls:  k6.{list_sections, get_documentation x4, generate_script(prompt),
                     validate_script x2, run_script}
                 splunk.{get_info, get_index_info, get_metadata, run_query x6}
the reading:
    🂠  The Fool: Diff mode revealed 1 endpoint under scrutiny.
    🂠  The Magician: Script authored, repaired once, validated successfully.
    🂠  The Tower: 2937 requests executed; p95 318.44 ms, 59.4% failure rate.
    🂠  The Lovers: /api/visits worst at 59.4% errors, database locked 1797 times.
    🂠  The Star: StateSpaceForecast forecast p95 ~312ms; anomalydetection flagged the anomalous bucket.
    🂠  Judgement: Server regression confirmed: /api/visits, database lock root cause.
```

What this proves, all at runtime against live Splunk: kassi read the new `POST /api/visits`
from the git diff (the healthy GET routes are untouched by the change, so it tests exactly
the new endpoint), the model authored a script that failed k6 validation, the `fix_script`
loop repaired it from the real k6 error and re-validated, real k6 drove 2937 requests, the
four `correlate` queries on the official Splunk MCP Server isolated the new endpoint
(`/api/visits` at 59.4% 5xx) and named the root cause k6 cannot see ("database is locked"),
and the AI Toolkit's `StateSpaceForecast` forecast the latency band while `anomalydetection`
flagged the anomalous bucket statistically.
Every upstream call is on the hash-chained ledger and in `mcp_provenance`. See
[`docs/SPLUNK_SETUP.md`](docs/SPLUNK_SETUP.md) for the full setup.

For a lighter reproduction without a target app, `scripts/verify_correlate_live.py` cans the
k6 metrics and ingests sample telemetry, but still queries the real official Splunk MCP Server.

## Demo scenarios

`examples/` ships five target apps, each a healthy baseline plus one "new" endpoint with a
distinct load-induced failure, so the suite spans the common regression classes (not just one
trick). Each ships the same `access_json` telemetry to Splunk, so kassi correlates them
unchanged; the failure only appears under concurrency.

| App | New endpoint | Failure signature | What it exercises |
| --- | --- | --- | --- |
| `petclinic` | `POST /api/visits` | 5xx, constant, `database is locked` | correlation isolates the root-cause error |
| `storefront` | `POST /api/checkout` | latency, **0 errors** (N+1 over a shared connection) | server-side `db_time`, invisible to the client error rate |
| `feed` | `POST /api/events` | latency **rising over the run** (unbounded recompute) | `detect_anomalies`: the forecast band and anomalydetection catch the trend |
| `gateway` | `GET /api/quote` | **4xx** 429 throttling (too-tight rate limit) | client-vs-server error split: "throttled, not broken" |
| `orders` | `POST /api/order` | latency **+ 504 mix** (downstream cascade) | dependency root cause, resilience recommendation |

Run any of them end-to-end (starts the app, real k6, live Splunk, the grounded analysis):

```bash
uv run python scripts/verify_scenario.py feed   # or petclinic | storefront | gateway | orders
```

`petclinic` is also the headline diff-mode run above.

## Benchmark

`kassi-bench` scores kassi against ground truth: 80 live runs over five change-induced fault classes
(5xx regression, two latency degradations, 4xx throttling, a 504 cascade) plus three healthy
controls, ten reps each, with the model authoring the load test and analysis and the verdict computed
deterministically from Splunk. Across the faults kassi detects 90%, localizes 92%, classifies 90%,
and names the root cause 95%; the controls hold a 0% false-alarm rate. A second suite,
`kassi-bench-ext`, runs kassi against go-httpbin (a third-party app it never instrumented, observed
through a generic access-log proxy) and scores 15/15. Against the canonical academic benchmark
**RCAEval RE3** (code-level faults in Online Boutique and Train Ticket), kassi's diagnosis engine
localizes the root-cause service at top-1 in **81%** of 57 cases (90% on Online Boutique) and within
top-3 in **100%**, competitive with the strongest published methods and well ahead of the classical
baselines. Between them the suites found and fixed three
real bugs: a missing throttling branch, a too-low latency floor, and a validation step that read a
crossed k6 threshold as a broken script. Full methodology and tables:
[`docs/benchmark/BENCHMARK.md`](docs/benchmark/BENCHMARK.md).

```bash
uv run python scripts/benchmark.py --reps 10            # demo suite -> docs/benchmark/results.json
docker run -d -p 8600:8080 ghcr.io/mccutchen/go-httpbin
uv run python scripts/benchmark_external.py --reps 5    # external suite (go-httpbin via the proxy)
uv run python scripts/benchmark_rcaeval.py --systems OB,TT   # canonical RCAEval RE3 (after download)
uv run python scripts/benchmark_report.py               # regenerates the report
```

## Dashboard

<p align="center"><img src="docs/assets/shot-dashboard.png" alt="kassi dashboard: change impact read from Splunk" width="820" /></p>

The `report` phase publishes each run to `index=kassi_runs` over HEC: a `kassi:run` summary
(verdict, k6 client metrics, the server-side correlation, the forecast, the root cause) and,
because the agent's own execution is telemetry too, one `kassi:step` event per state-machine
phase, the agent's walk from The Fool to Judgement, keyed by Burr's `app_id`. So the dashboard
shows both what the change did and how the agent reached the verdict. The agent stays CLI/MCP;
Splunk is the reporting surface. Provision it once:

```bash
uv run python scripts/setup_dashboard.py   # creates the index, an HEC token, and the dashboard
# paste the printed KASSI_HEC_TOKEN into .env, then every run self-publishes
```

The dashboard (`docs/dashboard/kassi_overview.xml`) shows the latest verdict, the agent's
state-machine walk for that run (each phase, its outcome, and the tool calls it made), step
outcomes across runs, k6 vs server-side p95, p95 across runs, server-side errors by endpoint,
and a run-history table. Gated on `KASSI_HEC_TOKEN`: with it unset, runs simply don't publish.

## Development

```bash
uv run ruff format . && uv run ruff check .
uv run pytest
```

The tests use Theodosia's `FakeUpstream` for both MCP servers and a fake LLM, so
they run offline with no k6, Splunk, Ollama, or network.

### Local Splunk

[`docs/SPLUNK_SETUP.md`](docs/SPLUNK_SETUP.md) walks through running Splunk Enterprise
locally, seeding sample telemetry, and verifying the integration. The two helper scripts:

```bash
uv run python scripts/seed_splunk.py            # index + HEC + sample data + verify the SPL kassi emits
uv run python scripts/verify_correlate_live.py  # drive the whole FSM; correlate hits live Splunk
KASSI_LLM=claude_agent uv run python scripts/verify_petclinic.py  # the real-app root-cause demo
```

`verify_petclinic.py` is the headline demo, nothing canned: it starts the
[`examples/petclinic`](examples/petclinic) app (a healthy baseline plus a new
`POST /api/visits` with a SQLite write-lock flaw that only bites under load, shipping
access logs to Splunk's HEC), runs real k6 through the k6 MCP server, and reads the
server-side regression back from Splunk: which endpoint, how bad, and the "database is
locked" root cause k6 alone can't see.

`scripts/dev_splunk_mcp.py` is a local stdio MCP bridge to Splunk REST, used only to
exercise the correlate path without the official app. Production uses the official Splunk
MCP Server via `KASSI_SPLUNK_MCP_ENDPOINT` + `KASSI_SPLUNK_TOKEN`. See the [Case
study](#case-study) for a verified run.