kassi-CLI

Divines disaster, crafts the cure.

kassi is an AI agent that load-tests a code change, finds the regression in live Splunk, and writes the fix, before it reaches production. Point it at a git diff or a plain-language intent. It drives real k6 load against the affected endpoints, correlates the result with server-side telemetry from the official Splunk MCP Server, names the root cause, and hands back a validated remediation diff. For engineering and SRE teams who need to catch load-induced regressions before a 2am page.

kassi runs as an audited Burr state machine over MCP, served by Theodosia: the driving agent sees one tool, illegal steps are refused, and every step and refusal lands on a hash-chained ledger. Named for Kassandra, who foresaw what others would not believe; the workflow is themed as a tarot draw, one Major Arcana card per phase (kassi arcana lays out the spread).

Built for the Splunk Agentic Ops Hackathon (Observability track). See DEVPOST.md for the writeup and architecture_diagram.md for the design.

The demo above is a recorded end-to-end run: it prints the state machine, then drives the whole workflow (script + analysis from the configured model, k6 docs + run, Splunk preflight and correlation) against a live Splunk.

Screenshots

What it does

• Diff or intent driven. Reads the changed endpoints from a git diff, or scores an OpenAPI spec against a plain-language intent.

• Real load. Generates and runs an actual k6 test through the Grafana k6 MCP server, on top of a deterministic scaffold so a run never fails for lack of a model.

• Server-side truth from Splunk. Correlates the run with windowed SPL through the official Splunk MCP Server, then runs the Splunk AI Toolkit (StateSpaceForecast + anomalydetection) to locate the saturation onset statistically. Catches latency degradations that produce zero errors and would slip past any threshold alert.

• Root cause and a fix. Writes a cited analysis and a validated remediation diff that applies cleanly, screened by an independent auditor model before the verdict is sealed.

• Audited by construction. A governed state machine refuses illegal steps; every step and refusal is on a hash-chained ledger that kassi verify checks.

• Hands-free guard. kassi watch runs the whole workflow when a commit changes an endpoint, catching the regression at commit time.

• Model-agnostic. The same loop (drive, write, audit) runs on a local open 8B, on-prem and air-gapped, or on a frontier model, unchanged.

• Self-observable. Publishes its own state-machine walk back to Splunk, so the agent is visible in the system it observes.

• Measured. 0% false alarms on a live ground-truth benchmark; root cause in the top 3 100% of the time on RCAEval RE3 (details).

Related MCP server: AIOps MCP

Install

uv sync

kassi drives k6 through the Grafana k6 MCP server and reads Splunk through the official Splunk MCP Server. Full setup: docs/SPLUNK_SETUP.md.

brew install k6 && kassi warm-k6   # install k6 2.0+; warms the extension cache on first use
# alternatives: standalone binary (KASSI_K6_CMD=mcp-k6) or Docker (KASSI_K6_DOCKER=1)

The Splunk step is optional: without KASSI_SPLUNK_MCP_ENDPOINT + KASSI_SPLUNK_TOKEN, kassi skips correlation and runs k6-only. Model backend: see Configuration.

Quickstart

The fastest first success needs no Splunk, k6, or model. It runs the whole state machine against fakes:

uv run pytest        # the full workflow against Theodosia's FakeUpstream and a fake model
kassi render         # print the state machine
kassi arcana         # the tarot spread, one card per phase

For a full live run against a bundled demo app (starts the app, drives real k6, queries live Splunk, writes the grounded analysis):

uv run python scripts/verify_scenario.py petclinic   # or storefront | feed | gateway | orders

Usage

Inspect and serve the workflow:

kassi doctor --runtime     # validate the graph and runtime tool shape
kassi render               # print the state machine
kassi serve                # mount as an MCP server over stdio (both upstreams wired in)

Drive it locally, no cloud agent. kassi pilot lets a local open model drive the FSM step by step: it reads the reachable actions and calls step for each phase itself, doing the per-phase work as it goes (the screen phase hands off to an independent auditor). Driver, writer, and auditor are all the local model:

kassi pilot --intent "load test the pet listing endpoint" \
  --repo-path examples/petstore --target-base-url http://localhost:8000 --splunk-index web
# or diff mode: kassi pilot --repo-path /path/to/repo --ref HEAD~1 --splunk-index web

Run it in the background, triggered on diff detection. kassi watch polls a repo's git HEAD and, when a new commit changes an HTTP endpoint, drives the whole workflow in diff mode against that change, then prints the verdict and a proposed fix and publishes the run to Splunk:

kassi watch --repo-path /path/to/repo --target-base-url http://localhost:8000 --splunk-index web
# one-shot for a post-commit hook or CI: kassi watch --once --repo-path . --target-base-url ...

Or drive it from Claude Code (or any MCP client) by registering the server:

claude mcp add --scope=user --transport=stdio kassi -- kassi serve

Then ask the agent to run the workflow with the step tool, for example: "Use the kassi step tool. Load test the pet listing endpoint against http://localhost:8000; the spec is under examples/petstore; correlate with Splunk index web."

The entry inputs for select_mode:

• diff mode: {"repo_path": "/path/to/repo", "ref": "HEAD~1", "target_base_url": "http://localhost:8000", "splunk_index": "web"}

• intent mode: {"repo_path": "/path/with/openapi.json", "intent": "load test the checkout endpoint", "target_base_url": "...", "splunk_index": "web"}

Review recorded runs:

kassi sessions ls
kassi sessions show <app-id>
kassi logs <app-id> --refusals
kassi verify <app-id>        # confirm the ledger has not been tampered with

Configuration

kassi serve loads these from a .env in the project root if present (see .env.example); real environment variables take precedence. Keep .env out of git (it is git-ignored) since the token is a credential.

When running the k6 server in Docker, a target on the host is reachable as http://host.docker.internal:<port> from inside the container.

How it works

stateDiagram-v2
    [*] --> Initialize

    state "Initialization" as Initialize {
        [*] --> LoadConfig
        LoadConfig --> ValidateEnvironment
        ValidateEnvironment --> DetectExecutionMode
        DetectExecutionMode --> Ready
        ValidateEnvironment --> FatalError: Invalid Config
    }

    Ready --> InputProcessing

    state "Input Processing" as InputProcessing {
        [*] --> SelectMode

        SelectMode --> ReadDiff : Diff Mode
        SelectMode --> ParseIntent : Intent Mode

        ReadDiff --> ValidateDiff
        ValidateDiff --> ExtractEndpoints
        ValidateDiff --> ParseFailure : Invalid Diff

        ParseIntent --> NLPAnalysis
        NLPAnalysis --> ExtractRequirements
        NLPAnalysis --> ParseFailure : Low Confidence

        ExtractEndpoints --> MergeContext
        ExtractRequirements --> MergeContext

        ParseFailure --> RetryParser
        RetryParser --> SelectMode : Retry
        RetryParser --> Abort : Max Retries
    }

    MergeContext --> KnowledgeDiscovery

    state "Knowledge Discovery" as KnowledgeDiscovery {
        [*] --> LookupDocs
        LookupDocs --> SearchExamples
        SearchExamples --> FetchSchemas
        FetchSchemas --> BuildContext

        LookupDocs --> MissingDocs : No Match
        MissingDocs --> AIInference
        AIInference --> BuildContext
    }

    BuildContext --> Planning

    state "Execution Planning" as Planning {
        [*] --> GeneratePlan
        GeneratePlan --> DependencyAnalysis
        DependencyAnalysis --> RiskAssessment
        RiskAssessment --> PlanValidation

        PlanValidation --> Scaffold
        PlanValidation --> Abort : Critical Risk
    }

    Scaffold --> CodeGeneration

    state "Generation Pipeline" as CodeGeneration {

        [*] --> GenerateScript
        GenerateScript --> StaticAnalysis

        StaticAnalysis --> AutoFix : Issues Found
        AutoFix --> StaticAnalysis

        StaticAnalysis --> SecurityScan
        SecurityScan --> Linting
        Linting --> SyntaxValidation

        SyntaxValidation --> CompileCheck
        CompileCheck --> GenerationSuccess

        CompileCheck --> Regenerate
        Regenerate --> GenerateScript

        Regenerate --> Abort : Max Attempts
    }

    GenerationSuccess --> TestExecution

    state "Execution & Validation" as TestExecution {

        [*] --> PrepareRuntime

        PrepareRuntime --> RunSmokeTests
        RunSmokeTests --> RunFunctionalTests
        RunFunctionalTests --> RunPerformanceTests

        RunPerformanceTests --> EvaluateResults

        EvaluateResults --> RetryExecution : Retryable Failure
        RetryExecution --> RunPerformanceTests

        EvaluateResults --> ValidationSuccess
        EvaluateResults --> ValidationFailure
    }

    ValidationSuccess --> Observability

    state "Observability Pipeline" as Observability {

        [*] --> SplunkPreflight

        SplunkPreflight --> CollectLogs
        CollectLogs --> CorrelateEvents
        CorrelateEvents --> DetectAnomalies

        DetectAnomalies --> RootCauseAnalysis
        RootCauseAnalysis --> MetricsAggregation
    }

    MetricsAggregation --> QualityGate

    state "Quality Gates" as QualityGate {

        [*] --> CoverageCheck
        CoverageCheck --> PerformanceThreshold
        PerformanceThreshold --> SecurityThreshold
        SecurityThreshold --> ReliabilityScore

        ReliabilityScore --> Pass
        ReliabilityScore --> NeedsImprovement
    }

    NeedsImprovement --> CodeGeneration

    Pass --> Reporting

    state "Reporting & Delivery" as Reporting {

        [*] --> GenerateSummary
        GenerateSummary --> GenerateHTMLReport
        GenerateHTMLReport --> ExportArtifacts
        ExportArtifacts --> NotifyUser
        NotifyUser --> Complete
    }

    ValidationFailure --> FailureAnalysis

    state "Failure Analysis" as FailureAnalysis {
        [*] --> CaptureDiagnostics
        CaptureDiagnostics --> RootCause
        RootCause --> SuggestFixes
        SuggestFixes --> GenerateFailureReport
    }

    GenerateFailureReport --> Complete
    Abort --> Complete
    FatalError --> Complete

    Complete --> [*]

• doc_lookup — consults the k6 MCP documentation tools and records version-grounded citations. Non-blocking; generation proceeds if the docs are unavailable.

• scaffold — composes a deterministic, self-contained k6 baseline from the OpenAPI schema (per-endpoint requests with sample bodies, baked base URL, load options). No model. This scaffold is the known-good fallback.

• generate_script — the model authors the final script on top of the scaffold, guided by k6's own generate_script MCP prompt and best_practices resource.

• validate_script — gates the script at k6.validate_script. Failures route to fix_script, which repairs the script from the real k6 error (stderr + the server's structured issues and suggestions) and loops back. Bounded by MAX_FIX_ATTEMPTS; gives up cleanly to the scaffold so an unvalidated script never reaches run_test.

• run_test — executes the validated script via k6.run_script and records the wall-clock test window.

• splunk_preflight — checks the target index exists, capturing event count, sourcetypes, and Splunk version before correlating. Catches the wrong-index failure early. Non-blocking.

• correlate — four windowed SPL queries (rollup, timeline, by-endpoint, dominant error) synthesize client-vs-server findings: which route degraded and why. Answers what the k6 summary alone cannot show.

• detect_anomalies — the AI Toolkit's StateSpaceForecast forecasts the latency band (falling back to predict when the add-on is absent) and anomalydetection flags outlying buckets. The saturation onset is found by Splunk's ML, not a fixed threshold. Non-blocking.

• analyze — the writer model produces a grounded analysis (root cause, evidence with source citations, recommendation) and, in diff mode, a proposed remediation diff that applies cleanly. Both fall back to deterministic text when no model is available.

• screen — an independent auditor model checks every claim in the analysis against the cited telemetry. The pass/fail is sealed to the report. Non-blocking.

• report — assembles the combined client-plus-server verdict, seals it to the ledger, and publishes the run and the agent's state-machine walk to Splunk over HEC.

The Major Arcana

Each phase is a card the agent turns. Run kassi arcana for the full spread.

Case study

Verified end-to-end against Splunk Enterprise 10.4.0 with the official Splunk MCP Server (Splunkbase 7931, v1.2.0), called live at runtime. scripts/verify_petclinic.py drives the whole FSM with nothing canned, in diff mode. A throwaway git repo holds a healthy petclinic baseline plus a second commit that adds POST /api/visits, so kassi picks the changed endpoint from the diff, runs real k6 through the k6 MCP server against it, and reads the server-side regression back from Splunk through the four correlate queries. It also runs the AI Toolkit's StateSpaceForecast (with predict as the fallback) and anomalydetection over the same window in detect_anomalies, all on the official splunk_run_query tool.

$ KASSI_LLM=claude_agent uv run python scripts/verify_petclinic.py
target app:  petclinic (flawed POST /api/visits) at http://127.0.0.1:8400
diff mode:   HEAD~1..HEAD adds POST /api/visits         # kassi tests exactly the changed endpoint
... extract_endpoints_ok count=1                        # one new route, read from the diff
... validation failed (attempt 0): Unexpected token ILLEGAL ... Missing k6 module imports
... fix_script_done attempt=1                           # repaired from the real k6 error
... run_test_ok exit_code=0 reqs=2937
verdict:         server-side regression: /api/visits p95 318.44ms, 59.4% 5xx, cause 'database is locked'
endpoints:       POST /api/visits
k6 client-side:  2937 reqs, p95 318.44 ms, 59.4% failed
worst endpoint:  /api/visits  59.4% errs  p95 318.44 ms
root cause:      database is locked  (1797x)
anomaly scan:    splunk StateSpaceForecast + anomalydetection over 13 buckets, 1 anomalous bucket
mcp tool calls:  k6.{list_sections, get_documentation x4, generate_script(prompt),
                     validate_script x2, run_script}
                 splunk.{get_info, get_index_info, get_metadata, run_query x6}
the reading:
    🂠  The Fool: Diff mode revealed 1 endpoint under scrutiny.
    🂠  The Magician: Script authored, repaired once, validated successfully.
    🂠  The Tower: 2937 requests executed; p95 318.44 ms, 59.4% failure rate.
    🂠  The Lovers: /api/visits worst at 59.4% errors, database locked 1797 times.
    🂠  The Star: StateSpaceForecast forecast p95 ~312ms; anomalydetection flagged the anomalous bucket.
    🂠  Judgement: Server regression confirmed: /api/visits, database lock root cause.

What this proves, all at runtime against live Splunk: kassi read the new POST /api/visits from the git diff (the healthy GET routes are untouched by the change, so it tests exactly the new endpoint), the model authored a script that failed k6 validation, the fix_script loop repaired it from the real k6 error and re-validated, real k6 drove 2937 requests, the four correlate queries on the official Splunk MCP Server isolated the new endpoint (/api/visits at 59.4% 5xx) and named the root cause k6 cannot see ("database is locked"), and the AI Toolkit's StateSpaceForecast forecast the latency band while anomalydetection flagged the anomalous bucket statistically. Every upstream call is on the hash-chained ledger and in mcp_provenance. See docs/SPLUNK_SETUP.md for the full setup.

For a lighter reproduction without a target app, scripts/verify_correlate_live.py cans the k6 metrics and ingests sample telemetry, but still queries the real official Splunk MCP Server.

Demo scenarios

examples/ ships five target apps, each a healthy baseline plus one "new" endpoint with a distinct load-induced failure, so the suite spans the common regression classes (not just one trick). Each ships the same access_json telemetry to Splunk, so kassi correlates them unchanged; the failure only appears under concurrency.

Run any of them end-to-end (starts the app, real k6, live Splunk, the grounded analysis):

uv run python scripts/verify_scenario.py feed   # or petclinic | storefront | gateway | orders

petclinic is also the headline diff-mode run above.

Benchmark

kassi-bench scores kassi against ground truth: 80 live runs over five change-induced fault classes (5xx regression, two latency degradations, 4xx throttling, a 504 cascade) plus three healthy controls, ten reps each, with the model authoring the load test and analysis and the verdict computed deterministically from Splunk. Across the faults kassi detects 90%, localizes 92%, classifies 90%, and names the root cause 95%; the controls hold a 0% false-alarm rate. A second suite, kassi-bench-ext, runs kassi against go-httpbin (a third-party app it never instrumented, observed through a generic access-log proxy) and scores 15/15. Against the canonical academic benchmark RCAEval RE3 (code-level faults in Online Boutique and Train Ticket), kassi's diagnosis engine localizes the root-cause service at top-1 in 81% of 57 cases (90% on Online Boutique) and within top-3 in 100%, competitive with the strongest published methods and well ahead of the classical baselines. Between them the suites found and fixed three real bugs: a missing throttling branch, a too-low latency floor, and a validation step that read a crossed k6 threshold as a broken script. Full methodology and tables: docs/benchmark/BENCHMARK.md.

uv run python scripts/benchmark.py --reps 10            # demo suite -> docs/benchmark/results.json
docker run -d -p 8600:8080 ghcr.io/mccutchen/go-httpbin
uv run python scripts/benchmark_external.py --reps 5    # external suite (go-httpbin via the proxy)
uv run python scripts/benchmark_rcaeval.py --systems OB,TT   # canonical RCAEval RE3 (after download)
uv run python scripts/benchmark_report.py               # regenerates the report

Dashboard

The report phase publishes each run to index=kassi_runs over HEC: a kassi:run summary (verdict, k6 client metrics, the server-side correlation, the forecast, the root cause) and, because the agent's own execution is telemetry too, one kassi:step event per state-machine phase, the agent's walk from The Fool to Judgement, keyed by Burr's app_id. So the dashboard shows both what the change did and how the agent reached the verdict. The agent stays CLI/MCP; Splunk is the reporting surface. Provision it once:

uv run python scripts/setup_dashboard.py   # creates the index, an HEC token, and the dashboard
# paste the printed KASSI_HEC_TOKEN into .env, then every run self-publishes

The dashboard (docs/dashboard/kassi_overview.xml) shows the latest verdict, the agent's state-machine walk for that run (each phase, its outcome, and the tool calls it made), step outcomes across runs, k6 vs server-side p95, p95 across runs, server-side errors by endpoint, and a run-history table. Gated on KASSI_HEC_TOKEN: with it unset, runs simply don't publish.

Development

uv run ruff format . && uv run ruff check .
uv run pytest

The tests use Theodosia's FakeUpstream for both MCP servers and a fake LLM, so they run offline with no k6, Splunk, Ollama, or network.

Local Splunk

docs/SPLUNK_SETUP.md walks through running Splunk Enterprise locally, seeding sample telemetry, and verifying the integration. The two helper scripts:

uv run python scripts/seed_splunk.py            # index + HEC + sample data + verify the SPL kassi emits
uv run python scripts/verify_correlate_live.py  # drive the whole FSM; correlate hits live Splunk
KASSI_LLM=claude_agent uv run python scripts/verify_petclinic.py  # the real-app root-cause demo

verify_petclinic.py is the headline demo, nothing canned: it starts the examples/petclinic app (a healthy baseline plus a new POST /api/visits with a SQLite write-lock flaw that only bites under load, shipping access logs to Splunk's HEC), runs real k6 through the k6 MCP server, and reads the server-side regression back from Splunk: which endpoint, how bad, and the "database is locked" root cause k6 alone can't see.

scripts/dev_splunk_mcp.py is a local stdio MCP bridge to Splunk REST, used only to exercise the correlate path without the official app. Production uses the official Splunk MCP Server via KASSI_SPLUNK_MCP_ENDPOINT + KASSI_SPLUNK_TOKEN. See the Case study for a verified run.