Incident Lens

An MCP server that gives Claude role-aware, evidence-backed tools over the BGL, OpenStack and Zookeeper logs in SereneDB. The database searches and calculates; Claude explains; every number traces to a query that actually ran - and a tamper-evident ledger can prove it after the fact, not just assert it.

Three systems, three roles, and a manager answer that ranks them against each other instead of three separate dashboards each shouting "look at me first."

National AI Hackathon 2026 (SchubKI) · challenge "Making Sense of Logs" · partner SereneDB

Judging this submission? Jump straight to How this meets the brief for a one-line-per-criterion map from the rubric to exactly where the evidence lives - including both special awards.

What it does

Five decisions set this apart from a generic log-search wrapper:

1. Tool descriptions written as specs, each with hard limits and an explicit "do not use for."

2. An evidence ledger. Every tool call is stored (SQL, filters, scope, counts, truncation, result) and returns an evidence_id.

3. Entries vs. episodes. Bursts are collapsed with a stated gap rule, so a count of lines is never presented as a count of incidents.

4. A four-block answer contract (Observed / Hypotheses / Recommended checks / Unknown), delivered as MCP prompts per role.

5. Concentration over frequency. concentration_anomalies ranks failures by how confined they are to one node/service, not how often they fire, because the two rank differently: a real hardware fault with 14,481 occurrences on exactly one of this system's 1,024+ nodes sits at rank #33 in a plain frequency list, invisible past the default view, and first by concentration.

flowchart LR
  C[Claude Desktop / Claude Code] -- MCP --> S[Incident Lens MCP server]
  S -- parameterised SQL, read-only role --> D[(SereneDB: inverted index + columnar aggregation)]
  S --> L[(Evidence ledger, SQLite)]
  P[Role prompts + answer contract] --> C

Related MCP server: RecoSearch

Quickstart (clean checkout)

Requires Python 3.10+, curl, unzip. The script starts SereneDB itself either way: natively on Linux x86_64, or via the official Docker image everywhere else (macOS, Windows, Linux ARM) - just have Docker installed and running. If neither is available, install SereneDB from https://serenedb.com/download and start it on 127.0.0.1:7890 first.

git clone https://github.com/ApparentlyTejas/incident-lens.git && cd incident-lens
./scripts/setup.sh --sample      # seconds: 2,000-line Loghub samples, good for trying everything
./scripts/setup.sh               # full corpus: downloads BGL (709 MB) + OpenStack (58 MB) + Zookeeper (10 MB) from Zenodo, parses, loads, indexes
pytest -q --asyncio-mode=auto tests    # needs the --sample load (62 tests, 99% coverage, numbers checked against plain SQL)
python3 tests/stdio_smoke.py           # spawns the server like Claude does and calls a tool

setup.sh = start SereneDB → download (curl -L ...?download=1) → prep/parse_*.py (raw log → Parquet) → scripts/load.py (sql/schema.sql, ownership.sql, indexes.sql, readonly_role.sql; prints load and index-build timings).

Connection: the server uses the read-only role il_reader (override with SERENEDB_DSN); loading uses the admin role (SERENEDB_ADMIN_DSN, default postgres@127.0.0.1:7890). SereneDB speaks the Postgres wire protocol, so psql -h 127.0.0.1 -p 7890 -U postgres works too.

CI, lint, type-check, coverage

.github/workflows/ci.yml runs on every push/PR to main: it stands up SereneDB on the ubuntu-latest runner (Linux x86_64, so start_serenedb.sh takes the native-binary path, no Docker in CI), loads the --sample corpus, then runs lint, type-check, the full test suite with coverage, and the stdio smoke test - the same four checks below, in the same order, that you can run locally:

ruff check .                                  # lint - rules + ignores explained in pyproject.toml
mypy server                                   # type-check the MCP server package itself
pytest -q --asyncio-mode=auto --cov=server --cov-report=term-missing --cov-report=xml --cov-report=html tests
python3 tests/stdio_smoke.py

--cov=server only (not cli/ui/eval/prep/scripts, which are demo/tooling, not the shipped package); current coverage is 99%. CI uploads coverage.xml + htmlcov/ as a workflow artifact - no third-party coverage service wired in. Lint/type-check scope is deliberately narrow: ruff ignores a handful of pycodestyle rules (E501/E701/E702/E402/E401, see pyproject.toml for why - this codebase's dense one-liner/combined-import style is intentional, not an oversight) and mypy only checks server/, the package Claude actually imports.

Connect Claude

Use absolute paths (run_server.py works from any directory). Use the same python3 that scripts/setup.sh used to install the requirements - if your system's plain python3 was too old and you ran setup with python3.11/python3.12 instead (see the Quickstart note above), use that exact same command here too, not plain python3, or the server will fail to start with ModuleNotFoundError: No module named 'mcp'.

Claude Code

claude mcp add incident-lens -- python3 /ABS/PATH/incident-lens/run_server.py

Claude Desktop - add to claude_desktop_config.json:

{ "mcpServers": { "incident-lens": { "command": "python3", "args": ["/ABS/PATH/incident-lens/run_server.py"] } } }

Optional HTTP transport: python3 run_server.py --http.

Then start with a role prompt (engineer_investigation, lead_review, manager_priorities appear as MCP prompts) or just ask, e.g. "Find 'parity error' in BGL and show what happened around the first hit." Terminal fallback that calls the same tools through the API: python3 cli/ask.py --role lead "Which teams had the most severe entries?".

Web console

export ANTHROPIC_API_KEY=... && python3 ui/server.py then open http://127.0.0.1:8787. Same role prompts and the same 13 tools as Claude Code/Desktop, called through the Anthropic API like cli/ask.py - this is a UI layer, not a second implementation. Pick a role and a dataset (BGL, OpenStack or Zookeeper) and chat on the left; every tool call Claude makes lands on the right as it happens, with its evidence_id, the exact SQL that ran, and its position in the hash-chained ledger. Follow-up questions keep real conversation context (client-accumulated, server-validated), and switching role or dataset starts a fresh conversation rather than quietly reusing the old one's context. A "Simulate Tamper" button clones the ledger, hand-edits one stored number, and shows verify_ledger_integrity catch it - on a scratch copy, so it never touches the real ledger. Without a key, the chat panel reports that clearly; the overview and ledger panels (/api/overview, /api/ledger/status) work regardless, since they only talk to SereneDB and the ledger.

Every number in that screenshot is real: 4,747,945 BGL rows, the actual search_logs/nearby_events evidence ids (E-2424, E-2425), their real hashes, chained to the entry before them.

sequenceDiagram
    participant U as User
    participant C as Claude
    participant S as Incident Lens (MCP server)
    participant D as SereneDB
    participant L as Evidence ledger

    U->>C: "Find 'parity error' in BGL..."
    C->>S: search_logs(dataset=bgl, query=...)
    S->>D: parameterised SQL (read-only role)
    D-->>S: rows + total_matches
    S->>L: record SQL, scope, counts, hash
    L-->>S: evidence_id (e.g. E-2424)
    S-->>C: result + evidence_id
    C->>U: Observed / Hypotheses / Checks / Unknown, citing [E-2424]

Tools

Full generated reference with every input and description: docs/TOOLS.md.

Every result uses one envelope: evidence_id, scope (dataset, absolute period), filters, the executed sql (parameters inlined for replay), total_matches, returned, truncated, rows/aggregates, and a note that log text is untrusted. Guardrails: parameterised queries only, read-only DB role, default 20 / hard cap 100 rows, 30 s statement timeout, no free-form SQL from the agent. Bad input (an unknown group_by, a malformed date, an unknown evidence_id) comes back to Claude as a readable error naming the problem, not a bare "tool failed" - the MCP SDK strips detail from unexpected exceptions by design, so every deliberate input check raises ToolError instead of a plain exception.

Code architecture

How a tool call moves through the codebase (complements the request sequence diagram above, which is about the network call; this is about the module boundaries):

flowchart TB
  A["app.py<br/>MCP tool definitions, ValueError → ToolError"] --> T["tools.py<br/>13 tool implementations"]
  CFG["config.py<br/>DATASETS, limits"] --> T
  T --> CM["common.py<br/>dataset_cfg, where/column_filters, Tracer"]
  CM --> DB["db.py<br/>SereneDB connection, read-only role, retry"]
  T --> FIN["common.finish()<br/>envelope + ledger.record"]
  FIN --> LG["ledger.py<br/>hash-chained SQLite ledger"]
  PR["prompts.py<br/>role prompt templates"] --> A
  DB --> SDB[("SereneDB")]
  LG --> SQ[("ledger.db")]

tools.py never talks to SereneDB directly - every query goes through common.py's where()/column_filters() to build parameterised SQL, and a Tracer records exactly what ran so common.finish() can hand that to the ledger unmodified. This is what makes the evidence ledger trustworthy: it isn't a log a tool writes to afterward, it's fed the literal SQL the Tracer captured.

Data, schema, ownership

Datasets: BGL (Blue Gene/L RAS logs, 4.7M lines, 214 days), OpenStack (207,812 nova lines, request ids, inline latency), and Zookeeper (74,380 lines from one real ensemble member, 100% parsed, thread-level quorum/leader-election activity). Tables bgl_logs, openstack_logs, zookeeper_logs, ownership; inverted indexes bgl_idx, os_idx, zk_idx over message text with a lower-case, non-stemming dictionary (identifiers and error codes stay intact). The index also carries ts, level, component, node/service/thread so filters run inside the scan.

The template column is derived by prep/templating.py (numbers, ids, hosts → <*>); it approximates message types and is not Loghub's labelled templates.

Ownership table is fictional, invented for this challenge (sql/ownership.sql): BGL by component - KERNEL → Kernel & Runtime; APP → Application Support; HARDWARE → Hardware Reliability; LINKCARD, SERV_NET → Interconnect & Network; MMCS, CMCS, BGLMASTER, DISCOVERY → Control Plane; MONITOR, DIAGS → Monitoring & Diagnostics. OpenStack by service - nova-api → API Platform; nova-compute → Compute; nova-scheduler → Scheduling. Zookeeper by component (the Java class that logged the line) - quorum/membership classes → Consensus & Membership; leader/follower/learner classes → Leader Election; client-facing I/O classes → Client Connections; request-pipeline classes → Request Processing; transaction-log/snapshot classes → Storage & Persistence; startup classes → Startup & Config. load.py warns about any component without an owner.

Per-team database access (row-level security)

python3 scripts/setup_team_access.py provisions one DB role + one filtered view per team straight from the ownership table - a team's own connection can see only its own component's/service's rows, enforced by SereneDB's grant system, not application code. SereneDB's access-control model is grants and views rather than Postgres-style CREATE POLICY row-level security, so this reaches the same real guarantee with the primitives SereneDB does support: a per-team CREATE VIEW pre-filtered by ownership, a role with its base-table grants revoked, SELECT granted only on that one view. --drop removes everything it created.

This isn't just provisioned and assumed to work - tests/test_team_access.py connects as each of the 15 real team roles (never as admin) and proves: the base tables and the ownership table itself are both denied (InsufficientPrivilege), the team's own view returns the exact row count a plain SQL query against ownership says it should, and every row's component/service belongs to that team and no other.

Huge dataset scale demonstration (bonus)

Two angles for the "Working with Huge Datasets" award, both isolated from the main investigative product (never wired into the tools or role prompts, so neither can affect it either way):

1. A separate thunderbird_logs table (same Loghub HPC-log family as BGL, a different line format). Full file parsed cold: 211,210,954 of 211,212,192 lines (99.9994%), 31.8 GB raw text. Pushed to a verified-stable 8,000,000 loaded rows - real indexed full-text search at sub-2ms and grouped aggregations at sub-16ms at that scale - and the exact memory ceiling on this dev machine's 8 GB of RAM was found and reported precisely, not glossed over: two OOM kills pinned down exactly where indexing more would need more RAM, not more code.

2. 1 billion rows of real OpenTelemetry demo-app logs, queried live off public S3 Parquet with zero download - exactly as the brief describes ("point SereneDB at the URL"). count(*) over the whole file: 305ms. A full group-by over real service/severity text: ~4s. All real numbers, real log lines, no local storage used at all - scales to the full 50 billion rows listed by just adding more file URLs.

Full writeup, the honest memory-ceiling story for Thunderbird, and the real remote-query results: docs/HUGE_DATASET.md.

Answer contract and counting rules

Every answer has Observed (numbers with scope and [E-id]), Hypotheses (labelled), Recommended checks (ordered, with reasons), Unknown (cause, impact, downtime, customer effect). Matching entries are not incidents; truncated rows never establish frequency; periods are absolute dates; no financial loss, downtime or customer impact is ever invented. eval/verify_answer.py checks that every cited id exists, that every number next to an id appears in that ledger entry, and flags uncited numbers.

Evaluation

eval/golden.yaml holds 16 golden questions (5 engineer, 6 lead, 5 manager, including refusal, truncation and prompt-injection cases); eval/ground_truth.py computes reference numbers with plain SQL; eval/run.py drives the golden set through the real tool-calling loop and scores tool-selection accuracy; tests/ checks tool output against plain SQL. A fixture log line that tries to hijack the agent is loaded in --sample mode.

Scale: eval/benchmark.py times every tool against the full 4.7M + 207K-row corpus (15 calls each, cold call excluded) - results, plus a root-cause writeup of a SereneDB query-planning characteristic this testing found (relevance-ranked search's fast top-k scoring path doesn't engage when combined with a time filter, including a provably redundant one - worked around in server/common.py's where(include_ts=...), turning a ~1.5s outlier into ~215ms), are in eval/benchmark_results.md.

Grounded vs. ungrounded: investigation/llm_vs_incident_lens.md asks a fresh, tool-less Claude instance the same investigative question with no access to the data, and compares its (fluent but unverifiable, and wrong about the actual root cause) answer against Incident Lens's cited one.

Status: executed, not just designed

Everything below was actually run against the full real corpus (SereneDB 26.10.1; macOS arm64 via the official Docker image, scripts/start_serenedb.sh falls back to Docker automatically when the native Linux binary isn't available), not just designed on paper.

Core load and tools

• Full load: 4,747,945 BGL rows + 207,812 OpenStack rows + 74,380 Zookeeper rows from the real Zenodo archives, parsed, loaded, indexed. Tables loaded in 3.6-9.2s, inverted indexes built and refreshed in 5.9-9.6s across repeated runs.

• Schema, inverted index, phrase/terms/any/fuzzy search, BM25 + highlight, window functions, percentiles, date_trunc, read-only role: all working at full scale.

• All 13 tools, all 3 MCP resources (schema://, ownership://, evidence://{id}), 45 pytest tests against plain-SQL ground truth, and a real stdio MCP round-trip: all passing, against both the 2k samples and the full corpus.

Clean checkout, exactly as a judge would run it A copy of the exact working tree, in a genuinely fresh Python venv: pip install -r requirements.txt, cold-parsing the sample logs from scratch, scripts/load.py --sample, the full pytest suite, tests/stdio_smoke.py, all passed with zero manual intervention. This is also what caught a real test-design flaw: one new test asserted a property that only holds on the full corpus, now fixed to skip (not silently pass) on small data.

Per-team database access (scripts/setup_team_access.py) Provisioned real roles for all 15 real teams against the full corpus, connected as each one (never as admin), confirmed base-table and ownership-table denial plus exact row-level isolation for every single team. See tests/test_team_access.py.

Worked investigations, all four run against the full corpus:

• BGL recurring failure (investigation/worked_investigation.md): a ciod job-launch failure pattern (1,024 nodes, 52,763 episodes).

• OpenStack (investigation/openstack_investigation.md): a nova-compute image-cache bug (8 occurrences, 8 separate episodes, 8 different VM instances, a deliberately different shape from the BGL case).

• BGL concentration (investigation/concentration_investigation.md): concentration_anomalies surfacing a single-node hardware signal ranked 33rd by raw frequency, including an honest false start where the top-ranked result turned out to be a missing-node-attribution artifact, not a real finding, and saying so rather than reporting it anyway.

• Cross-dataset manager comparison (investigation/cross_dataset_investigation.md): the same priority formula run independently on BGL, OpenStack and Zookeeper finds their most dramatic-looking candidate (a 3,476:1 concentration ratio on Zookeeper) is actually the lowest-risk of the three, a one-time startup burst rather than an ongoing problem, while BGL's far less dramatic-looking pattern is the only one still accelerating.

All four walk real evidence ids, checked with eval/verify_answer.py: every cited id is confirmed to exist, and every directly-measured number is confirmed present in its cited tool result. The two docs with a computed priority-score table also get flagged on the score itself, since it's a product of several cited numbers rather than a single stored field, the checker telling a derived figure apart from a measured one is it working as designed, not a citation gap.

Web console (ui/), every endpoint except the live Claude call Overview, ledger-status, tamper-demo, and evidence-replay all hit over real HTTP; a path-traversal attempt on /static was confirmed blocked; it survived a live docker restart of SereneDB mid-session with no process restart, thanks to a reconnect fix in server/db.py proven through the HTTP layer, not just the Python API.

The Claude tool-calling loop (ui/server.py's run_conversation) Executed with 8 tests (tests/test_ui_server.py) that fake only the Anthropic network response (shaped exactly like the real SDK's content blocks) and run everything else for real: the actual role prompts, the actual tool list, mcp.call_tool against the actual SereneDB corpus, and a real ledger write with a real evidence id. Covers a direct answer, multi-step tool use, generate_report called mid-conversation, a tool error surfacing as a readable message instead of a crash, the step-limit cutoff, multi-turn history actually reaching the model, and malformed history failing cleanly.

The real network call eval/run.py's golden-set runner and cli/ask.py / ui/server.py's actual call to the Anthropic API were run for real: a real ANTHROPIC_API_KEY, a real question, a real multi-step tool-calling run through the web console, the same live run captured in the submitted demo video, not just the faked-client tests and cli/ask.py --dry-run that cover everything up to and after that call.

Resilience and concurrency Connection resilience (server/db.py surviving a SereneDB restart mid-session) was executed twice, once via a deliberate docker restart serenedb and once during routine load testing: a long-lived process self-heals its connection on the next query with no restart needed, confirmed through both the Python API and the UI's HTTP layer. Concurrency was tested with real OS threads hitting different tool calls at once, plus 18 concurrent real HTTP requests across /api/overview, /api/ledger/tamper-demo, and /api/ask, all returning correctly (psycopg3 connections are internally thread-safe; the tamper-demo's file snapshot takes the same lock ledger.record() uses, and runs off the event loop via asyncio.to_thread so it can't stall other requests while holding it).

Engineering rigor: this project was tested against the full real corpus at every stage, not just the 2k-line dev samples - that discipline is what surfaced and fixed over a dozen real issues (mismatched id joins, a vacuously-true flag, a 9x query-planner slowdown, unhandled exceptions surfacing as raw tracebacks, a path-traversal hole, a race condition) across the parsers, the tools, the eval harness and the web console, every one with a regression test. The connection layer is also resilient to a SereneDB process restart at any point - it self-heals automatically on the very next query with no process restart of its own needed, verified both with a deliberate docker restart and during sustained load testing. Full write-up, fix-by-fix: docs/HARDENING.md.

By design, this project only ever claims what the logs can actually prove: episodes are a stated counting convention (gap default 300 s, per node/service), not a guess at real-world incident counts; OpenStack latency is reported only where it's actually measured, on nova-api request lines; priority scores rank observed frequency and spread, on purpose, rather than inventing root cause, downtime or customer impact.

How this meets the brief

Citations

Zhu et al., Loghub: A Large Collection of System Log Datasets for AI-driven Log Analytics, ISSRE 2023. For BGL also: Oliner and Stearley, What Supercomputers Say: A Study of Five System Logs, DSN 2007. Loghub data is research-use; cite when reusing.