game-analytics-agent

An analyst agent for game telemetry. It answers questions in plain English, runs an automatic anomaly sweep over retention, level funnel, economy, acquisition channels, crashes and A/B tests, and writes a self-contained HTML report with charts. Every answer is grounded in SQL you can read.

It ships with a synthetic telemetry generator that can seed five realistic live-ops problems, and an evaluation harness that scores the sweep on whether it finds them without crying wolf on healthy data.

Results

Sweep only (no LLM, deterministic), 3000 players x 30 days, 3 seeds:

recall = 1.00, precision = 1.00 on seeds 0-2, and on held-out seeds 3-4. Full table in examples/eval.md; sample reports in examples/ (open the HTML in a browser).

Related MCP server: G2 Product Analytics MCP

Quick start

pip install -e ".[dev]"
ga sweep                                   # healthy synthetic data -> "No anomalies detected."
ga sweep --issue android_crash_patch       # exits 1 on a critical finding (CI gate)
ga ask "which level is too hard?" --issue level_spike
ga report --issue currency_inflation --ask "is the economy healthy?" --out report.html
ga eval                                    # precision / recall table
pytest                                     # 23 tests

Bring your own data: two CSVs, players.csv and events.csv, with the schema in db.py, then --data path/.

With an LLM

pip install -e ".[llm]"
export ANTHROPIC_API_KEY=...
ga ask "why did revenue fall in the second half of the month?" --llm --show-sql
ga report --llm --out report.html          # findings get a root-cause hypothesis + action

The model gets three tools: run_sql (read-only DuckDB), metric (the vetted queries in metrics.py), and sweep. Every query it runs is kept in the transcript and printed in the report, so a human can audit the answer. Without a key, ask falls back to a keyword router over the same metric functions and says so in the output. Tests and CI never call the API.

A real run with claude-sonnet-5 on data seeded with the Android crash, the level-7 wall and the price-test regression is in examples/report_llm.html (about 2 minutes, 3 questions plus finding narratives). Highlights:

• Asked why did D1 retention drop for recent cohorts?, it pulled retention by cohort, then by platform, then crash rate by platform and version, and concluded: "Android-only, version-triggered regression ... 1.2.0 introduced a severe crash regression on Android only (35% of sessions crash vs 1.1% baseline elsewhere)". That is the seeded cause.

• Asked why did revenue fall in the second half of the month?, it split revenue by platform and by experiment arm, found iOS flat while Android fell 69%, and tied it to the same crash build plus arm B of the price test.

• Asked which country has the best D7 retention and is the difference meaningful?, it computed a two-proportion z-test itself and answered no: KR leads at 10.1% but z is about 1.0 against the lowest country, well within noise.

• Each sweep finding got a hypothesis and an action, for example: "Hotfix or roll back the android 1.2.0 build immediately and monitor D1 retention for cohorts installing after the fix."

As an MCP server

pip install -e ".[mcp]"
GA_ISSUES=level_spike,android_crash_patch ga-mcp

Tools: schema, run_sql, metric, kpis, anomaly_sweep, ask, load_synthetic. Point Claude Code at it with examples/mcp.json and ask "what's wrong with this game?".

How it works

 synth.py      players x days simulation: retention curve, level difficulty,
               currency sources/sinks, IAP, ads, crashes, 5 seedable issues
     |
 db.py         DuckDB warehouse: players, events, sessions view, player_days view
     |
 metrics.py    retention (Dn, by segment, by install cohort), level funnel, economy,
               monetization, crashes, channel quality, headline KPIs
     |
 sweep.py      7 generic checks -> Findings with severity + evidence
     |             change-point on Dn retention by cohort (overall and per platform)
     |             segment vs rest z-test · level vs neighbours · earn/player vs baseline
     |             channel vs others · (segment, version) crash rate · A/B two-proportion z
     |
 agent.py      Analyst: LLM with tools, or offline keyword router; SQL transcript
     |
 report.py     HTML: KPIs, findings, Q&A with queries, 6 charts (matplotlib, embedded)

Checks are generic. They compare a segment, level, day or arm against the rest of the data with sample-size guards and z-tests. None of them reads Issues. Porting to a real game means mapping your events into the schema; the checks and the report come for free.

Attribution, not just detection. The crash bug is reported twice on purpose: as a crash-rate spike on android 1.2.0 and as a D1 retention drop for android cohorts from day 12. The economy check names the source that grew fastest. A finding an analyst can act on names where and since when.

The change-point beat the sliding window. The first retention check compared the last five cohorts with everything earlier. It missed a 16-point Android drop because small daily cohorts are noisy. Scanning every split day and taking the most significant one finds it at day 12, which is exactly the version rollout day.

Design notes and tradeoffs

• Synthetic data with seeded issues makes the evaluation honest and the repo self-contained. The downside is the generator encodes my assumptions about player behaviour; the sweep thresholds are tuned to it and would need re-tuning on real data. Example: the A/B check fires at |z| >= 2.5 because healthy runs peak around 2.2 and seeded regressions start around 2.7. On a real game with more arms or more metrics you would want a multiple-comparison correction.

• Sweep first, LLM second. The statistical checks are cheap and reproducible; the model adds free-form questions and narrative. Keeping the model optional keeps CI free.

• Offline ask is a keyword router, not a model. It exists so the report and tests work without a key. It says [offline] in its output rather than pretending.

• Read-only SQL is enforced by a regex, which is fine for a demo and not for a multi-tenant deployment. DuckDB's read_only connection mode is the right fix when the warehouse is a file.

Roadmap

• Weekly diff mode: what changed since the last report

• Slack/Discord delivery of the findings card

• Ad-hoc segment discovery: let the model propose the cut instead of the fixed list

• Real-data adapter examples (Firebase / GA4 export, Unity Analytics)

Layout

src/game_analytics/
  synth.py       simulation + Issues
  db.py          DuckDB warehouse + schema doc
  metrics.py     KPI queries
  sweep.py       anomaly checks
  agent.py       Analyst (LLM tools / offline router)
  report.py      HTML + charts
  cli.py         ga synth|sweep|ask|report|eval
  mcp_server.py  ga-mcp
  eval/harness.py
tests/
examples/        eval.md, report_seeded.html, report_healthy.html, mcp.json

MIT licensed.