# game-analytics-agent

[![ci](https://github.com/kevinnie2003/game-analytics-agent/actions/workflows/ci.yml/badge.svg)](https://github.com/kevinnie2003/game-analytics-agent/actions/workflows/ci.yml) ![python](https://img.shields.io/badge/python-3.10%2B-blue) ![license](https://img.shields.io/badge/license-MIT-green)

An analyst agent for game telemetry. It answers questions in plain English, runs an automatic **anomaly sweep** over retention, level funnel, economy, acquisition channels, crashes and A/B tests, and writes a self-contained HTML report with charts. Every answer is grounded in SQL you can read.

It ships with a **synthetic telemetry generator** that can seed five realistic live-ops problems, and an **evaluation harness** that scores the sweep on whether it finds them without crying wolf on healthy data.

## Results

Sweep only (no LLM, deterministic), 3000 players x 30 days, 3 seeds:

| seeded issue | detected | how it shows up |
|---|---|---|
| level_spike | 3/3 | `Level 7 is a difficulty wall: win rate 10% vs neighbours 61% / 59%` |
| android_crash_patch | 3/3 | `Crash rate on android 1.2.0 is 34%` + `D1 retention dropped for cohorts installing from day 12 (platform=android)` |
| currency_inflation | 3/3 | `Currency earned per player exploded ... fastest-growing source: daily_reward` |
| paid_channel_fraud | 3/3 | `Channel paid_c delivers installs that do not play: D1 3.6% vs 46%` |
| price_test_regression | 3/3 | `Price test arm B converts far worse than arm A (z=4.4)` |
| (healthy data) | 0 false positives in 3/3 runs | |

**recall = 1.00, precision = 1.00** on seeds 0-2, and on held-out seeds 3-4. Full table in [`examples/eval.md`](examples/eval.md); sample reports in [`examples/`](examples/) (open the HTML in a browser).

## Quick start

```bash
pip install -e ".[dev]"
ga sweep                                   # healthy synthetic data -> "No anomalies detected."
ga sweep --issue android_crash_patch       # exits 1 on a critical finding (CI gate)
ga ask "which level is too hard?" --issue level_spike
ga report --issue currency_inflation --ask "is the economy healthy?" --out report.html
ga eval                                    # precision / recall table
pytest                                     # 23 tests
```

Bring your own data: two CSVs, `players.csv` and `events.csv`, with the schema in `db.py`, then `--data path/`.

### With an LLM

```bash
pip install -e ".[llm]"
export ANTHROPIC_API_KEY=...
ga ask "why did revenue fall in the second half of the month?" --llm --show-sql
ga report --llm --out report.html          # findings get a root-cause hypothesis + action
```

The model gets three tools: `run_sql` (read-only DuckDB), `metric` (the vetted queries in `metrics.py`), and `sweep`. Every query it runs is kept in the transcript and printed in the report, so a human can audit the answer. Without a key, `ask` falls back to a keyword router over the same metric functions and says so in the output. Tests and CI never call the API.

A real run with `claude-sonnet-5` on data seeded with the Android crash, the level-7 wall and the price-test regression is in [`examples/report_llm.html`](examples/report_llm.html) (about 2 minutes, 3 questions plus finding narratives). Highlights:

- Asked *why did D1 retention drop for recent cohorts?*, it pulled retention by cohort, then by platform, then crash rate by platform and version, and concluded: "Android-only, version-triggered regression ... 1.2.0 introduced a severe crash regression on Android only (35% of sessions crash vs 1.1% baseline elsewhere)". That is the seeded cause.
- Asked *why did revenue fall in the second half of the month?*, it split revenue by platform and by experiment arm, found iOS flat while Android fell 69%, and tied it to the same crash build plus arm B of the price test.
- Asked *which country has the best D7 retention and is the difference meaningful?*, it computed a two-proportion z-test itself and answered no: KR leads at 10.1% but z is about 1.0 against the lowest country, well within noise.
- Each sweep finding got a hypothesis and an action, for example: "Hotfix or roll back the android 1.2.0 build immediately and monitor D1 retention for cohorts installing after the fix."

### As an MCP server

```bash
pip install -e ".[mcp]"
GA_ISSUES=level_spike,android_crash_patch ga-mcp
```

Tools: `schema`, `run_sql`, `metric`, `kpis`, `anomaly_sweep`, `ask`, `load_synthetic`. Point Claude Code at it with [`examples/mcp.json`](examples/mcp.json) and ask "what's wrong with this game?".

## How it works

```
 synth.py      players x days simulation: retention curve, level difficulty,
               currency sources/sinks, IAP, ads, crashes, 5 seedable issues
     |
 db.py         DuckDB warehouse: players, events, sessions view, player_days view
     |
 metrics.py    retention (Dn, by segment, by install cohort), level funnel, economy,
               monetization, crashes, channel quality, headline KPIs
     |
 sweep.py      7 generic checks -> Findings with severity + evidence
     |             change-point on Dn retention by cohort (overall and per platform)
     |             segment vs rest z-test · level vs neighbours · earn/player vs baseline
     |             channel vs others · (segment, version) crash rate · A/B two-proportion z
     |
 agent.py      Analyst: LLM with tools, or offline keyword router; SQL transcript
     |
 report.py     HTML: KPIs, findings, Q&A with queries, 6 charts (matplotlib, embedded)
```

**Checks are generic.** They compare a segment, level, day or arm against the rest of the data with sample-size guards and z-tests. None of them reads `Issues`. Porting to a real game means mapping your events into the schema; the checks and the report come for free.

**Attribution, not just detection.** The crash bug is reported twice on purpose: as a crash-rate spike on *android 1.2.0* and as a D1 retention drop *for android cohorts from day 12*. The economy check names the *source* that grew fastest. A finding an analyst can act on names where and since when.

**The change-point beat the sliding window.** The first retention check compared the last five cohorts with everything earlier. It missed a 16-point Android drop because small daily cohorts are noisy. Scanning every split day and taking the most significant one finds it at day 12, which is exactly the version rollout day.

## Design notes and tradeoffs

- **Synthetic data with seeded issues** makes the evaluation honest and the repo self-contained. The downside is the generator encodes my assumptions about player behaviour; the sweep thresholds are tuned to it and would need re-tuning on real data. Example: the A/B check fires at |z| >= 2.5 because healthy runs peak around 2.2 and seeded regressions start around 2.7. On a real game with more arms or more metrics you would want a multiple-comparison correction.
- **Sweep first, LLM second.** The statistical checks are cheap and reproducible; the model adds free-form questions and narrative. Keeping the model optional keeps CI free.
- **Offline `ask` is a keyword router, not a model.** It exists so the report and tests work without a key. It says `[offline]` in its output rather than pretending.
- **Read-only SQL is enforced by a regex**, which is fine for a demo and not for a multi-tenant deployment. DuckDB's `read_only` connection mode is the right fix when the warehouse is a file.

## Roadmap

- Weekly diff mode: what changed since the last report
- Slack/Discord delivery of the findings card
- Ad-hoc segment discovery: let the model propose the cut instead of the fixed list
- Real-data adapter examples (Firebase / GA4 export, Unity Analytics)

## Layout

```
src/game_analytics/
  synth.py       simulation + Issues
  db.py          DuckDB warehouse + schema doc
  metrics.py     KPI queries
  sweep.py       anomaly checks
  agent.py       Analyst (LLM tools / offline router)
  report.py      HTML + charts
  cli.py         ga synth|sweep|ask|report|eval
  mcp_server.py  ga-mcp
  eval/harness.py
tests/
examples/        eval.md, report_seeded.html, report_healthy.html, mcp.json
```

MIT licensed.