Skip to main content
Glama

Player Review Intelligence

CI

Player Review Intelligence collects public app-store reviews for mobile games, classifies them with Claude, links complaints to game updates with RAG, and serves the results as a weekly report, a dashboard and an MCP server. Claude Code skills and subagents run the workflows on top.

Dashboard demo

The problem

A live-service mobile game gets hundreds of store reviews a week across Google Play and the App Store, in many languages, with a star rating that says little about why. The useful signals are buried: an ad change that annoys players, a crash after an update, a weapon that feels pay-to-win. Game teams usually find these late, by skimming. This project turns the review stream into numbers a team can trust, plus a short weekly read that says what changed, what players say and who should act. Every number comes from the database, never from the model.

Games are configured in config/apps.yaml; adding one needs no code changes.

Related MCP server: google-play-analytics-mcp

What it does

Step

How

Measured on real data

Collect

Google Play scraper plus the App Store RSS feed: polite pacing, retries, idempotent upserts, no usernames stored

950 reviews across both stores

Classify

claude-haiku-5-5, one strict tool call per review, Pydantic validation, prompt caching, Message Batches

0.89 test accuracy against a stronger model's labels; ~$0.08 per 1,000 reviews

Evaluate

Blind reference labels, dev/test split, per-class P/R/F1, confusion matrix, Sonnet judge

README table updated by eval --write-readme

Link to releases

App Store version history, local embeddings in sqlite-vec, cited answers, per-store spike test

Answers cite real review ids and the right version

Report

Code computes the facts, Sonnet writes the prose, code checks it, a judge scores it

Example report, ~$0.03 each

Serve

FastAPI dashboard, MCP server (tools, resources, prompts), Slack webhook

Works in Claude Code via .mcp.json

Automate

Claude Code skills and subagents in .claude/

/weekly-review-report, /add-new-game, /improve-classifier

Architecture

flowchart LR
  subgraph stores["Public stores"]
    GP["Google Play"]
    AS["App Store feed<br/>and version history"]
  end
  GP --> COL
  AS --> COL
  COL["collectors<br/>pacing, retries"] --> DB[("SQLite<br/>reviews, labels, patch notes, reports<br/>plus sqlite-vec vectors")]
  DB --> CLS["classify<br/>Haiku 5.5, strict tool<br/>prompt cache, batches"]
  CLS --> DB
  DB --> IDX["index<br/>local MiniLM embeddings"]
  IDX --> DB
  EV["evals<br/>blind labels, metrics, judge"] -.->|keeps the prompt honest| CLS
  DB --> INS["insights<br/>SQL answers"]
  DB --> RAG["ask and spikes<br/>RAG, z-test"]
  INS --> REP["weekly report<br/>Sonnet prose, checks, judge"]
  INS --> DASH["dashboard<br/>FastAPI, SVG"]
  INS --> MCP["MCP server<br/>tools, resources, prompts"]
  RAG --> MCP
  REP --> SL["Slack"]
  MCP --> CC["Claude Code / Desktop<br/>skills, subagents"]

Each module is described in two sentences in docs/ARCHITECTURE.md.

Quickstart

Requires uv, which installs Python 3.12 for you. Only an ANTHROPIC_API_KEY is needed.

git clone https://github.com/LightWonkas/player-review-intelligence && cd player-review-intelligence
uv sync
cp .env.example .env          # then set ANTHROPIC_API_KEY in .env
uv run review-intel fetch --app critical-strike --store all --limit 100
uv run review-intel classify --app critical-strike --limit 100     # ~$0.02
uv run review-intel report --app critical-strike --days 7          # ~$0.03
uv run review-intel dashboard                                      # http://127.0.0.1:8765

This exact sequence was run from a fresh clone. The database is created on first use, and every command is safe to re-run. uv run review-intel --help lists the rest: eval, label, patches, index, ask, spikes.

Example report

docs/example-report.md is a real report from the review data, unedited. It has headline numbers against the previous week, the top issues with player quotes and review ids, a version watch, a four-week sentiment table, prioritized actions for dev, product and marketing, and the judge's score.

Classification

Each review gets one strict tool call (record_classification) to claude-haiku-5-5. The result is a primary category plus up to three more from the 12 in config/taxonomy.yaml, along with sentiment, severity (1-5), a bug-report flag, mentioned features and a one-line summary.

  • Validation: Pydantic checks every response. Invalid output is retried once and then kept in classification_failures, never dropped.

  • Untrusted input: Review text goes in as escaped data inside <review> tags, and a test checks that a malicious review can't break out.

  • Prompt versions: prompt_version (e.g. v1-10aea4a5) is a hash of the prompt and tool schema, so editing either creates a new version and results never mix.

  • Cost: The ~1,900-token shared prefix is prompt-cached, so each review pays for about 300 input and 210 output tokens. classify --dry-run prints the count and the estimated cost before you spend anything.

Evals

The reference set has 100 reviews, labeled blind: the labeler never sees the model's answer. The current labels were set by claude-opus-5-5, a stronger model than the classifier, and recorded per item in the labeler field. So the scores measure agreement with a stronger reference, not human accuracy; review-intel label lets a person relabel or extend the set.

The sample is 40% random (mostly praise, like the real mix) and 60% low-rated reviews with real text, where the hard categories are, with at least 25% from the App Store. A hash of each review fixes it in a 70/30 dev/test split: prompts are tuned on dev, and the headline number is test. With 35 test reviews, one review moves accuracy by about 3 points, so every score comes with a 95% bootstrap interval.

review-intel eval reports:

  • primary-category accuracy, per-category precision/recall/F1 and macro-F1

  • sentiment accuracy and a confusion matrix

  • a comparison with the previous prompt or model

  • an LLM judge's faithfulness score for the per-review summaries (claude-sonnet-5-5)

Predictions are cached per prompt version, so re-running an unchanged prompt is free.

Date

Prompt

Model

Test acc [95% CI]

Test macro-F1 [95% CI]

Sentiment acc (all)

Judge

2026-11-19

v1-10aea4a5

claude-haiku-5-5

0.89 [0.77-0.97] (n=35)

0.70 [0.52-0.95]

0.97

4.65/5

Read the intervals before the point estimates. Accuracy is solid (0.77-0.97), but macro-F1 spans 0.52-0.95: categories with 1-3 examples pull the average down, and the set is too small to pin down rare categories. The weakest spot is the game's own auto-aim, which gets read as cheating (controls_ux -> cheating_hackers). The improve-classifier skill is built for exactly this kind of fix.

Linking complaints to releases (RAG)

uv run review-intel patches fetch --app critical-strike   # release notes, one request
uv run review-intel index --app critical-strike           # local embeddings, free
uv run review-intel ask "When did matchmaking complaints start?" --app critical-strike
uv run review-intel spikes --app critical-strike
  • Patch notes: The App Store page lists the last ~25 versions with dates and notes, the only public release history with dates. Notes for other stores can be imported from markdown (patches import). Version numbers are shared across stores, so one note serves both.

  • Embeddings: Reviews and single patch-note lines are embedded locally with all-MiniLM-L6-v2 and stored in sqlite-vec tables in the same SQLite file, so search results join straight back to SQL. Re-indexing only embeds changed text.

  • ask: Retrieves similar reviews and patch notes, adds the notes for the versions those reviews were written on, and adds a per-version timeline from SQL. Sonnet answers and cites [r<id>] and [v<version>]. Every citation is checked against what was retrieved; invented ones are removed and reported. Asked when complaints about a new weapon started, it named the release that added the weapon and the earliest review that mentions it.

  • spikes: No LLM is involved. Per store, a version's share of reviews mentioning a category is compared with the previous versions using a two-proportion z-test. Stores are kept apart, so a shift in store mix can't fake a spike. Each spike links to the patch note in force, with its lines ranked by similarity to the complaints. When nothing is significant, the closest candidates are shown and labeled as not significant.

Weekly report and dashboard

Code lays out the report and fills in every number. claude-sonnet-5-5 writes only the prose, as structured output, and code then checks it:

  • Numbers: every number in the prose must appear in the facts the model was given. If one doesn't, the model gets one retry with the offending numbers listed.

  • References: review ids and categories that weren't in the facts are dropped.

  • Judge: a separate call scores the prose for faithfulness, and the score goes in the footer.

On real data the first version scored 2/5: it said "players say" on the strength of one review, and it treated an old version as current. Fixing the facts (per-store versions, marked partial weeks, the severity scale) and asking for claims sized to their evidence brought it to 4/5.

The dashboard is FastAPI and Jinja2 with server-side SVG charts and no JavaScript. It shows 7-day stat tiles, top issues, reviews by category (7/30/90 days), weekly sentiment and the latest report. Every chart has a table view, and both color modes were run through a color-vision deficiency check. JSON routes live under /api/apps/{app}/.... The dashboard is read-only and listens on localhost.

Dashboard

MCP server

review-intel-mcp serves the data over the Model Context Protocol with the official Python SDK (MCPServer, the SDK 2.x name for FastMCP). It runs over stdio, or over streamable HTTP with --transport streamable-http.

Kind

Name

What it does

Tool

top_issues(app, days, limit)

Top complaint categories vs the previous period, with example reviews

Tool

sentiment_trend(app, days, bucket)

Sentiment and average rating per day or week

Tool

compare_versions(app, version_a, version_b)

Category shares with z-scores, rating, sentiment and patch notes

Tool

search_reviews(app, query, category?, version?, limit)

Semantic search with filters

Tool

ask(app, question)

Cited answer (calls Claude, ~$0.014)

Resource

reviews://{app}/{review_id}, reports://{app}/latest, taxonomy://categories

One review, the latest report, the categories

Prompt

weekly_report(app), triage_bugs(app, days)

Step-by-step plans that use the tools

All tools except ask are read-only and free, and they share the insights query layer with the report and the dashboard, so all three show the same numbers.

Claude Code: a session opened in this folder picks up .mcp.json; approve it once. To add the server elsewhere: claude mcp add --transport stdio review-intel -- uv run --directory /path/to/repo review-intel-mcp.

Claude Desktop: add this to claude_desktop_config.json:

{
  "mcpServers": {
    "review-intel": {
      "command": "uv",
      "args": ["run", "--directory", "C:\\path\\to\\player-review-intelligence", "review-intel-mcp"]
    }
  }
}

Inspector: npx @modelcontextprotocol/inspector uv run review-intel-mcp

Claude Code skills and subagents

The repo uses Claude Code to operate the system, not just to build it.

Name

What it does

Skill

weekly-review-report

Fetch, classify and report for a game, then summarize. Prices each paid step before running it.

Skill

add-new-game

Adds a game to the config, checks both store ids against the stores, fetches a sample.

Skill

improve-classifier

Eval, confusion matrix, one proposed change shown as a diff, re-eval, before/after table.

Subagent

bug-analyst

Deduplicated bug list with versions, severity and example review ids.

Subagent

sentiment-analyst

Sentiment by week and version, backed by numbers.

Subagent

report-reviewer

Checks a report's quotes and claims against the data before it is shared.

Each subagent only gets the free, read-only MCP tools. CLAUDE.md gives every session the commands, the conventions and the rule: run tests and evals before claiming a change works.

Cost

Prices as of November 2026 (pricing.py); check Anthropic's pricing page before budgeting.

What

Model

Cost

Classify 1,000 reviews, --batch

Haiku 5.5

~$0.08

Classify 1,000 reviews, synchronous

Haiku 5.5

~$0.17

Weekly report, with judge

Sonnet 5.5

~$0.03

ask, one question

Sonnet 5.5

~$0.014

Eval run (100 labels + 20 judged summaries)

Haiku + Sonnet

~$0.08; ~$0.06 when cached

Embeddings and index (950 reviews in 19 s on CPU)

local MiniLM

$0

A game with 1,000 new reviews a week costs about $0.11 per week: a batch classification plus one report. All the live runs behind this README cost under $1 in total.

Design decisions

  • Haiku to classify, Sonnet to write and judge. Classification is high-volume and narrow, and Haiku 5.5 reaches 0.89 test accuracy at about $0.08 per 1,000 reviews. Prose a product manager forwards, and grading other models' output, need the stronger model. The judge is never the model it grades.

  • Tool use for structured output. One forced, strict tool call with enum categories gives parseable JSON every time, and Pydantic covers what JSON Schema can't express. Results are keyed by prompt version and model, so comparisons stay clean.

  • Code computes, the model writes. Every number in a report, answer or dashboard comes from SQL. The model's prose is checked for numbers and ids that aren't in the facts, and a judge scores what's left. This is what makes the report safe to forward.

  • Local embeddings in the same database. MiniLM on CPU is free, keeps review text on the machine, and needs no second vendor. sqlite-vec keeps vectors next to the rows they describe, so retrieval joins straight back to SQL. The trade-off is an English-focused model, which is one setting to change.

  • Evals keep the prompt honest. Labels are blind, the dev/test split is fixed by hash, and predictions are cached per prompt version. The README table updates from real runs, so any prompt change shows up as a number before it ships.

  • Untrusted input everywhere. Review text is escaped and delimited in every prompt, flattened and escaped in reports, and autoescaped in the dashboard. Tests cover injection attempts at each layer.

  • Statistics before stories. Spikes need a z-score and a minimum sample, per store. With today's data that honestly means "no significant spikes" plus the closest candidates, rather than a confident narrative.

Limitations and next steps

  • Reference labels come from a stronger model, not people, and there are only 100. Scores measure agreement with Opus, and the macro-F1 interval is wide (0.52-0.95). A larger, human-labeled set with more rare-category examples would measure real accuracy (review-intel label is ready for it).

  • Shallow history. The App Store feed serves only ~500 recent reviews per country, and fetching newest-first covers a few weeks of Google Play. With ~20 reviews per version, spikes rarely reach significance. A scheduled weekly fetch would build the history.

  • One storefront. Only the US store is tracked. Other countries are a config entry each, and multilingual reviews would want a multilingual embedding model (EMBEDDING_MODEL).

  • Judge scores vary. Scores for the same week ranged from 3/5 to 4/5 between runs. The report-reviewer subagent is the second check before sharing.

  • SQLite. Time windows and counts run as SQL filters and aggregates, but per-version timelines and spikes still read a game's full classified history, and vector search uses sqlite-vec. For hundreds of thousands of reviews: Postgres through DATABASE_URL, pgvector, and per-version counts in SQL.

  • Next steps: human labels, a scheduled pipeline with Slack alerts when a spike becomes significant, and more storefronts.

Development

uv run pytest                                     # 217 tests, no network, no real LLM
uv run ruff check . && uv run ruff format --check .
uv run mypy                                       # strict
uv run python scripts/record_fixtures.py          # re-record store fixtures (by hand)
uv run scripts/make_demo_gif.py                   # rebuild docs/demo.gif from a running dashboard

CI runs ruff, mypy and pytest on every push. The full build plan is in docs/SPEC.md.

License

MIT

Available Tools

5 tools
askA
Read-only

Answer an open question (e.g. "when did matchmaking complaints start?") from retrieved reviews and patch notes, citing review ids and versions. Calls Claude Sonnet: costs about $0.014 per question, so prefer the other tools first.

Inputs: app (required), question (required). Returns: notes, answer, cost_usd, cited_versions, +2 more

compare_versionsA
Read-only

Compare two app versions: reviews, rating, sentiment, the share of reviews mentioning each category (with a z-score; |z| >= 2 is roughly significant) and the patch notes in force for each.

Inputs: app (required), version_a (required), version_b (required). Returns: a, b, app, categories, +1 more

search_reviewsA
Read-only

Semantic search over reviews, optionally filtered by category and app version. Returns review ids, ratings, versions, dates, classification and (untrusted) text.

Inputs: app (required), query (required), limit, version, +1 more. Returns: app, query, results, version, +1 more

sentiment_trendB
Read-only

Positive / neutral / negative counts and average rating per day or week.

Inputs: app (required), days, bucket. Returns: app, bucket, points, period_end, +1 more

top_issuesA
Read-only

The most common complaint categories in the last days days (by primary category; praise and other excluded), with counts, share, severity, the count in the previous period of the same length, and example reviews.

Inputs: app (required), days, limit. Returns: app, issues, period_end, period_start, +3 more

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    A3.7/5.0

    Scored across 5 tools

    Disambiguation4/5

    Each tool targets a distinct analysis mode: raw semantic retrieval (search_reviews), complaint aggregation (top_issues), version diffing (compare_versions), time-series sentiment (sentiment_trend), and open-ended Q&A (ask). The main soft spot is `ask`, which is a general-purpose catch-all that could plausibly answer anything the other four handle, but its description explicitly tells agents to prefer the others, which mitigates the overlap.

    Naming Consistency4/5

    All names are snake_case and self-descriptive, but the grammatical pattern varies: verb_noun (search_reviews, compare_versions), noun phrases (top_issues, sentiment_trend), and a bare verb (ask). It reads cleanly and an agent can infer intent, with `ask` being the only notably vague outlier.

    Tool Count5/5

    Five tools is tightly scoped for a review-analytics server, with each tool covering a genuinely different question type (retrieval, ranking, comparison, trend, synthesis). Nothing feels padded or redundant.

    Completeness4/5

    The surface covers search, aggregation, version comparison, trends, and open Q&A with cited review ids, which is most of the review-intel lifecycle. Minor gaps: no direct fetch-by-review-id, no standalone patch-notes or category-taxonomy listing (those are only reachable indirectly through compare_versions and ask).

    Maintenance

    ActivityMaintained
    ResponsivenessNo recent community issues

    Related MCP Connectors

    • Ask Claude, ChatGPT, or any MCP client about your game's players, retention, revenue, and win rates.

      https://gamestats.ai/api/v1/mcp
    • MCP server unifying ERPs, CRMs, APIs and knowledge base for Claude, ChatGPT and Gemini.

      https://api.artificialwit.com/mcp
    • Your agent needs app-store data — what an app looks like on the App Store and Google Play, what reviewers say, and what ranks for a search in a given country. **What you can ask for** • "What does this app's store listing look like, and how is it rated?" • "Pull recent reviews for this app and group the complaints." • "What apps rank for this search term in Japan?" • "List the top apps in this category on both stores." • "Compare this app's listing on iOS and Android." **How to use it** Point any MCP client at https://mcp.aisa.one/seo-apps/mcp and sign in with OAuth — there is no key to create or paste. 37 tools: Apple App Store and Google Play app info, app lists, category listings, reviews and store search results, in live and queued forms, with categories, languages and locations for each. **It is also a door to the rest** The same login reaches 26 sources and 580+ operations. Read the store listing here, then ask the same agent what the app's website ranks for — without adding a second server. **What it costs** Finding and inspecting an operation is free. Running one is billed per call at API prices, with no seat and no monthly minimum, and every call takes max_price_usd so an agent cannot overspend by accident. **Where else it reaches** https://mcp.aisa.one/seo/mcp for all of it at once — rankings, keywords, backlinks, site health and AI-answer visibility across DataForSEO, Semrush and Ahrefs.

    • ASOMobile MCP server: App Store Optimization data for the iOS App Store and Google Play inside AI agents. 29 tools: keyword research (traffic, competition, suggestions), app keyword indexing, competitors and keyword gaps, top charts and chart history, reviews with rating summaries, Apple Search Ads insights, organic download estimates and Keyword Monitor rank tracking, plus six ASO skills (app audit, keyword research, competitor audit, localization check, review analysis, ASA planning). Remote streamable HTTP with OAuth 2.1 (DCR + PKCE); sign in with your ASOMobile account.

      https://mcp.asomobile.net/mcp

    Related MCP Servers

    • A
      license
      Not graded
      quality
      D
      maintenance
      Remote MCP server that exposes Airspeed/Glyphic call data to Claude web, enabling tools to list, retrieve, and query calls, transcripts, snippets, and playbooks.
      ISC
    • A
      license
      Not graded
      quality
      C
      maintenance
      An MCP server that exposes Google Play Console analytics to AI agents, including store-listing acquisitions by traffic source, installs, uninstalls, ratings, crashes, subscriptions, and reviews by reading bulk CSV reports from Cloud Storage.
      MIT
    • A
      license
      A
      quality
      D
      maintenance
      MCP server for Claude Desktop that exposes App Store Connect and RevenueCat APIs, enabling indie iOS developers to query portfolio metrics, sales reports, subscriptions, and customer reviews directly in chat.
      17
      MIT
    • A
      license
      Not graded
      quality
      A
      maintenance
      Enables AI agents to query live App Store and Google Play data, including app search, revenue/download metrics, keyword difficulty, and reviews, via a hosted MCP server with API-key authentication.
      76 npm
      MIT