Retention Agent

Tells your team what to do next for every customer, and shows why.

Every click, ticket and payment is your customer telling you what to do next. Retention Agent reads them, picks the next action for each account (save it, fix billing, upsell, expand seats...), and cites the exact events behind every recommendation. Then an eval harness grades it, because an agent that moves revenue should be measured, not trusted.

Why

Net revenue retention is decided account by account. Customer success teams already have the signals: product usage, support tickets, billing events. What they lack is time to read them all. So they fall back on spreadsheet rules like "logins dropped, call them", which fire on holiday weeks and miss the account whose champion quietly left.

Retention Agent reads the whole timeline, makes one call per account, and cites its evidence so a human can check it in seconds.

Related MCP server: Indraft

What it does

• Recommends one action per account from a fixed set: rescue call, fix billing, win-back offer, upsell plan, expand seats, onboarding nudge, hold steady.

• Cites its evidence. Every recommendation lists the event ids behind it. Hover one in the UI and the matching event lights up in the timeline. Ids the model invents are kept and shown struck through, not silently dropped.

• Compares against a baseline. A rule table, the kind a CS team keeps in a spreadsheet, runs on the same accounts.

• Grades both on labeled accounts: accuracy, per-action precision and recall, evidence grounding, and dollars of at-risk MRR caught.

• Pushes to a CRM. Creates a HubSpot task with the reason and cited events, or shows a preview when no token is set.

• Works over MCP, so any MCP client can list accounts, read timelines, get recommendations and push tasks.

How it works

flowchart LR
  S[Usage, support and billing events] --> A[Agent: one forced tool call]
  A --> R[Action + reason + cited event ids]
  R --> U[UI: evidence highlighted in timeline]
  R --> C[CRM task]
  subgraph Eval loop
    G[Seeded scenarios with ground truth] --> S
    R --> E[Scorer: accuracy, grounding, MRR caught]
    B[Rule table baseline] --> E
  end

• lib/types.ts defines the data. Event is a union over usage, support and billing. Recommendation is { accountId, play, confidence, reason, evidence, hallucinated }.

• lib/plays.ts is the single table of actions. Labels, CRM task templates, and which actions count as churn risk all live there.

• lib/scenarios.ts is the ground truth. Each scenario builds its signal events and names the expected action. A seeded generator makes 40 accounts with made-up names and background noise, written to data/accounts.json by npm run seed. Output is deterministic.

• lib/rules.ts is the baseline: first matching rule wins.

• lib/agent.ts sends the account timeline, with event ids, to the model and forces one recommend tool call. The output is parsed with zod. Cited ids that aren't on the account are moved to hallucinated, and eval counts them.

• lib/score.ts is pure scoring logic, covered by npm test.

Tricky scenarios

Each runs on several accounts. The rule table gets the first five wrong.

Quickstart

npm i
npm run seed    # regenerate data/accounts.json (already committed)
npm run eval    # score both recommenders, write data/scoreboard.json
npm run dev     # http://localhost:3000
npm test        # scoring tests

Without an API key, eval runs the rule table only and the UI shows those results.

Environment

Copy .env.example to .env.

Eval results

From data/scoreboard.json, 40 accounts, agent on claude-sonnet-5:

The rule table catches every rescue case, but flags 19 accounts to get 7 right: 37% precision. Every miss comes from the first five tricky scenarios above. The agent gets all of them, including the holiday dip and the card that was already fixed.

Two honest caveats. A perfect score on 40 seeded accounts says this test set is too easy, not that the agent is done; harder, noisier scenarios are the next thing to add. And the agent over-cites: it backs its call with healthy invoices and routine usage, not just the signals that decided it, which is why its grounding trails the rules.

Definitions.

• Accuracy: predicted action equals the scenario's expected action.

• Evidence grounding: cited ids that exist on the account and are among the scenario's signal events, divided by all cited ids, hallucinated ones included.

• MRR at risk caught: total MRR of accounts whose expected action is rescue call, fix billing or win-back offer, and which got exactly that action.

MCP

npm run mcp starts a stdio MCP server with four tools. It runs locally with your own keys, so it has no token gate.

• list_accounts returns accounts by MRR.

• get_timeline(accountId) returns the event timeline.

• recommend(accountId) uses the agent when a key is set, otherwise the rule table.

• push_to_crm(accountId) recommends, then creates a CRM task or returns a preview.

Add it to your MCP client config. Point at the project's own tsx binary, since npm prints a banner on stdout and the client may start in another directory:

{
  "mcpServers": {
    "retention-agent": {
      "command": "/absolute/path/to/retention-agent/node_modules/.bin/tsx",
      "args": ["/absolute/path/to/retention-agent/mcp/server.ts"],
      "env": { "ANTHROPIC_API_KEY": "sk-...", "HUBSPOT_TOKEN": "" }
    }
  }
}

What I'd build next

• Real event sources. Stripe and Segment webhooks in place of the seeded data.

• Outcome tracking. Record what happened after each action, then learn which actions actually saved accounts.

• Per-segment playbooks. An enterprise account at risk needs a different response than a self-serve one.

• Human-in-the-loop approval. CSMs approve or edit a recommendation before it becomes a task, and those edits feed back into the eval set.

License

MIT