# Retention Agent

**Tells your team what to do next for every customer, and shows why.**

Every click, ticket and payment is your customer telling you what to do next. Retention Agent reads them, picks the next action for each account (save it, fix billing, upsell, expand seats...), and cites the exact events behind every recommendation. Then an eval harness grades it, because an agent that moves revenue should be measured, not trusted.

![Accounts view: the rules say fix billing, but the card was already updated and the invoice paid. Hovering the cited event lights it up in the timeline.](docs/accounts.png)

![Scoreboard: rules baseline vs agent on accuracy, evidence grounding and MRR at risk caught.](docs/scoreboard.png)

## Why

Net revenue retention is decided account by account. Customer success teams already have the signals: product usage, support tickets, billing events. What they lack is time to read them all. So they fall back on spreadsheet rules like "logins dropped, call them", which fire on holiday weeks and miss the account whose champion quietly left.

Retention Agent reads the whole timeline, makes one call per account, and cites its evidence so a human can check it in seconds.

## What it does

- **Recommends one action per account** from a fixed set: rescue call, fix billing, win-back offer, upsell plan, expand seats, onboarding nudge, hold steady.
- **Cites its evidence.** Every recommendation lists the event ids behind it. Hover one in the UI and the matching event lights up in the timeline. Ids the model invents are kept and shown struck through, not silently dropped.
- **Compares against a baseline.** A rule table, the kind a CS team keeps in a spreadsheet, runs on the same accounts.
- **Grades both** on labeled accounts: accuracy, per-action precision and recall, evidence grounding, and dollars of at-risk MRR caught.
- **Pushes to a CRM.** Creates a HubSpot task with the reason and cited events, or shows a preview when no token is set.
- **Works over MCP**, so any MCP client can list accounts, read timelines, get recommendations and push tasks.

## How it works

```mermaid
flowchart LR
  S[Usage, support and billing events] --> A[Agent: one forced tool call]
  A --> R[Action + reason + cited event ids]
  R --> U[UI: evidence highlighted in timeline]
  R --> C[CRM task]
  subgraph Eval loop
    G[Seeded scenarios with ground truth] --> S
    R --> E[Scorer: accuracy, grounding, MRR caught]
    B[Rule table baseline] --> E
  end
```

- `lib/types.ts` defines the data. `Event` is a union over `usage`, `support` and `billing`. `Recommendation` is `{ accountId, play, confidence, reason, evidence, hallucinated }`.
- `lib/plays.ts` is the single table of actions. Labels, CRM task templates, and which actions count as churn risk all live there.
- `lib/scenarios.ts` is the ground truth. Each scenario builds its signal events and names the expected action. A seeded generator makes 40 accounts with made-up names and background noise, written to `data/accounts.json` by `npm run seed`. Output is deterministic.
- `lib/rules.ts` is the baseline: first matching rule wins.
- `lib/agent.ts` sends the account timeline, with event ids, to the model and forces one `recommend` tool call. The output is parsed with zod. Cited ids that aren't on the account are moved to `hallucinated`, and eval counts them.
- `lib/score.ts` is pure scoring logic, covered by `npm test`.

### Tricky scenarios

Each runs on several accounts. The rule table gets the first five wrong.

| Scenario | Rule table says | Expected |
| --- | --- | --- |
| Usage dip over a holiday the customer announced | rescue call | hold steady |
| Two failed payments, but seats and features growing | rescue call | fix billing |
| Payment failed, card updated, invoice paid | fix billing | hold steady |
| Ticket spike that is all feature requests from a growing team | rescue call | expand seats |
| Repeated API limit hits plus an angry ticket about 429s | rescue call | upsell plan |
| Champion went quiet and a negative ticket | rescue call | rescue call |

## Quickstart

```bash
npm i
npm run seed    # regenerate data/accounts.json (already committed)
npm run eval    # score both recommenders, write data/scoreboard.json
npm run dev     # http://localhost:3000
npm test        # scoring tests
```

Without an API key, eval runs the rule table only and the UI shows those results.

## Environment

Copy `.env.example` to `.env`.

| Variable | Purpose |
| --- | --- |
| `ANTHROPIC_API_KEY` | Enables the agent in eval, the "Run live" button, and the MCP `recommend` tool. |
| `RETENTION_AGENT_MODEL` | Model id for the agent. Defaults to `claude-sonnet-5`. |
| `HUBSPOT_TOKEN` | Optional. With it, "Push to CRM" creates a real HubSpot task. Without it, you get a preview. |
| `DEMO_ACCESS_TOKEN` | Needed anywhere but `next dev` for "Run live" and "Push to CRM", which spend API credits and write to the CRM. Open the app once with `?token=<value>`; the UI moves it to session storage, strips it from the URL, and sends it as `x-demo-token`. Without it, a deployed instance shows the cached eval results only. |

## Eval results

From `data/scoreboard.json`, 40 accounts, agent on `claude-sonnet-5`:

| Metric | Rule table | Agent |
| --- | --- | --- |
| Accuracy | 62.5% (25/40) | 100% (40/40) |
| Evidence grounding | 95.7% (66/69 cited ids) | 77.1% (138/179 cited ids) |
| Hallucinated evidence ids | 0 | 0 |
| MRR at risk caught | $133,500 of $152,900 | $152,900 of $152,900 |

The rule table catches every rescue case, but flags 19 accounts to get 7 right: 37% precision. Every miss comes from the first five tricky scenarios above. The agent gets all of them, including the holiday dip and the card that was already fixed.

Two honest caveats. A perfect score on 40 seeded accounts says this test set is too easy, not that the agent is done; harder, noisier scenarios are the next thing to add. And the agent over-cites: it backs its call with healthy invoices and routine usage, not just the signals that decided it, which is why its grounding trails the rules.

**Definitions.**
- **Accuracy**: predicted action equals the scenario's expected action.
- **Evidence grounding**: cited ids that exist on the account and are among the scenario's signal events, divided by all cited ids, hallucinated ones included.
- **MRR at risk caught**: total MRR of accounts whose expected action is rescue call, fix billing or win-back offer, and which got exactly that action.

## MCP

`npm run mcp` starts a stdio MCP server with four tools. It runs locally with your own keys, so it has no token gate.

- `list_accounts` returns accounts by MRR.
- `get_timeline(accountId)` returns the event timeline.
- `recommend(accountId)` uses the agent when a key is set, otherwise the rule table.
- `push_to_crm(accountId)` recommends, then creates a CRM task or returns a preview.

Add it to your MCP client config. Point at the project's own `tsx` binary, since npm prints a banner on stdout and the client may start in another directory:

```json
{
  "mcpServers": {
    "retention-agent": {
      "command": "/absolute/path/to/retention-agent/node_modules/.bin/tsx",
      "args": ["/absolute/path/to/retention-agent/mcp/server.ts"],
      "env": { "ANTHROPIC_API_KEY": "sk-...", "HUBSPOT_TOKEN": "" }
    }
  }
}
```

## What I'd build next

- **Real event sources.** Stripe and Segment webhooks in place of the seeded data.
- **Outcome tracking.** Record what happened after each action, then learn which actions actually saved accounts.
- **Per-segment playbooks.** An enterprise account at risk needs a different response than a self-serve one.
- **Human-in-the-loop approval.** CSMs approve or edit a recommendation before it becomes a task, and those edits feed back into the eval set.

## License

MIT