# call-reason-insights-mcp

An MCP server that exposes historical call-center "call reason notes" as agent-callable tools,
paired with an evaluation harness that scores how well an automated categorizer performs against
those notes' ground truth. Built as a direct callback to enterprise call-center operations at
AT&T: the same category of data (customer contact records, category tags, free-text agent notes)
that historically only got queried in retrospective monthly reports now becomes something an agent
can query directly, on demand, to find a specific emerging problem while it is still small.

## The problem this targets

Call reason notes accumulate by the millions at scale. What they get used for, almost universally,
is after-the-fact category counts: "billing calls were up 4% this month." What they rarely get used
for is surfacing a *new, specific* problem while it's still forming — a firmware update quietly
causing a cluster of "modem overheating" calls this week, invisible next to routine billing volume
unless you're looking at trend against baseline, not total volume.

`get_emerging_problem_points` is the tool this repo exists for: it compares a recent window against
each category's own historical baseline and flags real statistical spikes, not just whatever has
the most raw calls.

## What's in this repo

- **`server.py`** — the MCP server. Five tools: `get_call_volume_summary`, `get_top_call_reasons`,
  `get_emerging_problem_points`, `get_call_examples`, `search_call_notes`.
- **`data/generate_call_notes.py`** — synthetic call-note generator. Injects two things a real
  analysis needs to handle: (1) a genuine emerging problem concentrated in the most recent 14 days,
  and (2) ~12% category mislabeling noise, the same way a rushed agent picks the first plausible
  dropdown option. Entirely synthetic — no real customer or account data.
- **`harness/`** — the evaluation half, four pieces that build on each other:
  - `classifier.py` — a keyword-rule classifier, a stand-in for whatever real classifier you'd
    deploy (an LLM call, an embedding lookup, a fine-tuned model). Loads its rules from
    `generated_rules.json` if present, falling back to a small hand-written rule set otherwise.
  - `keyword_generator.py` — builds those rules from data instead of by hand. Scores which words
    are most concentrated in each category (vs. every other category), after two dedupe passes:
    stemming ("install", "installing", "installation" collapse to one signal instead of splitting
    into three weaker ones, no model needed) and an optional model-based semantic check
    (`--semantic-dedupe`, defaults to a higher-capability model than the classifier or labeler)
    that catches cross-category overlap stemming can't, words like "router" and "modem" that
    share no root but mean the same kind of problem. Also the reason `classifier.py` matches on
    whole words, not substrings: the generator surfaced "all" as a top word for equipment_issue,
    which silently matched inside "install"/"installation" until that got fixed to word-boundary
    matching.
  - `llm_labeler.py` — the independent-labeling half. Calls a second model to label a sample of
    notes, kept deliberately separate from whatever categorizer is under test, since a system
    can't meaningfully grade itself. Runs in a dry-run mode with no API key required, so the rest
    of the pipeline can be inspected without live network access or an Anthropic key.
  - `theme_eval.py` — scores the classifier against trusted labels, either the dataset's built-in
    `true_category` (for testing the harness itself) or a labels file from `llm_labeler.py` (for a
    genuinely independent check). Reports per-category precision/recall/F1, a full confusion
    matrix, and the categories the classifier confuses most, the same "don't just report an
    aggregate score, tell me where it breaks" principle used in
    [apex-overlay](https://github.com/futurerichdad/apex-overlay), applied here to text
    classification instead of tool-calling execution.
- **`tests/`** — pytest suite covering every MCP tool, the eval harness, keyword generation, and
  the word-boundary matching fix, including a test that asserts the injected spike is actually
  detected and a test that would have caught the "all" bug directly.

## Design philosophy

- Each MCP tool does exactly one job with a narrow, explicit input schema.
- Every tool returns structured JSON, never free-form prose, so both a human and an eval harness
  can score the output.
- The eval harness is a first-class part of this repo, not an afterthought: a classifier swapped
  in to fix the raw label noise needs to prove it's actually better, per category, before anyone
  trusts its output for a decision like "escalate this to network engineering."
- Synthetic data throughout, with injected ground truth, so both the MCP tools and the eval
  harness can be verified against a known-correct answer instead of eyeballed.

## Setup

```bash
pip install -r requirements.txt
python data/generate_call_notes.py --out call_notes.db --days 120
pytest tests/ -v

# build classifier rules from the data instead of hand-writing them
python -m harness.keyword_generator --db call_notes.db --out harness/generated_rules.json

# optional: label a sample with a second, independent model (requires ANTHROPIC_API_KEY)
python -m harness.llm_labeler --db call_notes.db --sample 50 --out harness/llm_labels.json
python -m harness.theme_eval --db call_notes.db --labels-file harness/llm_labels.json
```

Register with Claude Code:

```bash
claude mcp add call-reason-insights -- python /path/to/call-reason-insights-mcp/server.py
```

Run the eval harness directly:

```bash
python -m harness.theme_eval --db call_notes.db
```

See `examples/example_prompts.md` for natural-language prompts that route to each tool.

## Related

Part of a connected set of MCP + evaluation projects:
[fleet-data-mcp](https://github.com/futurerichdad/fleet-data-mcp),
[apex-overlay](https://github.com/futurerichdad/apex-overlay),
[claude-code-eval-mcp](https://github.com/futurerichdad/claude-code-eval-mcp).

## License

MIT — see [LICENSE](LICENSE).