call-reason-insights-mcp

An MCP server that exposes historical call-center "call reason notes" as agent-callable tools, paired with an evaluation harness that scores how well an automated categorizer performs against those notes' ground truth. Built as a direct callback to enterprise call-center operations at AT&T: the same category of data (customer contact records, category tags, free-text agent notes) that historically only got queried in retrospective monthly reports now becomes something an agent can query directly, on demand, to find a specific emerging problem while it is still small.

The problem this targets

Call reason notes accumulate by the millions at scale. What they get used for, almost universally, is after-the-fact category counts: "billing calls were up 4% this month." What they rarely get used for is surfacing a new, specific problem while it's still forming — a firmware update quietly causing a cluster of "modem overheating" calls this week, invisible next to routine billing volume unless you're looking at trend against baseline, not total volume.

get_emerging_problem_points is the tool this repo exists for: it compares a recent window against each category's own historical baseline and flags real statistical spikes, not just whatever has the most raw calls.

Related MCP server: Salesloft MCP Demo Server

What's in this repo

• server.py — the MCP server. Five tools: get_call_volume_summary, get_top_call_reasons, get_emerging_problem_points, get_call_examples, search_call_notes.

• data/generate_call_notes.py — synthetic call-note generator. Injects two things a real analysis needs to handle: (1) a genuine emerging problem concentrated in the most recent 14 days, and (2) ~12% category mislabeling noise, the same way a rushed agent picks the first plausible dropdown option. Entirely synthetic — no real customer or account data.

• harness/ — the evaluation half, four pieces that build on each other:

  • classifier.py — a keyword-rule classifier, a stand-in for whatever real classifier you'd deploy (an LLM call, an embedding lookup, a fine-tuned model). Loads its rules from generated_rules.json if present, falling back to a small hand-written rule set otherwise.

  • keyword_generator.py — builds those rules from data instead of by hand. Scores which words are most concentrated in each category (vs. every other category), after two dedupe passes: stemming ("install", "installing", "installation" collapse to one signal instead of splitting into three weaker ones, no model needed) and an optional model-based semantic check (--semantic-dedupe, defaults to a higher-capability model than the classifier or labeler) that catches cross-category overlap stemming can't, words like "router" and "modem" that share no root but mean the same kind of problem. Also the reason classifier.py matches on whole words, not substrings: the generator surfaced "all" as a top word for equipment_issue, which silently matched inside "install"/"installation" until that got fixed to word-boundary matching.

  • llm_labeler.py — the independent-labeling half. Calls a second model to label a sample of notes, kept deliberately separate from whatever categorizer is under test, since a system can't meaningfully grade itself. Runs in a dry-run mode with no API key required, so the rest of the pipeline can be inspected without live network access or an Anthropic key.

  • theme_eval.py — scores the classifier against trusted labels, either the dataset's built-in true_category (for testing the harness itself) or a labels file from llm_labeler.py (for a genuinely independent check). Reports per-category precision/recall/F1, a full confusion matrix, and the categories the classifier confuses most, the same "don't just report an aggregate score, tell me where it breaks" principle used in apex-overlay, applied here to text classification instead of tool-calling execution.

• tests/ — pytest suite covering every MCP tool, the eval harness, keyword generation, and the word-boundary matching fix, including a test that asserts the injected spike is actually detected and a test that would have caught the "all" bug directly.

Design philosophy

• Each MCP tool does exactly one job with a narrow, explicit input schema.

• Every tool returns structured JSON, never free-form prose, so both a human and an eval harness can score the output.

• The eval harness is a first-class part of this repo, not an afterthought: a classifier swapped in to fix the raw label noise needs to prove it's actually better, per category, before anyone trusts its output for a decision like "escalate this to network engineering."

• Synthetic data throughout, with injected ground truth, so both the MCP tools and the eval harness can be verified against a known-correct answer instead of eyeballed.

Setup

pip install -r requirements.txt
python data/generate_call_notes.py --out call_notes.db --days 120
pytest tests/ -v

# build classifier rules from the data instead of hand-writing them
python -m harness.keyword_generator --db call_notes.db --out harness/generated_rules.json

# optional: label a sample with a second, independent model (requires ANTHROPIC_API_KEY)
python -m harness.llm_labeler --db call_notes.db --sample 50 --out harness/llm_labels.json
python -m harness.theme_eval --db call_notes.db --labels-file harness/llm_labels.json

Register with Claude Code:

claude mcp add call-reason-insights -- python /path/to/call-reason-insights-mcp/server.py

Run the eval harness directly:

python -m harness.theme_eval --db call_notes.db

See examples/example_prompts.md for natural-language prompts that route to each tool.

Related

Part of a connected set of MCP + evaluation projects: fleet-data-mcp, apex-overlay, claude-code-eval-mcp.

License

MIT — see LICENSE.