MetricHelpDesk

Ask questions about your metrics in plain English, answered only through a MetricForge semantic layer. MCP server for Claude Code and Claude Desktop, plus an optional Claude API CLI. Python, MCP, DuckDB, MetricForge, Anthropic SDK.

                  ┌─ MCP server ──▶ Claude Code / Claude Desktop   (your Claude subscription)
metric tools ─────┤
(MetricForge)     └─ ask CLI ─────▶ Claude API                     (only with API credentials)

This file is the reference: what exists and how it works. RUNBOOK.md is the procedures, CLAUDE.md the agent rules, TASK.md the open items.

What it is

Four tools over a MetricForge project and a DuckDB warehouse, exposed two ways: as an MCP server that Claude Code or Claude Desktop launches, and as an ask loop against the Claude API. A seeded demo warehouse with one planted story comes with it, so the whole thing runs offline except for the model.

Related MCP server: pg-analytics-mcp

Why it exists

The model never writes SQL. It picks metrics and dimensions that MetricForge defines, and supplies filter values and dates, which are validated and escaped. So an answer can only use numbers from definitions you control, and it cannot make up a formula for "revenue".

Quick start

Needs Python 3.12+ and uv.

make build
make demo       # seeded 2024 e-commerce warehouse + metric definitions

With Claude Code (no API key): run claude in this directory. .mcp.json starts the server, so you can ask:

Why did revenue dip in July 2024?

Claude lists the metrics, sees revenue fall 7.9% while cancellations tripled, breaks the change down by country and payment method, and lands on German card payments. The full session is in transcripts/claude-code-july-dip.md.

To use it from Claude Desktop, add the same server to its config with an absolute path:

{"mcpServers": {"metric-helpdesk": {"command": "uv", "args": [
  "run", "--directory", "/path/to/metric-helpdesk", "metric-helpdesk", "serve"]}}}

Offline: make replay plays a saved session through the real tools; no Claude access needed.

With the Claude API: set ANTHROPIC_API_KEY (or ant auth login), then

uv run metric-helpdesk ask "Which product category brought in the most revenue in 2024?"
uv run metric-helpdesk ask "..." --record transcripts/<name>.json

Without credentials, ask explains the two options above instead of failing.

Your own data: point the server at any MetricForge project: metric-helpdesk serve --metrics path/to/metrics --db path/to/warehouse.duckdb.

Commands

The CLI runs as uv run metric-helpdesk <command>; the Make targets wrap the common calls. The table is regenerated from metric-helpdesk --help and the Makefile by the doc check; edit the help strings in src/metric_helpdesk/cli.py or the Makefile, not this table.

serve takes --metrics and --db to point at another MetricForge project; ask takes --record FILE to save a live session and --replay FILE to play one back through the real tools; eval takes --runner claude-code|api and --out FILE for the graded results. make evals and make evals-api are those two runners with their result files.

How it works

Tools

compare_periods is what makes "why" questions answerable: it reports each dimension value's change and, for sums and counts, its share of the total change. Averages and ratios get no share, because their parts do not add up to the whole.

Guardrails

• No model-written SQL. Metric and dimension names are checked against the catalog (with did-you-mean suggestions), filter values become escaped literals, and dates arrive as parsed dates. The SQL itself comes from MetricForge.

• Read-only warehouse. The DuckDB file is attached READ_ONLY into an in-memory database.

• Only questions MetricForge can answer. Metrics from different tables are joined on the dimensions every table declares, so list_metrics offers only those for a cross-table metric, and a group_by or filter outside them is refused with the valid names.

• Bounded results. At most 500 rows per query, with a truncated flag.

• Errors the model can act on. Bad names and arguments come back as tool errors with the fix in the message, so Claude corrects itself instead of guessing.

The demo data

A year of orders (~15k) and website sessions (~90k) with mild seasonality. One planted story: in July 2024 card checkout in Germany failed for most of the month, so German card orders were cancelled. June and July demand is otherwise equal. Revenue drops $7,265 (-7.9%), Germany accounts for 113% of it, and 129 of Germany's 130 extra cancellations are card orders. The tools can find all of this; nothing tells the model where to look.

Evals

evals/questions.yaml holds questions with checkable facts rather than reference answers: tools that must be called, numbers the answer must contain (computed from the warehouse when grading), and patterns that must or must not appear. One question (customer acquisition cost) has no answer in the data; the check is that the model says so. The latest recorded Claude Code run, answers included, is evals/results/claude-code.json; the score is in TASK.md.

Limits

• Metrics from different tables join on dimensions that share a name, so a dimension declared by one table only (the demo's device, payment_method) cannot group a cross-table query.

• The data explains where and when a number moved, not why in the business sense; answers are prompted to say so.

• DuckDB warehouses only, like MetricForge.

Layout

src/metric_helpdesk/
├── catalog.py      # queryable metrics and their dimensions
├── warehouse.py    # read-only attach
├── tools.py        # the four tools, no AI dependency
├── models.py       # tool inputs and outputs
├── server.py       # MCP server
├── ask/            # Claude API loop, tool schemas, record/replay
├── demo.py         # seeded demo warehouse
├── demo_metrics/   # demo MetricForge definitions
├── evals.py        # questions, grader, Claude Code runner
└── cli.py          # serve, demo, ask, eval
evals/              # questions and recorded results
transcripts/        # replayable and example sessions
tests/
.mcp.json           # registers the demo server for Claude Code
Makefile

make demo writes the warehouse under data/, which is gitignored; make clean removes it.

Outside the repo

Development

uv run ruff check . && uv run ruff format --check . && uv run mypy src/
uv run pytest

• Offline, in CI: tools, validation and escaping, the MCP server over the MCP protocol, the ask tool loop against a scripted client, and a full replay through the real tools. CI also builds the demo and runs the bundled replay.

• Live, by hand: the eval set through Claude Code, recorded in evals/results/.

• Not yet live: the Claude API path. transcripts/july-dip.json is a hand-built fixture (marked as such) showing the loop's shape; --record replaces it with a real session once credentials are available.

Licensed MIT (LICENSE).

Pointers

• RUNBOOK.md: run, record an eval, bump MetricForge, release, restore.

• CLAUDE.md: the agent rules.

• TASK.md: what is open, and the latest eval score.