Skip to main content
Glama
kpi-agent

kpi-agent

by WiteKnite

kpi-agent-mcp-eval

A small prototype of a conversational BI agent with an evaluation harness.

  • an MCP server that exposes read-only analytics tools over a sales-funnel database

  • a tool-using agent (Anthropic Messages API) that answers KPI questions with those tools

  • a golden question set whose expected answers are computed from SQL, and a scoring script

  • optional Langfuse tracing and scores

It is a weekend-sized prototype built to show the pattern: tools behind MCP, an agent that must call them, and an evaluation that does not trust the model to grade itself. The data is synthetic. It is not a production system.

Layout

data/generate_data.py      synthetic, deterministic SQLite funnel data (24 months of leads, 2025-2026)
src/kpi_agent/tools.py     read-only tools: list_tables, describe_table, run_query, kpi_summary (optional channel and rep filters), kpi_drivers, forecast_kpi
src/kpi_agent/mcp_server.py  MCP server (stdio) exposing the same tools
src/kpi_agent/agent.py     tool-use loop with the Anthropic SDK
src/kpi_agent/tracing.py   optional Langfuse tracing (no-op without keys)
eval/build_golden.py       builds eval/questions.json from SQL
eval/run_eval.py           scores the agent, or validates the golden set with --offline
tests/test_tools.py        pytest checks for the tools and the golden set

Related MCP server: Enterprise MCP Knowledge Platform

Run it

python -m venv .venv
.venv\Scripts\activate            # macOS/Linux: source .venv/bin/activate
pip install -r requirements.txt

python data/generate_data.py      # creates data/sales.db
python eval/build_golden.py       # regenerates eval/questions.json (already committed)
pytest                            # 35 tests, no API key needed
python eval/run_eval.py --offline # validates the golden set against the tools

With an Anthropic key (copy .env.example, or set the variables in your shell):

python -m kpi_agent.agent "Which channel converted best in 2025-12?"
python eval/run_eval.py           # runs the agent on every question and scores it
python -m kpi_agent.mcp_server    # serve the tools to any MCP client

Set LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY to send traces and a correct score per question to Langfuse.

Demo in five minutes

  1. pytest shows the tools, the read-only guard, a real MCP round trip over stdio, and the agent loop (with a stubbed model client) all passing.

  2. python eval/run_eval.py --offline shows the golden questions and their SQL-derived answers.

  3. Open the tools in an MCP client. For example, with the MCP Inspector:

    npx @modelcontextprotocol/inspector python -m kpi_agent.mcp_server

    Run from the repo root with PYTHONPATH=src. Any MCP client works with this server entry:

    {
      "mcpServers": {
        "kpi-agent": {
          "command": "python",
          "args": ["-m", "kpi_agent.mcp_server"],
          "env": {"PYTHONPATH": "/absolute/path/to/kpi-agent-mcp-eval/src"}
        }
      }
    }
  4. With ANTHROPIC_API_KEY set, python eval/run_eval.py runs the real agent on every question and prints PASS or FAIL per question. A FAIL is a useful result: it shows what the evaluation catches.

What the evaluation does

Each question in eval/questions.json has an expected answer computed directly from the database. Numbers must match within a tolerance, and names must appear in the answer. The model never produces the expected values, so a wrong answer is caught instead of graded by the model.

Design choices

  • Read-only by construction. run_query accepts a single SELECT or WITH statement, and the database is opened with SQLite mode=ro, so a write fails even if the check is bypassed.

  • One set of tools. The MCP server and the agent call the same functions in tools.py.

  • Tracing is optional. Without Langfuse keys the decorators are no-ops, so the repo runs anywhere.

Limits

  • Synthetic data and eleven questions: a demonstration of the method, not a benchmark. Two of the eleven (fc_conv_jan27, fc_leads_jan27) check that the agent calls forecast_kpi and reports its number. Their expected values are a plain SQL average of the last three months. They test the agent's tool use, not forecast accuracy.

  • Channel and rep cells have few conversions each (often under 10 a month), so a single rate can swing a lot. Read per-channel and per-rep rates as indicative, not precise.

  • The offline tests and the golden-set check run without an API key. The online agent run and the Langfuse integration need keys and are not part of the automated tests.

  • Live runs on 14 November 2026 with claude-sonnet-5-5:

    • 9 of 9 questions, before the forecast questions were added (eval/results/live_20261114_9q_claude-sonnet-5-5.json).

    • 11 of 11 questions, including the two forecast questions (eval/results/live_20261114_11q_claude-sonnet-5-5.json). Two earlier runs (6 of 6 each) were recorded in text only. Langfuse tracing has not been exercised yet. The forecast questions check tool use, not forecast accuracy.

  • kpi_drivers ignores mix effects, so its contributions only approximate the total change.

  • forecast_kpi uses the mean of the last 3 months, the model the forecast study chose. It is not a statistical model. The 2026 months continue the 2025 post-launch pattern, so backtests on them are kinder to the model than real data would be.

  • The backtest uses 7 development-window origins with a 3-month horizon, on synthetic data. That is a small sample. Its MAE and MAPE can disagree, so both are reported.

Forecast backtest

forecast_kpi forecasts the next 3 months with the mean of the last 3 months, and backtests it against a naive "last value repeats" baseline. The backtest uses the study's protocol: origins 9 to 15, horizon 3, development window only (through 2026-06), so the holdout is never used to score the tool. Output from the current data (7 origins):

Metric

Model MAE

Naive MAE

Model MAPE %

Naive MAPE %

Beats naive on MAE

Beats naive on MAPE

Conversion rate (%)

1.31

1.57

17.8

22.3

yes

yes

Leads per month

20.6

19.4

10.2

9.5

no

no

Avg first response (min)

0.087

0.089

2.9

2.9

yes, negligibly

yes, negligibly

Only conversion beats naive clearly, and the study's confidence interval for that gap still touches zero. Leads lose on this protocol. Response time ties. An earlier version of this table, which included the holdout and earlier origins, showed leads winning. That result came from the wrong evaluation window and is withdrawn. The conversion result is also partly in-sample, since the same development data chose the model. The agent's answers about these forecasts have not yet been checked with a live model run.

Forecast study (pre-registered)

DECISIONS.md fixes the data split, candidates and choice rule before any results. eval/forecast_study.py runs the development steps (--step 1 to 3), and its --holdout run refuses to repeat.

Development (2025-01 to 2026-06), horizon 3, 210 backtest points per metric across 10 series:

  • Conversion: ma3 (3-month average) MAE 3.38 vs naive 4.30. Clearly better (CI excludes zero).

  • Leads: ma3 8.16 vs naive 9.16. Clearly better.

  • Response time: ma3 0.126 min vs naive 0.140. Better, but the difference is tiny.

  • Post-launch training fixed the response-time failure of the trend model (36.5 min error to 0.10). A log transform made no difference.

Holdout (2026-07 to 2026-12), run once, horizon 6, pooled over 10 series:

  • Conversion: ma3 3.35 vs naive 4.08 (CI [-1.33, -0.13]). Holds.

  • Leads: ma3 8.57 vs naive 9.38 (CI [-1.64, +0.08]). Points the same way, but not clearly.

  • Response time: ma3 0.146 vs naive 0.173 (CI [-0.052, -0.003]). Holds, but small.

Limits: one holdout forecast, 6 months, on synthetic data; the 10 series overlap, so the points are not independent. forecast_kpi still uses the trend model, which the study did not select.

Next steps

  • Add questions that should be refused or answered "no data", and score those too.

  • Compare models and prompts on the same golden set and track the scores in Langfuse.

  • Add an MCP client integration test.

MIT licensed.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    Related MCP Connectors

    Related MCP Servers

    • F
      license
      Not graded
      quality
      D
      maintenance
      Enables AI assistants to query internal business data for insights into customers, revenue, subscriptions, sales, and churn through controlled, read-only MCP tools.
      -
    • F
      license
      Not graded
      quality
      C
      maintenance
      Enables governed, read-only SQL analytics, hybrid RAG knowledge search, financial metric calculation, and database schema introspection through MCP, with role-based access control, audit logging, and AI observability.
      -
    • A
      license
      A
      quality
      B
      maintenance
      Enables MCP clients to query Google Analytics 4 reporting and configuration data through 27 read-only tools, including pivots, funnels, realtime reports, Admin API inspection, and diagnostics, with identifiers redacted by default.
      27
      Apache 2.0
    • A
      license
      Not graded
      quality
      B
      maintenance
      Enables asking product analytics questions in English and receiving answers as real SQL along with the resulting numbers, through read-only MCP tools for funnels, retention, trends, paths, and event counts.
      MIT