Skip to main content
Glama

grounded-reports

LLM-written performance summaries that are not allowed to be wrong about the numbers.

A small, runnable distillation of a pattern I run in production: pre-compute the facts, let the model write the prose, then validate every quantitative claim before anything ships — and retry or fall back when it doesn't pass.

daily metrics ──▶ fact sheet ──▶ LLM draft ──▶ claim validator ──┬─▶ ship
                       ▲                                         │
                       └──────── retry with violations ◀─────────┘
                                                                 └─▶ (budget exhausted) deterministic fallback

No API key needed to try it. --mock runs the whole loop offline, including a deliberately wrong first draft so you can watch the validator catch it.

Why this exists

A narrative that says "signups grew" when they fell by 10% is worse than no narrative. The model isn't bad at writing; it's unreliable at arithmetic and at keeping polarity straight ("cost per signup improved" means it went down). The fix is structural, not prompt-tuning:

  1. Ground. Compute current vs prior, percent change, direction, and polarity in code. The model receives a fact sheet and is told to only restate it.

  2. Validate. Parse the draft for direction words and percentages near metric mentions and check each one against the fact sheet. Rule-based, deterministic, free.

  3. Retry with feedback. Violations go back to the model in the next turn. Static context is marked cacheable so retries are cheap.

  4. Fall back. If the retry budget is exhausted, ship a plain template built from the facts. Boring, never wrong.

  5. Eval. A fixed set of narratives with known defects keeps the validator honest as it evolves.

Related MCP server: Semantic Rails MCP Server

Quickstart

git clone https://github.com/mike-larchick/grounded-reports
cd grounded-reports
python3 -m grounded_reports.cli --mock

You'll see the fact sheet, attempt 1 failing with a flagged sentence, attempt 2 passing, and the final output. Then:

python3 evals/run_evals.py          # 11 validator cases, exit 1 on any failure
pip install pytest && pytest -q     # unit tests

With a real model

pip install 'anthropic>=0.40'
export ANTHROPIC_API_KEY=...
python3 -m grounded_reports.cli            # default model: claude-sonnet-4-5
python3 -m grounded_reports.cli --json     # machine-readable, includes every attempt

Validate your own text

python3 -m grounded_reports.cli --validate-only my_draft.txt

Bring your own data

Any CSV with date,sessions,signups,revenue,cost and at least 2× the comparison window:

python3 -m grounded_reports.cli --data path/to/metrics.csv --period 14 --mock

As an MCP server

The same three steps are exposed as tools so an agent can call them directly.

pip install 'mcp>=1.2'
python3 -m grounded_reports.mcp_server

Claude Desktop / any MCP client config:

{
  "mcpServers": {
    "grounded-reports": {
      "command": "python3",
      "args": ["-m", "grounded_reports.mcp_server"],
      "cwd": "/path/to/grounded-reports"
    }
  }
}

Tools: fact_sheet, validate, generate_report. An agent that drafts its own narrative can call validate on it before showing it to a user.

What the validator checks

Claim type

Example

How it's resolved

Direction

"signups grew", "cost eased"

Word → up/down/flat; compared to fact sheet

Polarity-dependent direction

"cost per signup improved"

improved = down for lower-is-better metrics

Percent change

"sessions were up 16%"

Compared to actual within ±1.5 points

Level, not change

"conversion rate of 3.2%", "from 4.1%"

Recognized by preceding word; skipped

Multiple metrics per sentence

"revenue climbed while CPA rose"

Each claim attaches to the nearest preceding metric

Run python3 evals/run_evals.py to see each of these exercised.

What it deliberately doesn't do

  • No second LLM as judge. The gate is rules so it's deterministic and costs nothing per run. A model-based judge can layer on top for softer qualities (tone, causality) — this catches the hard failures first.

  • No causal claims. The prompt forbids inferring why a number moved; the validator doesn't try to adjudicate causality.

  • Not a general fact-checker. It knows about the metrics in the fact sheet and nothing else. Unknown nouns are ignored, which is the safe default for a report that must only restate given data.

Layout

grounded_reports/
  facts.py        load CSV → FactSheet (current, prior, % change, direction, polarity)
  validate.py     narrative + FactSheet → ValidationResult (violations with sentence context)
  generate.py     AnthropicGenerator (cacheable system prompt) and offline MockGenerator
  fallback.py     deterministic narrative from facts
  pipeline.py     generate → validate → retry → fallback
  cli.py          command line
  mcp_server.py   FastMCP server exposing the same steps as tools
data/sample_metrics.csv   28 days of synthetic data
evals/                    cases.json + runner
tests/                    pytest

Background

The production version of this pattern runs weekly against a BigQuery warehouse and ships client-facing executive summaries with no human review step. This repo is a clean-room rewrite on synthetic data; none of the production code, prompts, or data are included.

MIT License.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      A
      quality
      A
      maintenance
      Production Model Context Protocol (MCP) server for business — replaces stale generic descriptions with concrete 2026 technical intelligence and data agents. Exposes dynamic KPI analytics, forecasting, anomaly detection, and schema resources across 10 enterprise domains via FastMCP over stdio and SSE transports.
      9
      AGPL 3.0
    • F
      license
      Not graded
      quality
      B
      maintenance
      Enables agents to perform controlled enterprise data queries through semantic intent, with runtime validation of statistics, filters, granularity, permissions, and physical bindings. Exposes tools like semantic_query for safe, fail-closed access to data horizons and capabilities.
      -
    • A
      license
      Not graded
      quality
      C
      maintenance
      Enables agents to run hybrid retrieval, build token-budgeted attributed context packs, answer questions with citation and groundedness-gated release, explore temporal graph entities, evaluate claim support, and ingest screened, embedded documents through typed MCP tools.
      Apache 2.0