Skip to main content
Glama
README.md
# grounded-reports

LLM-written performance summaries that are **not allowed to be wrong about the numbers**.

A small, runnable distillation of a pattern I run in production: pre-compute the facts, let the
model write the prose, then validate every quantitative claim before anything ships — and retry
or fall back when it doesn't pass.

```
daily metrics ──▶ fact sheet ──▶ LLM draft ──▶ claim validator ──┬─▶ ship
                       ▲                                         │
                       └──────── retry with violations ◀─────────┘
                                                                 └─▶ (budget exhausted) deterministic fallback
```

No API key needed to try it. `--mock` runs the whole loop offline, including a deliberately
wrong first draft so you can watch the validator catch it.

## Why this exists

A narrative that says "signups grew" when they fell by 10% is worse than no narrative. The
model isn't bad at writing; it's unreliable at arithmetic and at keeping polarity straight
("cost per signup improved" means it went *down*). The fix is structural, not prompt-tuning:

1. **Ground.** Compute current vs prior, percent change, direction, and polarity *in code*.
   The model receives a fact sheet and is told to only restate it.
2. **Validate.** Parse the draft for direction words and percentages near metric mentions and
   check each one against the fact sheet. Rule-based, deterministic, free.
3. **Retry with feedback.** Violations go back to the model in the next turn. Static context is
   marked cacheable so retries are cheap.
4. **Fall back.** If the retry budget is exhausted, ship a plain template built from the facts.
   Boring, never wrong.
5. **Eval.** A fixed set of narratives with known defects keeps the validator honest as it
   evolves.

## Quickstart

```bash
git clone https://github.com/mike-larchick/grounded-reports
cd grounded-reports
python3 -m grounded_reports.cli --mock
```

You'll see the fact sheet, attempt 1 failing with a flagged sentence, attempt 2 passing, and
the final output. Then:

```bash
python3 evals/run_evals.py          # 11 validator cases, exit 1 on any failure
pip install pytest && pytest -q     # unit tests
```

### With a real model

```bash
pip install 'anthropic>=0.40'
export ANTHROPIC_API_KEY=...
python3 -m grounded_reports.cli            # default model: claude-sonnet-4-5
python3 -m grounded_reports.cli --json     # machine-readable, includes every attempt
```

### Validate your own text

```bash
python3 -m grounded_reports.cli --validate-only my_draft.txt
```

### Bring your own data

Any CSV with `date,sessions,signups,revenue,cost` and at least 2× the comparison window:

```bash
python3 -m grounded_reports.cli --data path/to/metrics.csv --period 14 --mock
```

## As an MCP server

The same three steps are exposed as tools so an agent can call them directly.

```bash
pip install 'mcp>=1.2'
python3 -m grounded_reports.mcp_server
```

Claude Desktop / any MCP client config:

```json
{
  "mcpServers": {
    "grounded-reports": {
      "command": "python3",
      "args": ["-m", "grounded_reports.mcp_server"],
      "cwd": "/path/to/grounded-reports"
    }
  }
}
```

Tools: `fact_sheet`, `validate`, `generate_report`. An agent that drafts its own narrative can
call `validate` on it before showing it to a user.

## What the validator checks

| Claim type | Example | How it's resolved |
|---|---|---|
| Direction | "signups **grew**", "cost **eased**" | Word → up/down/flat; compared to fact sheet |
| Polarity-dependent direction | "cost per signup **improved**" | *improved* = down for lower-is-better metrics |
| Percent change | "sessions were up **16%**" | Compared to actual within ±1.5 points |
| Level, not change | "conversion rate **of 3.2%**", "from 4.1%" | Recognized by preceding word; skipped |
| Multiple metrics per sentence | "revenue climbed while CPA rose" | Each claim attaches to the nearest preceding metric |

Run `python3 evals/run_evals.py` to see each of these exercised.

## What it deliberately doesn't do

- **No second LLM as judge.** The gate is rules so it's deterministic and costs nothing per run.
  A model-based judge can layer on top for softer qualities (tone, causality) — this catches the
  hard failures first.
- **No causal claims.** The prompt forbids inferring *why* a number moved; the validator doesn't
  try to adjudicate causality.
- **Not a general fact-checker.** It knows about the metrics in the fact sheet and nothing else.
  Unknown nouns are ignored, which is the safe default for a report that must only restate given data.

## Layout

```
grounded_reports/
  facts.py        load CSV → FactSheet (current, prior, % change, direction, polarity)
  validate.py     narrative + FactSheet → ValidationResult (violations with sentence context)
  generate.py     AnthropicGenerator (cacheable system prompt) and offline MockGenerator
  fallback.py     deterministic narrative from facts
  pipeline.py     generate → validate → retry → fallback
  cli.py          command line
  mcp_server.py   FastMCP server exposing the same steps as tools
data/sample_metrics.csv   28 days of synthetic data
evals/                    cases.json + runner
tests/                    pytest
```

## Background

The production version of this pattern runs weekly against a BigQuery warehouse and ships
client-facing executive summaries with no human review step. This repo is a clean-room rewrite
on synthetic data; none of the production code, prompts, or data are included.

MIT License.