# Iris — verifiable BI over federated data

A business user asks a question, gets a number, and decides on it. Usually nobody in that
loop — not the user, not the analyst, not the system itself — can tell whether the number
came from the data or from the model's imagination.

Iris is an attempt at the other thing: **a number you can open.**

```
Q: Tip rate for card payments by borough on rainy versus dry days

  Manhattan  dry   23.54%      ← click it
  Manhattan  wet   23.62%

  displayed 23.54  ·  recomputed from these rows 23.54  ·  ✓ agrees
  3,979,472 contributing rows (card-paid, positive fare)
    fare=13.27  tip=2.00  pickup=2024-01-27 00:38  file=yellow_tripdata_2024-01.parquet
```

Three sources, three engines, two joins, one question — and a receipt.

![Iris architecture — one question, stage by stage](docs/architecture.gif)

*Regenerate with `.venv/bin/python scripts/make_architecture_gif.py`; the row counts are
read from `data/load_manifest.json`, so the diagram cannot drift from the data.*

---

## Three claims, and the evidence for each

**1. The model cannot invent a metric.** It never writes SQL. It emits a `SemanticQuery`
naming measures and dimensions from a declared manifest; anything else is rejected before
execution, with the valid names in the error. A deterministic compiler turns the validated
struct into SQL. → `semantic/`, `tests/test_planner_fake_llm.py`

**2. Every number opens into its source rows.** Click a figure and Iris returns the rows
behind it — each naming the vendor file it came from — plus the measure *recomputed over
exactly those rows*. If the two disagree, it says so. → `verify/drilldown.py`

**3. The evaluation can measure things the agent's own machinery can't express.** Ground
truth is hand-written SQL against the raw views, so a case is not limited to the shapes
the system already handles. → `eval/fixtures/`

Where those claims stop is in [Limitations](#limitations-owned-not-buried), and the
scoring method is in [`docs/eval_methodology.md`](docs/eval_methodology.md).

---

## Numbers

Current run — `eval/results/latest.json`, 15 cases × 3 runs:

| Category | Result | What it means |
|---|---|---|
| `in_manifest` | **5/5 clean, 100%** | questions phrased the way the manifest phrases things |
| `held_out` | **4/4 clean, 100%** | questions whose key terms are *absent* from the manifest |
| `ambiguous` | **2/2 clean, 100%** | correct behaviour is to ask, not answer |
| `adversarial` | **4/4 clean, 100%** | unanswerable, injected, or empty |
| grounding | **30/30** | every figure in every summary appeared in its table |

p50 **3.4s**, $0.35 for all 45 runs. **124 tests**, of which **79 need no database, no
services and no API key** — including the ones covering the no-fabrication guarantee.

Never quote a single-run number from this harness: planning is an LLM call and is not
deterministic. Use `--repeat`; a case that passes sometimes is reported as **FLAKY** with
its rate rather than resolved by whichever way the dice fell.

---

## Data — three independent origins

| Source | Origin | Lands in | Rows |
|---|---|---|---|
| Yellow-taxi trips, 2024 Q1 | NYC TLC | Parquet ("the lake") | 9,554,778 |
| Taxi zone lookup | NYC TLC | PostgreSQL | 265 |
| GHCN-Daily weather, 4 stations | NOAA | MongoDB | 5,204 |

Nobody designed these to fit together, so their seams are real. Six were measured and are
catalogued in [`docs/join_hazards.md`](docs/join_hazards.md). The important one:

> **"What is the average tip rate?" has three defensible answers, 9.4 points apart** —
> 17.60%, 22.39%, 27.00% — because cash fares never record a tip (1,330,105 trips, 13.92%,
> 100% zero-tip). Iris does not pick one. The manifest declares the ambiguity and the agent
> **asks**.

Runner-up: NOAA publishes tenths, so a naive loader reports `TMAX=83` as 83 °C for a New
York January. Plausibly typed, never errors, poisons every aggregate downstream.

---

## Quick start

```bash
python3.12 -m venv .venv
.venv/bin/python -m pip install -r requirements.txt
( cd ui && npm install )

cp .env.example .env          # set ANTHROPIC_API_KEY
./run.sh load                 # download + load all three sources (~215 MB, one-off)
./run.sh                      # UI :9300 · API :9301 · MCP :9302
```

Requires Python 3.12+, Node 18+, a local PostgreSQL, and Docker (for Iris's own MongoDB on
27018 — it will not write into a container it does not own). `run.sh` refuses to start on a
bound port and prints who holds it. See [`docs/local_setup.md`](docs/local_setup.md).

```bash
.venv/bin/python -m pytest tests/ -q          # 124 tests
.venv/bin/python -m eval.harness --repeat 3   # the eval (needs data + a key)
```

---

## How it works

```
   question
      │
      ▼
  ┌─────────┐  sees only declared measures and dimensions — never a column
  │  plan   │  emits SemanticQuery  ────────► clarify   (declared ambiguity)
  └────┬────┘                        └──────► refuse    (nothing can answer it)
       │ validated struct
       ▼
  ┌──────────┐  deterministic. Rejections go back to the planner verbatim, once.
  │ compile  │  join-path resolution · fan-out refusal · per-measure FILTER
  └────┬─────┘
       ▼
  ┌──────────┐   DuckDB: Parquet natively · Postgres ATTACH · Mongo → Arrow
  │ execute  │
  └────┬─────┘
       ▼
  ┌──────────┐   join match rates · sentinel impact · drill-down
  │  verify  │
  └────┬─────┘
       ▼
  ┌──────────┐   every numeral must appear in the table, or it is rewritten
  │ narrate  │
  └──────────┘
```

**The manifest is the contract.** `semantic/manifest.yaml` declares sources, entities,
joins *with cardinality*, dimensions *with sentinel values*, measures *with their
populations*, and ambiguity groups. Unit conversions and metric definitions live there and
nowhere else.

**Fan-out is refused, not hoped away.** Aggregating across a one-to-many join multiplies
every additive measure. The compiler checks join direction and declines. A test builds the
unsafe weather join by hand and shows the exact 4× inflation it would have produced.

**Portability is tested, not asserted.** `tests/test_portability.py` takes an unrelated
dataset (library lending), supplies only data and a manifest, and gets correct
hand-computable answers. **Zero** dataset references survive in executable code across
`federation/` and `semantic/`; seven remain in docstrings, where they are explaining a
hazard rather than driving behaviour. A test tokenises both packages and holds that
number, so a stray `if borough ==` cannot appear unnoticed.

Two honest edges on that claim. The foreign dataset is Parquet-only, so it demonstrates
*dataset* portability — a new schema, new manifest, unchanged code — not *engine*
portability; a second Postgres or Mongo deployment is not spun up in tests. And "adding an
engine is one class with three methods plus a registry line" is a structural fact about
`federation/connectors/factory.py`, not something a test proves end-to-end; what is tested
is that an unknown engine fails with a message naming the three methods.

**MCP is an external surface.** `describe_semantic_model`, `query_semantic` (verified),
`drill_down`, and `run_sql` — the last returning `verified: false` with a warning naming
what it bypassed. The agent itself talks to the engine in-process.

---

## Limitations, owned not buried

**The eval validates the compiler, not the definitions.** Reference SQL checks that Iris
computes what the manifest *declares*. If `avg_tip_rate_card` is declared wrongly, the
fixture written to match it is wrong the same way. That is exactly why the tip-rate case is
a `clarify` case rather than an `answer` case — the choice of definition is put to the
user, not graded against one.

**`run_sql` has none of the guarantees.** It exists for the long tail the manifest doesn't
cover. Results carry `verified: false` and the UI badges them. Anything through that path is
ordinary text-to-SQL.

**Two of the three sources are live; MongoDB is a snapshot.** Parquet is read from disk per
query and Postgres is `ATTACH`ed, so both reflect the current state. The Mongo collection is
materialised into Arrow **once at startup** — a document written afterwards is invisible until
`refresh()` or a restart. That is a deliberate choice for a cached third-party feed of 5,204
station-days, not a limit of the bridge, but "federated across three engines" should not be
read as "three live connections." `tests/test_federation.py` proves the boundary by writing to
MongoDB and asserting the view does *not* move.

**Pushdown is narrower than federation demos usually imply.** Every connector reports only
`filter` and `project`, and only where `EXPLAIN` confirms it below the scan node. Joins and
aggregates always run in DuckDB. An earlier version of these connectors claimed `join` and
`aggregate` pushdown; the query plan did not support it, and the test now pins each claim to a
real plan.

**15 cases is small.** All four categories at 100% over 3 runs means *no failures observed
in 45 runs*, not *proven correct*. Earlier single runs did show flaky failures — that is why
`--repeat` exists and why FLAKY is a distinct verdict.

**The planner sees the whole manifest.** Fine at 10 measures and 11 dimensions; at hundreds
this needs retrieval rather than dumping everything into the prompt. That is the first thing
that breaks at scale.

**Held-out is a vocabulary test, not a reasoning test.** It proves the planner maps
unfamiliar wording onto declared names. It does not prove it would handle a genuinely novel
analytical shape.

**Single dataset.** Portability is demonstrated on a synthetic second dataset in tests, not
on a second production-scale one.

---

## Layout

```
semantic/     manifest.yaml (the contract) · model · query + validation · compiler
federation/   DuckDB engine + per-ENGINE connectors, built from the manifest
verify/       provenance · drilldown · grounding
agent/        planner (LLM → SemanticQuery) · graph (3 terminal states) · llm
eval/         cases · scorer · harness · fixtures/*.sql (hand-written ground truth)
data_pipeline/  the three loaders — load faithfully, filter in the semantic layer
api/ ui/      FastAPI + Next.js dashboard and Trace Inspector
toolserver/   MCP server (external clients)
docs/         join_hazards · design_decisions (ADRs) · eval_methodology · local_setup
```

Built with Claude (Sonnet 5 planning, Haiku 4.5 narration) · DuckDB · LangGraph · MCP ·
FastAPI · Next.js.