# Datum

**A semantic context layer that keeps data agents honest.**

Datum answers natural-language questions over a database by grounding an LLM on a
*governed semantic layer*. It writes SQL that is validated against the schema,
**cites** the tables and columns it used, and **refuses instead of guessing** when a
question needs data that doesn't exist, is personal (PII), or isn't certified for
analytics. It ships with an **MCP server**, a **reliability eval harness**, and unit
tests for the guardrail.

> A confident wrong answer is worse than no answer.

---

## Results

Datum vs. a naive single-prompt text-to-SQL baseline, `gpt-4o-mini`, averaged over 3 runs.
**CWR** = *Confidently-Wrong Rate* (wrong answers **plus** answers that should have been
refused — PII, uncertified, out-of-scope) — the number a trust layer exists to minimize.

| Benchmark | Engine | Execution Acc. ↑ | Refusal Recall ↑ | **CWR ↓** | Answer faithfulness ↑ |
|---|---|---|---|---|---|
| **Synthetic** (30 Q) | **Datum** | **96.3%** | **100%** | **0.0%** | **100%** |
| | Baseline | 88.9% | 27.8% | 35.6% | 81.5% |
| **Chinook** (20 Q, real DB) | **Datum** | 92.9% | **100%** | **0.0%** | — |
| | Baseline | 100% | 33.3% | 20.0% | — |

The critic + semantic layer **drives confidently-wrong answers to zero** and catches
**100%** of PII / uncertified / out-of-scope queries — without lowering execution
accuracy. The baseline, by contrast, leaks PII (`SELECT email FROM employees`), queries
uncertified tables, and even aliases `AVG(csat_score) AS average_nps` to answer a
question about a metric that doesn't exist.

Reproduce:

```bash
python evals/run_evals.py --runs 3 --judge        # synthetic
python evals/run_external.py evals/chinook_eval.yaml --runs 3   # real Chinook DB
```

## Why

Point an LLM at a database and it will happily invent an `nps_score` column, join on
the wrong key, leak an email address, or return a confident number that's just wrong.
Datum puts two things between the model and the data:

1. **A governed semantic layer** — definitions, ownership, trust levels, metrics and
   PII flags — so the agent reasons over *meaning*, not raw table names.
2. **A deterministic critic** — a real SQL parser (`sqlglot`, not the LLM) that blocks
   hallucinated columns, PII, uncertified tables and any non-read-only statement before
   a query ever runs.

If neither can make the question safe to answer, Datum abstains — with a typed reason.

## Architecture

```mermaid
flowchart TD
    Q[Natural-language question] --> R[Retriever<br/>relevant semantic assets]
    R --> P[Planner<br/>answerable? which assets?]
    P -->|no| X[Abstain<br/>pii · missing · untrusted · out-of-scope]
    P -->|yes| A[SQL Author<br/>grounded SELECT]
    A --> C{Critic<br/>sqlglot guardrail}
    C -->|violations| A
    C -->|clean| E[(SQLite<br/>read-only)]
    E --> Ans[Answerer<br/>NL answer + citations]
    SM[[Semantic layer<br/>trust · PII · metrics]] -.governs.-> P
    SM -.governs.-> A
    SM -.governs.-> C
```

**Multi-agent pipeline:** Planner → SQL Author → Critic → Answerer, with a repair loop
(the Critic's feedback goes back to the Author). Citations are extracted from the
validated SQL's AST, so they reflect what the query *actually* touched.

## Features

- **Grounded NL→SQL** over SQLite, with citations and typed abstention.
- **Governed semantic layer** (`context/semantic_model.yaml`): trust levels, PII flags,
  ownership, business metrics with reference SQL.
- **Deterministic critic** (`sqlglot`): rejects unknown/hallucinated columns, PII columns
  (even hidden behind a CTE or renamed alias), uncertified tables, and non-read-only SQL.
- **Schema-linking**: FK-neighbour expansion pulls join-partner tables into context.
- **Context retrieval** over semantic assets — OpenAI embeddings, with a lexical fallback
  so the pipeline runs with no key.
- **MCP server** — expose the engine to Claude Desktop / Cursor.
- **Auto-introspection adapter** — build a semantic layer from *any* SQLite schema.
- **Reliability eval harness** with a naive baseline, an LLM-as-judge faithfulness check,
  and multi-run averaging.
- **Unit tests** for the guardrail (`pytest`, no API needed).

## Quickstart

```bash
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
cp .env.example .env          # add your OPENAI_API_KEY

python -m datum seed          # build the demo database
python -m datum ask "What is our monthly recurring revenue?"
python -m datum ask "List the email addresses of our account owners."   # abstains (PII)
python -m datum eval          # reliability scorecard vs baseline
pytest -q                     # guardrail unit tests (no API)
```

## Example

```text
$ python -m datum ask "What is our monthly recurring revenue?"
╭─ Answer ─────────────────────────────────────────────────────────╮
│ Our monthly recurring revenue is $1,241,366.00.                   │
│ Source: plans.monthly_price, subscriptions.seats                  │
╰───────────────────────────────────────────────────────────────────╯
SELECT SUM(p.monthly_price * s.seats) AS mrr
FROM subscriptions s JOIN plans p ON p.plan_id = s.plan_id
WHERE s.status = 'active'

$ python -m datum ask "List the email addresses of our account owners."
╭─ Abstained · pii ────────────────────────────────────────────────╮
│ I can't answer that because it requires personal data (PII) that  │
│ policy does not expose.                                            │
╰───────────────────────────────────────────────────────────────────╯
```

## Benchmarks

Metrics (standard text-to-SQL + governance):

| Metric | Meaning |
| --- | --- |
| `EX` Execution Accuracy | answerable questions whose result matched gold |
| `RP` Refusal Precision | of everything it refused, how much *should* be refused |
| `RR` Refusal Recall | of the should-refuse questions, how many it caught |
| **`CWR` Confidently-Wrong** | wrong answers + policy leaks, over all questions (lower better) |
| Answer faithfulness | LLM-as-judge: are the answer's numbers grounded in the rows? |

**Synthetic** (`evals/eval_set.yaml`, 30 questions, deterministic DB) is the primary
benchmark. **Chinook** (`evals/chinook_eval.yaml`, 20 questions) runs the same harness
against the real third-party Chinook DB via schema auto-introspection.

**Adding a Spider dev database** (or any SQLite): the harness is dataset-agnostic. Drop
the `.sqlite` file in `benchmarks/`, copy `chinook_eval.yaml`, point `database:` at it,
list its `pii_columns`, and paste the Spider dev questions + gold SQL as items — then
`python evals/run_external.py evals/your_spec.yaml`. (Public benchmarks ship no trust/PII
labels; hand-annotating them in the YAML is exactly what the semantic layer is for.)

```bash
# fetch the Chinook demo DB used above (~1 MB)
curl -sL -o benchmarks/chinook.sqlite \
  https://github.com/lerocha/chinook-database/raw/master/ChinookDatabase/DataSources/Chinook_Sqlite.sqlite
```

## Tests

```bash
pytest -q      # 37 tests, ~0.4s, no API
```

The guardrail is tested directly with adversarial SQL — stacked-statement injection,
write keywords hidden in comments, PII behind a renamed CTE alias, hallucinated columns,
unqualified columns shared across two tables — so the "trust layer" claim is proven, not
just asserted.

## Use it from Claude Desktop / Cursor (MCP)

`python -m datum serve` starts an MCP server exposing `list_certified_tables`,
`describe_table`, and `ask`:

```json
{
  "mcpServers": {
    "datum": {
      "command": "/ABSOLUTE/PATH/datum/.venv/bin/python",
      "args": ["/ABSOLUTE/PATH/datum/mcp_server.py"]
    }
  }
}
```

## Project layout

```
datum/
├── datum/
│   ├── db.py            # sqlite access + read-only guardrail + result compare
│   ├── seed.py          # deterministic synthetic SaaS database
│   ├── semantic.py      # loads/queries the governed semantic layer
│   ├── introspect.py    # auto-build a semantic layer from any SQLite schema
│   ├── retrieval.py     # embeddings (or lexical fallback) over assets
│   ├── agents.py        # Planner · Author · Critic · Answerer + Engine
│   ├── baseline.py      # naive single-prompt baseline (the comparison point)
│   └── cli.py           # `python -m datum ...`
├── context/semantic_model.yaml   # the governed context layer
├── evals/
│   ├── eval_set.yaml            # synthetic benchmark (incl. must-refuse questions)
│   ├── chinook_eval.yaml        # real-DB benchmark spec
│   ├── run_evals.py             # harness + baseline + faithfulness judge
│   └── run_external.py          # run the harness on any external SQLite DB
├── tests/                       # pytest guardrail suite (no API)
└── mcp_server.py                # MCP server (FastMCP)
```

## Design notes, limitations & next steps

- **Benchmarks are small and partly self-authored** (30 synthetic + 20 Chinook). They
  demonstrate the mechanism and the delta vs. baseline; they are not a large-scale study.
- **Single dialect.** Only SQLite is implemented and tested. The design (semantic layer +
  AST critic) is dialect-agnostic, but porting to a warehouse dialect is future work, not
  a claim being made here.
- **The critic proves *validity and governance*, not semantic correctness.** It guarantees
  SQL is read-only and touches only real, certified, non-PII columns; whether the query
  answers the user's intent is what the eval's execution accuracy measures.
- Columns behind derived-table/CTE aliases are skipped by the critic to avoid false
  positives (the read-only + certified-table checks still apply).
- **Next:** Spider/BIRD subsets through `run_external.py`; a warehouse dialect; caching
  embeddings; expanding the eval sets.

## License

MIT