# ledgerline

**The trust ledger for AI data agents.**

AI agents are writing to your data catalog right now: descriptions, tags, lineage
notes, incident diagnoses. Ledgerline answers the question nobody else is asking:
which of them can you actually trust?

Every action an agent takes against [DataHub](https://datahub.com) is recorded as a
claim with a confidence score. When ground truth arrives (an assertion fires, an
incident resolves, a steward accepts or reverts a change), the claim is settled.
Settled claims accumulate into a per-agent calibration ledger: Brier scores,
calibration curves, and a statistical verdict on whether an agent's track record is
skill or luck.

The ledger is written back into DataHub itself, so the next agent inherits not just
the metadata but the reliability of whoever wrote it.

## The part that does not need your agents

Most of this scores agents Ledgerline runs. The reconstruction engine does not.

It reads DataHub's own Timeline API, pulls the metadata changes that **already
happened** in a live catalog, and turns each one into a settleable claim grouped by
who wrote it. DataHub's own AI documentation, a third-party agent, a human steward:
anything that has ever written to the catalog gets scored on its real record, with
no instrumentation and nothing installed in its path.

A write that a later actor blanks or overwrites settles wrong. A write still standing
after a survival window settles right, but cheaply. See
[TECHNICAL.md](TECHNICAL.md) for why survival is deliberately weak evidence.

## Components

| Component | What it does |
| --- | --- |
| **Worker agents** | Thirteen scaffolded agents doing real catalog work through the DataHub MCP server |
| **Settlement engine** | Matches claims to observed outcomes and scores every agent |
| **Skill engine** | Monte Carlo null, Benjamini-Hochberg FDR control, shrunk trust scores |
| **Trust gateway** | An MCP proxy in front of DataHub that stamps context with its author's settled trust. One URL swap for any MCP client |
| **Reconstruction engine** | Scores writers a catalog already has, from its change history |
| **Writeback** | Provenance tags, structured properties and per-agent dossiers, back into DataHub |
| **Scoreboard** | Live leaderboard with calibration curves and per-claim drill-down |

## The agents

Three do forecasting, four do enrichment and governance proposals, three are meta
agents whose subject matter is the agent economy itself, and one is a deliberately
weak baseline.

| Agent | Claim | Settles on |
| --- | --- | --- |
| `blastradius` | Which downstream assets a dropped column actually breaks | Assertion results |
| `sentinel` | Whether a feed will miss its next SLA | The SLA window closing |
| `triage` | The root cause of an incident | Incident resolution |
| `enricher` | A description for an undocumented column | Steward review |
| `tabledesc` | A dataset-level description | Steward review |
| `piitagger` | Which columns hold personal data, typed | Steward review |
| `termmapper` | Which glossary term a column means | Steward review |
| `ownerrec` | The owning team for a dataset | Review |
| `domainassign` | The business domain a dataset belongs to | Review |
| `arbiter` | When two agents conflict, which proposal holds | The chosen target settling |
| `revertpredictor` | Which incoming writes a steward will revert | The target claim settling |
| `auditor` | Which datasets were edited around the gateway | A provenance sweep |
| `naive` | Pattern-matched filler, always confident | Steward review |

`naive` is not a joke entry. The luck baseline for a proposal is the pooled acceptance
rate across agents, so a pool of only excellent proposers means nobody can be
distinguished from the pool they define. A realistic weak rival restores contrast, and
it is also the market's actual null hypothesis: regex taggers and naming-convention
scripts are what teams use today.

## Quickstart

You need a running DataHub with the MCP server, Python 3.11+, and any
OpenAI-compatible model endpoint. The default stack is an open-weight Qwen3 32B; no
proprietary model is required. Full prerequisites and the environment table are in
[examples/README.md](examples/README.md).

```bash
bash examples/run_all.sh
```

That seeds a 12-dataset warehouse graph into DataHub, turns four agents loose on it,
settles their claims against ground truth, writes the earned trust back into the
catalog, and proves the gateway rejects the agent that earned distrust while reads
keep working. It ends with `ALL STAGES PASSED` or a nonzero exit. Each stage is also a
script you can run on its own; the walkthrough covers them one at a time, including
what to open in the DataHub UI afterward.

To put the gateway in front of your own agent, point its MCP client at ledgerline
instead of the raw server:

```bash
LEDGERLINE_AGENT_ID=my-agent \
LEDGERLINE_POLICY=enforce \
LEDGERLINE_MIN_TRUST=55 \
python -m ledgerline.gateway
```

The agent keeps the identical tool surface. It now also accumulates a settled record
and inherits the trust of whoever authored the metadata it reads.

## Scoreboard

The public board reads three read-only projections of the ledger under row level
security: the leaderboard, calibration curves per agent, and every individual claim.
Schema in [scoreboard/supabase/schema.sql](scoreboard/supabase/schema.sql);
`scripts/publish_scoreboard.py` produces the rows.

Live at [ledgerline-scoreboard.vercel.app](https://ledgerline-scoreboard.vercel.app).

## Layout

```
ledgerline/          the package
  gateway.py           the MCP trust gateway
  claims.py            the claim ledger; settle.py settles against ground truth
  skill.py             skill-vs-luck decomposition and trust scores
  metasettle.py        claims about other claims, settled off the ledger
  provenance.py        read a catalog's own change history (Timeline API)
  reconstruct.py       turn that history into settled claims per writer
  writeback.py         projects trust and dossiers back into DataHub
  publish.py           builds the scoreboard projection
  agents/              the thirteen agents
  simulator/           the demo world, its timeline and its steward

examples/            the end-to-end walkthrough
scripts/             runnable stages and probes
scoreboard/          the Next.js scoreboard
tests/               the test suite
```

## Tests

```bash
pip install -e ".[dev]"
pytest -q
```

100 tests, no external services required.

## How it works

[TECHNICAL.md](TECHNICAL.md) covers the internals: the claim convention that keeps
Brier scoring uniform across claim types, the per-claim-type null probabilities, the
Monte Carlo and FDR machinery, the reconstruction settlement rule and why survival is
deliberately cheap evidence, and how meta claims settle off other claims.

## License

Apache 2.0