ledgerline

The trust ledger for AI data agents.

AI agents are writing to your data catalog right now: descriptions, tags, lineage notes, incident diagnoses. Ledgerline answers the question nobody else is asking: which of them can you actually trust?

Every action an agent takes against DataHub is recorded as a claim with a confidence score. When ground truth arrives (an assertion fires, an incident resolves, a steward accepts or reverts a change), the claim is settled. Settled claims accumulate into a per-agent calibration ledger: Brier scores, calibration curves, and a statistical verdict on whether an agent's track record is skill or luck.

The ledger is written back into DataHub itself, so the next agent inherits not just the metadata but the reliability of whoever wrote it.

The part that does not need your agents

Most of this scores agents Ledgerline runs. The reconstruction engine does not.

It reads DataHub's own Timeline API, pulls the metadata changes that already happened in a live catalog, and turns each one into a settleable claim grouped by who wrote it. DataHub's own AI documentation, a third-party agent, a human steward: anything that has ever written to the catalog gets scored on its real record, with no instrumentation and nothing installed in its path.

A write that a later actor blanks or overwrites settles wrong. A write still standing after a survival window settles right, but cheaply. See TECHNICAL.md for why survival is deliberately weak evidence.

Related MCP server: Agent Identity MCP Server

Components

The agents

Three do forecasting, four do enrichment and governance proposals, three are meta agents whose subject matter is the agent economy itself, and one is a deliberately weak baseline.

naive is not a joke entry. The luck baseline for a proposal is the pooled acceptance rate across agents, so a pool of only excellent proposers means nobody can be distinguished from the pool they define. A realistic weak rival restores contrast, and it is also the market's actual null hypothesis: regex taggers and naming-convention scripts are what teams use today.

Quickstart

You need a running DataHub with the MCP server, Python 3.11+, and any OpenAI-compatible model endpoint. The default stack is an open-weight Qwen3 32B; no proprietary model is required. Full prerequisites and the environment table are in examples/README.md.

bash examples/run_all.sh

That seeds a 12-dataset warehouse graph into DataHub, turns four agents loose on it, settles their claims against ground truth, writes the earned trust back into the catalog, and proves the gateway rejects the agent that earned distrust while reads keep working. It ends with ALL STAGES PASSED or a nonzero exit. Each stage is also a script you can run on its own; the walkthrough covers them one at a time, including what to open in the DataHub UI afterward.

To put the gateway in front of your own agent, point its MCP client at ledgerline instead of the raw server:

LEDGERLINE_AGENT_ID=my-agent \
LEDGERLINE_POLICY=enforce \
LEDGERLINE_MIN_TRUST=55 \
python -m ledgerline.gateway

The agent keeps the identical tool surface. It now also accumulates a settled record and inherits the trust of whoever authored the metadata it reads.

Scoreboard

The public board reads three read-only projections of the ledger under row level security: the leaderboard, calibration curves per agent, and every individual claim. Schema in scoreboard/supabase/schema.sql; scripts/publish_scoreboard.py produces the rows.

Live at ledgerline-scoreboard.vercel.app.

Layout

ledgerline/          the package
  gateway.py           the MCP trust gateway
  claims.py            the claim ledger; settle.py settles against ground truth
  skill.py             skill-vs-luck decomposition and trust scores
  metasettle.py        claims about other claims, settled off the ledger
  provenance.py        read a catalog's own change history (Timeline API)
  reconstruct.py       turn that history into settled claims per writer
  writeback.py         projects trust and dossiers back into DataHub
  publish.py           builds the scoreboard projection
  agents/              the thirteen agents
  simulator/           the demo world, its timeline and its steward

examples/            the end-to-end walkthrough
scripts/             runnable stages and probes
scoreboard/          the Next.js scoreboard
tests/               the test suite

Tests

pip install -e ".[dev]"
pytest -q

100 tests, no external services required.

How it works

TECHNICAL.md covers the internals: the claim convention that keeps Brier scoring uniform across claim types, the per-claim-type null probabilities, the Monte Carlo and FDR machinery, the reconstruction settlement rule and why survival is deliberately cheap evidence, and how meta claims settle off other claims.

License

Apache 2.0