provgraf

A "bank of verified facts" for AI agents — a W3C PROV knowledge graph on plain Postgres, with automatic staleness propagation.

⚠️ This is a proof-of-concept / engineering demo, not a supported product. It was designed, evaluated and deployed for one real client engagement (anonymised here), and is published to document the idea and its implementation. Expect rough edges; issues are welcome, a roadmap is not promised.

The problem

If you let an AI agent write client-facing documents, the model is not the risk — the facts are. A language model will happily "correct" a true, contract-grade number (a rent rate, a buyout rule, a share-capital figure) toward whatever the internet believes. In our real deployment, an overnight agent once tried to fix six verified facts to match plausible-but-wrong public sources.

Not every certain fact arrives as a PDF. Plenty of what you are told is true and consequential — a director confirms on a call how a scheme works, and you act on it. People also change their minds. So a source here is either a document with a file, or a testimony: a dated, attributed record that on this day this person vouched for this. That is what lets you stop relitigating a settled question, and it is why the integrity check differs by shape rather than demanding a file for everything.

There is a second, more mundane version of the same problem. An agent left to its own devices dumps everything into Markdown files as it goes — notes, half-verified numbers, copies of copies — with no validation at the door. If you're disciplined about your files, maybe you can live with that. If you're messy (I am), the workspace silently rots: three files disagree about the same number and none of them says where it came from. That was the actual trigger for building this.

provgraf is the counter-measure: a small database of atomic facts, each with provenance (which official document it came from, who put it there, when), plus one mechanism you don't get from a notes file:

Change a source → the system tells you everything downstream that just went stale.

It cuts both ways: the validating write path keeps the human's workspace from rotting, and the read-only agent surface keeps the agent honest — it can quote the bank, but it can't quietly "improve" it.

The idea in 60 seconds

• Facts are nodes (entity), sources are nodes, people/software are agents, decisions are activitys — W3C PROV core plus documented extensions (provenance_class, entity_status; PROV sanctions extension via prov:type subtyping).

• Derived facts (sums, report figures) link to their inputs via wasDerivedFrom and store an inputs_hash of the input values.

• When a source fact is revised, a recursive SQL CTE walks the derivation graph and flags every transitively dependent fact as stale (measured: a 2-level cascade in ~2.5 ms).

• Versioning is a single trick: valid_to + a partial unique index (WHERE valid_to IS NULL). That gives as-of queries over the bank's own history for free.

• The bank is bitemporal: valid_from/valid_to is transaction time (what the bank believed when), world_valid_from/world_valid_to is world time (when the fact holds per its source). Only both axes together answer "per what we knew on 15 June, what was in force in May" — the question a backdated correction creates. The four-timestamp model is borrowed from Graphiti.

• Contradictory sources coexist as disputed alternates; a human resolves them with a recorded decision (the rejected alternative stays in the graph as a trail).

• No triplestore, no graph database. Postgres gives constraints, transactions, pg_dump, and a recursive CTE is all the graph traversal this problem needs.

graph LR
    D1["📄 datasheet<br/>(source doc)"] --> F1["riverside.units = 152"]
    D1 --> F2["hillside.units = 84"]
    D1 --> F3["lakeside.units = 58"]
    F1 --> T["units_total = 294<br/>inputs_hash ✓"]
    F2 --> T
    F3 --> T
    T --> R["report.units = 294"]
    D2["📄 NEW datasheet<br/>units 152 → 154"] -.revise.-> F1
    F1 -.cascade.-> T
    T -.cascade.-> R
    style D2 fill:#fdd,stroke:#c33
    style T stroke:#c33,stroke-dasharray: 5 5
    style R stroke:#c33,stroke-dasharray: 5 5

What's implemented

How it was evaluated (the part that mattered)

The build was gated, not vibes-driven:

1. PRD first, reviewed by a panel of independent AI reviewer agents; their objections (e.g. "at ~30 facts Postgres barely beats a JSON file") were recorded as open tensions, not deleted.

2. A 1-day spike on flat files to prove the staleness-cascade design before writing any DDL.

3. A go/no-go milestone on real data: cascade correctness, version windows, invariants that actually reject bad writes, hash parity Python↔SQL.

4. Standards check: the PROV-JSON export deserializes cleanly with the reference W3C library, enforced by a test. Scope stated honestly: that proves the serialization is well-formed PROV-JSON, not PROV-CONSTRAINTS conformance (typing, causality loops, provenance travelling backwards in time), which needs a validator we have not run.

5. In production the bank grew to ~150 facts from 22 source documents and was used to fill investor-facing and grant documents, with a validator that blocks any hard number lacking a bank qname tag.

Quickstart (demo)

Requirements: Docker, uv.

docker compose -f infra/postgres/docker-compose.yml up -d --wait
uv run provgraf init
uv run pytest                      # 27 tests: hash parity, invariants, cascade, as-of, conflicts
bash examples/demo_cascade.sh      # seed a fictional company → check → revise a source → watch the cascade

(Run the tests before the demo — they read global check state, so a seeded database makes two of them fail. examples/reset.sh wipes and re-seeds; it connects through DATABASE_URL and refuses to touch a database holding anything other than the demo owner.)

The demo seeds a fictional social-housing company ("Acme Community Housing"), builds a 2-level derivation chain, plants a source conflict and an overdue fact, then revises one source number and shows check flagging the transitively dependent facts. It closes on the bitemporal query: a rent recorded today but in force since 1 June answers for June and correctly finds nothing in force in May.

Optional extras:

uv sync --group rag                # local embeddings + reranker (configure models in .env)
uv run provgraf embed acme-housing && uv run provgraf search "how much is the rent"
uv run --group dashboard streamlit run dashboard/app.py
uv run --group mcp provgraf-mcp    # read-only MCP server for AI agents

The default embedding/reranker models in .env.example are Polish (sdadas/mmlw-retrieval-roberta-large, sdadas/polish-reranker-large-ranknet) because the original deployment was Polish-language; swap them for any sentence-transformers-compatible pair.

Design decisions worth stealing

• Boring storage was the right call — and historically the norm. PROV is a data model, not a technology choice, and most deployed provenance recorders (Karma, Komadu, the IVOA provenance store) were relational too. Postgres gives constraints, transactions and pg_dump; a recursive CTE is all the graph traversal this problem needs. Presenting "PROV without a graph database" as a discovery would be a tell that you hadn't read the field.

• Hash parity enforced by tests. The staleness hash exists in Python and in a SQL function; a unit test feeds both the same fixtures. Divergence = the whole staleness feature silently lies.

• Read/write asymmetry for agents. Agents get a read-only MCP surface; writes go through a validating CLI with a human in the loop. Prompts can't enforce this — architecture can.

• Auto-glosses for retrieval. One dictionary entry per field type generates the embedded description for every fact of that type. New field → one JSON entry, not N hand-written descriptions.

• Decisions are nodes. Resolving a conflict creates an activity with an agent and a basis; precedents searches past decisions before you resolve a new dilemma.

Prior art, and what is actually different

Before publishing this I ran a prior-art survey across ~20 systems. Every ingredient here exists somewhere; the assembly is what does not. Being specific about that is more useful than a novelty claim:

• Graphiti / Zep — the closest neighbour and the one that beat us on an axis: genuine bitemporality with four timestamps per edge (which is why provgraf now has it). It has per-fact provenance back to the ingested episode. What it does not have is a staleness cascade — and its conflict handling is the mirror image of this project's: a small LLM decides at write time which contradicting fact loses and silently expires it, and its MCP surface hands the agent add_memory, delete_entity_edge and clear_graph.

• Dagster — hashes code and input data versions to mark downstream assets stale. That is inputs_hash + cascade, shipping and battle-tested, at the granularity of assets and tables. provgraf's difference is the unit (one number, with a citation) and the human in the loop, not the mechanism.

• Truth maintenance systems (Doyle 1979, de Kleer 1986) and content-addressed build systems (Nix, Bazel) are the real ancestors of justification-plus-invalidation. Nothing here is new under the sun; it is applied to facts instead of beliefs or build artifacts.

• TrustGraph — real prov: vocabulary in shipped code and a provenance CLI. Instructive twice over: it wrote this project's staleness feature down as a motivating use case and never built it, and it deliberately retreated from per-triple provenance to per-chunk containment because reification got expensive.

• WhyHow.AI — shipped per-triple → chunk → page-offset provenance back in 2024. The repositories have been untouched since, and the domain no longer resolves. Building this is demonstrably possible and demonstrably not sufficient on its own.

• Knowledge-base tools (Notion, Guru, Slab, Slite, Document360, GitBook) — verification exists, but the trigger is a calendar interval or an LLM judging that a document drifted from a connected source, and the unit is a page or a card. None of them holds a typed number, and none computes transitive dependents of a changed input. The industry is converging on probabilistic drift detection with accept/dismiss; the cascade here is deterministic and replayable. (Guru's "an edit by a non-owner unverifies the card" is where this project's agent gating comes from.)

• Agent memory (Mem0, Letta, LlamaIndex, LangMem, Google Memory Bank) — contradiction is resolved by overwriting or deleting, decided by a model. Mem0's prompt says it outright: if the retrieved facts contradict the memory, delete it. Coexisting disputed alternates plus a recorded human decision with a basis appears nowhere in that category.

• Not database provenance in the Green–Karvounarakis–Tannen sense (ProvSQL, GProM). That field annotates query results with semiring lineage; this is retrospective/workflow provenance. Different camp, easy to confuse.

Uncontested, as far as the survey found: fact-level staleness propagation, and conflicts that survive as alternates until a human records a decision. Everything else in the list above is prior art we are standing on.

Known limitations

Honest list (an adversarial code review ran before publishing; the notable leftovers):

• Clean PROV-JSON deserialization is tested; PROV-CONSTRAINTS conformance is not checked.

• No incremental recomputation: the cascade tells you what went stale, it does not recompute derived values for you.

• Single-user by design: no auth, no concurrency story beyond Postgres transactions.

• No rm-doc: documents are never deleted, only reported as dangling (with the orphaned facts they would take down).

Not built (on purpose)

• Query-time provenance — recording which facts fed a particular answer or document (TrustGraph does this). Likely belongs in the application consuming the bank rather than in the engine.

• LLM-resolved contradictions and auto-repairing memory. Both are well-trodden elsewhere and both defeat the point of a bank whose contents a human vouched for.

Working with it as an agent

If you are an AI agent operating on this bank, read AGENTS.md first: what you may read, what you may not write, and the two ways the machinery will stop you on purpose.

License

MIT