Attest a dataset against its schema
attest_datasetProves a written dataset holds its declared schema by recomputing invariants with DuckDB, hashing files by SHA-256, and writing in-toto/DSSE attestations with a verifier script.
Instructions
Prove a written dataset holds what its schema declared, in a standard format.
Recomputes every invariant the schema implies (outcome-curve totals, rate
anchors, group shares, ledger identities, foreign keys, unique columns)
with DuckDB against the files, and pins each file by SHA-256. Writes
attestation.intoto.json (an in-toto v1 Statement that cosign and
GitHub's actions/attest accept), attestation.dsse.json and
verify.py into dataset_dir. Anyone can rerun python verify.py
with only duckdb installed.
Use it after generate_from_schema with the same schema and seed, and
report result to the user: PASSED means every declared property
holds in the files on disk.
Args:
dataset_dir: Folder with one <table>.csv or <table>.parquet
per table (the output_dir of generate_from_schema).
schema: The same schema dict passed to generate_from_schema.
seed: The seed used to generate, recorded in the Statement.
max_invariants: How many invariants to include in the response.
Returns:
{"ok", "result", "invariants_held", "invariants_total", "failures", "subjects", "files", "invariants"}.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| seed | No | ||
| schema | Yes | ||
| dataset_dir | Yes | ||
| max_invariants | No |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| result | Yes |