Fadlie

The same data lives in four systems. Your catalog knows every path between them. Only one copy is governed.

Fadlie is an MCP agent for DataHub. It finds datasets that hold the same data across different platforms, then finds the governance — owners, domain, descriptions, PII tags, glossary terms — that stopped at one of them. It can level the difference, and it never writes a value it did not read from a dataset it names.

Built for Build with DataHub: The Agent Hackathon, on the contest's own showcase-ecommerce dataset. Every number below is produced by a script in this repository, against a live DataHub instance.

What the measurements found

Run python scripts/mesurer-jumeaux.py to reproduce all of this.

The lineage graph tells you nothing about whether two datasets hold the same data. The graph is a single component: 103 nodes, 161 edges, no isolated dataset. All 88 same-name pairs are connected — at distance 2 or 4. The median distance between two datasets picked at random is also 4. Twins are indistinguishable from strangers.

Names tell you almost as little. Three of fifteen same-name groups are not the same thing: four Tableau datasets called Custom SQL Query share 0 % of their columns; promotions shares 9 %.

And the governance does not travel. Eleven tables exist identically on dbt, Snowflake, Postgres and S3 — 100 % column overlap, customers has the same 22 columns in all four. dbt/customers has three owners, a domain and a description. Its three twins have none, none and none. Twelve identical columns are annotated on one side and bare on the other; customers.customer_id carries the PII_Data tag on Postgres alone.

Ask that catalog where the personal data is, and it answers with two datasets, with the confidence of something that has looked.

Related MCP server: AWS Data Discovery & PII Detection Agent

How it decides

Structure suggests. A model decides. Nothing is invented.

python scripts/mesurer-le-juge.py puts the judge against 16 pairs drawn from the real catalog, chosen to be hard in both directions: replicas that differ in case and column count, reference tables of identical shape, aggregates computed from a table rather than copied from it, and the four homonymous Tableau queries that share only measure names. 16 out of 16. Ten runs of the same pair give the same verdict ten times.

Two rules that shape everything

Fadlie copies; it never writes. No value put into the catalog is produced by a model. Each one comes from a dataset that already carried it, and every proposal names its source — Ecart refuses to exist without one. A description generated by a machine is indistinguishable from a description written by the team that knows the data; six months later nobody can tell which is which.

A disagreement is not a gap. snowflake/ORDER_DETAILS sits in Ecommerce Operations while its four twins sit in Data Platform Team. Fadlie reports it and does not choose. Someone decided, or someone erred, and neither is an agent's call.

A failure must not look like good news

The judge raises rather than returns "different". If it could silently fail, Fadlie would report no duplicates found — a catalog in good order. Nobody audits good news. So the server probes the model before its first verdict, and both real failure modes are verified: a bare model id (Frankfurt requires the regional inference profile) and credentials without Bedrock access.

The tools

No tool takes a user, a token, or an identity — authorisation comes from the Authorization header, and a check enforces it because nothing in the language does.

Running it

cp .env.example .env && $EDITOR .env
set -a && . ./.env && set +a

python -m pytest tests/          # 70 tests, no network, no bill
python -m fadlie check           # reads config, connects to nothing
python -m fadlie report          # the full analysis, printed
python -m fadlie serve           # the MCP server, locally

python scripts/mesurer-jumeaux.py    # every number in this README
python scripts/mesurer-le-juge.py    # the judge against 16 hard pairs
python scripts/verifier-mcp.py       # 24 checks through a real MCP client

What it runs on

• DataHub Core v1.7.0 on EC2, loaded with the contest's showcase-ecommerce datapack — 67 datasets across Snowflake, dbt, Postgres, S3, Tableau, PowerBI and Looker.

• Amazon Bedrock, Amazon Nova Micro as the judge, in eu-central-1.

• AWS App Runner for the MCP server, over HTTPS.

Licence

Apache License 2.0 — see LICENSE.