Loom

A declarative ontology framework over Apache Iceberg, wired to LLM agents via MCP.

Loom is an open take on the idea behind Palantir Foundry's Ontology: a semantic layer that turns raw Iceberg tables into a navigable graph of objects, links, and actions — and exposes that graph to agents as typed, governed tools instead of raw SQL.

The premise: Iceberg is now a commodity (Foundry, Snowflake, Databricks, AWS, GCP all speak it). The value isn't storage — it's the layers on top. Loom is those layers.

The core idea

One declarative YAML spec is the single source of truth, and it compiles to four surfaces at once — you never hand-write a migration, a SQL join, or an MCP tool:

                    ┌──────────────────────────┐
   ontology/*.yaml ─┤  Ontology Model (typed)   ├─┬──▶ Iceberg schema (DDL / migrations)
   (source of truth)└──────────────────────────┘ ├──▶ Query resolver (ops → engine SQL)
                                                   ├──▶ Action runtime (validate + writeback)
                                                   └──▶ MCP tools (auto-registered)

Think dbt/Prisma, but the generated "client" is an MCP tool surface for agents.

Related MCP server: DataLakeHouseMCP

Example

# ontology/customer.yaml
objectType:
  apiName: Customer
  primaryKey: customerId
  title: name
  backing: { catalog: rest_main, table: crm.customers }
  properties:
    - { name: customerId, type: string, column: id, unique: true }
    - { name: name,       type: string, column: full_name }
    - { name: tier,       type: enum, values: [bronze, silver, gold], column: tier }
  searchable: [name, tier]

# ontology/actions/upgrade-tier.yaml
action:
  apiName: upgradeTier
  description: Raise a customer to a higher membership tier   # becomes the MCP tool description
  targetObjectType: Customer
  operation: modify
  parameters:
    - { name: customer, type: objectRef, objectType: Customer }
    - { name: newTier,  type: enum, values: [silver, gold] }
  validation:
    - { rule: "newTier != object.tier", message: New tier must differ from current tier }
  effects:
    - modifyObject: { key: "{{ customer }}", set: { tier: "{{ newTier }}" } }

At serve time this exposes get_customer, search_customer, list_customer, and run_upgrade_tier to any MCP client — with input schemas derived from the property/parameter types, and row/column governance enforced below the tool layer.

Today the reads, traverse and the run_ tools are all live over MCP. Writes are off by default: mcp.writes: true in loom.yaml is what turns a declared action into a tool, because declaring one and serving it to every client that connects are different decisions. Governance withholds both halves now: a governance.policies entry withholds a property from every caller and filters the rows every caller sees — loom query included, because the mask is applied to the projection and the predicate to the table, below both of them. See Status.

Architecture (5 layers)

5. Agent / MCP layer       ← LLMs call the ontology as typed verbs (never raw SQL)
4. Action & function layer ← single-object writeback via the Iceberg catalog, optimistic concurrency, audited
3. SEMANTIC / ONTOLOGY layer ← object types, links, mappings   ★ the moat
------------------------------------------------------------
2. Query / compute engine  ← engine-agnostic IR → DuckDB / Trino / Spark adapters
1. Storage                 ← Apache Iceberg tables + REST catalog

Two decisions shape the framework:

• Reads flow through an engine-agnostic IR (GetByKey / Search / Traverse / Project) that per-engine adapters lower to dialect SQL.

• Writes are single-object and bypass the compute engine entirely — an equality-delete + append committed as one atomic Iceberg transaction — which keeps the write path uniform across engines. They also go through their own ports: one for a table's shape, one for its rows, and one for Loom's own record, so a migration cannot touch data and an action cannot touch DDL — nor name a table through the port it records itself with.

Status

Early, but the read path works end to end: a YAML spec over real Iceberg tables, served to an MCP client as typed tools. The migration path is complete — loom plan classifies what a spec change would do to the physical tables, loom apply executes it, and loom rollback puts an earlier spec back. The action runtime writes: one row, as one Iceberg commit, with the snapshot its read saw asserted inside that commit, so a competing write refuses the run rather than silently losing to it — and every run that named a row is recorded in _loom_meta.edits, including the ones that refused. loom serve speaks stdio and HTTP, and serves writes only where a deployment asked for them. Governance withholds columns and rows below the tool layer, so loom query is filtered exactly as an agent is, and mcp.auth gives a policy an attested caller to name. Filters are fully typed. A bulk load lands through a declared entry and the lake records it. Semantic search is the milestone in progress.

Each of those sentences was a decision with an argument behind it, and the arguments are the point: docs/status.md is that story milestone by milestone.

Quickstart

python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev,iceberg,duckdb,mcp]"

pytest                              # 1213 tests
loom validate tests/fixtures/valid  # → ok — 2 object type(s), 1 link type(s), 3 action(s)
                                    #   · no loom.yaml found — spec checked, deployment not

embed is not in that list, and this page gets as far as the ranked plane. Add it — .[dev,iceberg,duckdb,mcp,embed] — or leave it out and everything below still runs except loom embed and match_support_ticket, both of which name the missing extra rather than failing on an import. (.[dev,all] is the same list plus auth, which attested identity needs and this page does not.)

Then run the whole stack against a real Iceberg warehouse — examples/retail ships the worked example plus a seed script that builds one locally, with no services to start:

python examples/retail/seed.py                        # build the warehouse
loom query Customer examples/retail/ontology --key c1 # → one row, through DuckDB
loom run upgradeTier examples/retail/ontology \
  --param customer=c3 --param newTier=gold            # → one row rewritten, one row recorded
loom serve examples/retail/ontology                   # → 14 MCP tools over stdio

The validator accumulates every problem and reports them in one pass with source locations:

$ loom validate ./broken-ontology
3 problems in ontology spec:
  - customer.yaml · objectType 'Customer': unexpected key 'titel'
        hint: did you mean 'title'?
  - actions/up.yaml · action 'upgradeTier': expression references unknown parameter 'ghost'
  - actions/up.yaml · action 'upgradeTier': effect set 'bogus' is not a property of 'Customer'

Documentation

License

MIT — see LICENSE.