dagsmith

Agentic scaffolding for data engineering. Turn a plain-English description of a new data source into a complete, tested, production-shaped pipeline: dbt models (staging → marts), an Airflow DAG, a Great Expectations suite, and freshness monitors — generated, executed, and self-corrected by an LLM agent that reads your warehouse's live metadata over MCP.

┌───────────────────────────────────────────────────────────────────────┐
│                                                                       │
│   analyst writes:                                                     │
│   "Ingest daily registry events from Postgres. Grain = one event      │
│    per registry_id + item_id. Join to customers for LTV rollups."     │
│                                                                       │
│                                    │                                  │
│                                    ▼                                  │
│                          ┌─────────────────────┐                      │
│                          │  dagsmith agent     │                      │
│                          │  (Claude Sonnet 5)  │                      │
│                          └──┬──────────────┬───┘                      │
│                             │              │                          │
│           MCP tools ───────▶│              │──── Jinja templates      │
│           (Snowflake meta)  │              │      + verifier loop     │
│                             ▼              ▼                          │
│           ┌─────────────────┐    ┌─────────────────┐                  │
│           │ dbt models      │    │ Airflow DAG     │                  │
│           │  stg_*.sql      │    │  registry_ev.py │                  │
│           │  fct_*.sql      │    │                 │                  │
│           │  schema.yml     │    │ GE checkpoint   │                  │
│           └─────────────────┘    └─────────────────┘                  │
│                             │              │                          │
│                             └──────┬───────┘                          │
│                                    ▼                                  │
│                          ┌─────────────────────┐                      │
│                          │  verifier           │                      │
│                          │  (real Snowflake +  │                      │
│                          │   sandbox dbt run)  │                      │
│                          └──────┬──────────────┘                      │
│                                 │                                     │
│                          pass ──┴── fail → loop back to agent         │
│                                                                       │
└───────────────────────────────────────────────────────────────────────┘

Why this exists

Every data engineer builds the same eight files for every new source: a staging model, a mart, a schema.yml, a snapshot config, a DAG, a GE suite, a monitor, a docs entry. Ninety percent of that work is mechanical — but it's not homogeneous enough to templatize with pure code. That gap is what an agent covers well.

dagsmith treats pipeline generation as a compile step, not a checklist. You describe the intent; the agent generates the artifacts; a verifier runs them against a sandbox schema in real Snowflake; failures loop back into the agent's context until they pass. When you merge, you get real tested code — not a demo.

Related MCP server: databricks-mcp

Design in one paragraph

The agent is a Claude Sonnet 5 tool-use loop with three tool categories: discovery (MCP calls into Snowflake's INFORMATION_SCHEMA and ACCOUNT_USAGE so the agent can inspect what already exists), generation (write files from Jinja templates), and verification (run dbt build --target ci and Great Expectations checkpoints against an isolated sandbox schema). The verifier's stdout+stderr feed back into the agent's context, and it iterates until green or hits a retry cap.

What's in the box

Quickstart

git clone https://github.com/<you>/dagsmith && cd dagsmith
pip install -e ".[dev]"

# 1. Point at your Snowflake sandbox (see docs/setup.md)
export SNOWFLAKE_ACCOUNT=abc12345.us-east-1.aws
export SNOWFLAKE_USER=you@example.com
export SNOWFLAKE_ROLE=DATA_ENG
export SNOWFLAKE_WAREHOUSE=DEV_WH
export ANTHROPIC_API_KEY=sk-ant-...

# 2. Generate a pipeline from a spec
dagsmith generate examples/babylist_registry_events/input/spec.md \
    --out examples/babylist_registry_events/generated \
    --sandbox-schema DAGSMITH_CI_TAYLOR

# 3. Or run the MCP server for use in Claude Desktop / Cursor / Zed
dagsmith mcp

MCP server (usable on its own)

Even if you don't run the agent, the MCP server is useful. Point any MCP client at it and Claude/Cursor/Zed gains 12 read-only tools:

• list_databases, list_schemas, list_tables

• describe_table (columns, types, cluster keys, comments)

• sample_rows (limit-capped preview)

• query_history (top N recent queries by cost / duration)

• access_history (who reads what)

• column_lineage (upstream columns for a target column via ACCESS_HISTORY)

• dead_tables (tables not read in N days)

• storage_by_schema and credits_by_warehouse

• check_freshness (last successful write per table)

MCP config for Claude Desktop:

{
  "mcpServers": {
    "dagsmith": {
      "command": "dagsmith",
      "args": ["mcp"],
      "env": {
        "SNOWFLAKE_ACCOUNT": "abc12345.us-east-1.aws",
        "SNOWFLAKE_USER":    "you@example.com",
        "SNOWFLAKE_ROLE":    "READONLY_MC"
      }
    }
  }
}

What the agent will and won't do

Will: generate staging + mart dbt models with tests, an Airflow DAG that uses Cosmos, a Great Expectations checkpoint with reasonable defaults (uniqueness on the grain, row-count band, not-null on FKs), a freshness monitor query, and a docs.md entry.

Won't: run destructive DDL against production, guess at business logic it can't see, or generate code without verifying it compiles. Every generated file is dbt parse-clean before the agent claims done.

Roadmap

• dagsmith audit — point at an existing dbt project and produce a report on coverage gaps (missing tests, stale models, over-warehouse'd queries)

• dagsmith migrate — port a legacy Fivetran → SQL pipeline into the dbt+Airflow layout the agent generates

• Evaluation harness — golden-output tests over 20+ source specs so regressions are catchable

• Backends beyond Snowflake — BigQuery + Redshift adapters share ~80% of the templates

Why the JD's language fits

This project directly maps to the "harnesses and agentic scaffolding that generate, test, and maintain them" phrasing in the Babylist Senior Data Engineer role:

• Meta-layer, not just pipelines — the agent is a pipeline factory, not a pipeline.

• MCP servers real users depend on — the metadata server is usable standalone, not just as an agent input.

• Agentic workflows with tool integrations — 12+ tools, real verifier feedback loop.

• Airflow + dbt + Snowflake + Python — the whole stack the JD names, wired end-to-end.

• Data monitoring across multi-system journeys — freshness + GE suites are part of the generated bundle.

License

MIT. See LICENSE.