Duckle MCP Server

Truly local

The Qwen model runs as a llama-server subprocess on 127.0.0.1. No API keys. No network calls. Disconnect your wifi and it keeps working.

Streamed responses

Tokens arrive as they're generated, with a blinking caret in the bubble. No "wait 20 seconds for the spinner to vanish" UX.

One-click insert

When Duckie produces a JSON pipeline, an Insert into canvas button appears. The graph populates with positioned nodes, wired edges, and the props the model chose.

Bring-your-own-model option

The chat plumbing is the same OpenAI-compatible HTTP interface used by xf.ai.llm / xf.ai.embed connectors. Point baseUrl at Ollama, llama.cpp, Cohere, OpenAI, Voyage - anything that speaks the OpenAI shape.

Sandboxed

The model has no fs / net / tool access. It can only emit text - your pipeline JSON.

Visual, never opaque

The canvas compiles to SQL you can read, and every node has a live preview tab. No black box.

Local-first AI

An assistant that runs on your laptop without an API key. Your prompts, your data, your machine.

Single-file binary, no bundled DB

~65 MB app (it embeds the headless runner + MCP server). DuckDB downloads on first launch with a guided step. AI engine is opt-in.

Native speed

Execution runs through DuckDB: vectorized, columnar, local. A clean-and-export job that crawls in a spreadsheet finishes in milliseconds.

Git-friendly by design

Pipelines, connections, contexts, and routines persist as plain files in a folder you pick. Diff them, branch them, review them.

290+ connectors that work

Files, databases, warehouses, lakehouses, object stores, SaaS APIs, NoSQL, streaming brokers, vector DBs, FTP, IMAP, SMTP. Each is covered by tests.

Honest about scope

Single-machine and embedded by design. Built to make local and small-team data work fast, not to replace a distributed warehouse.

60 UI languages

Topbar, palette, chat assistant, properties panel, and common dialogs ship localized. English, Spanish, Chinese (Simplified + Traditional), Hindi, Arabic, Portuguese (Brazil), Bengali, Russian, Japanese, Punjabi, German, Korean, French, Vietnamese, Telugu, Marathi, Turkish, Tamil, Urdu, Persian, Polish, Italian, Ukrainian, Indonesian, Thai, Dutch, Hebrew, Swedish, Greek, Czech, Hungarian, Romanian, Filipino, Malay, Norwegian, Danish, Finnish, Catalan, Bulgarian, Slovak, Croatian, Serbian, Slovenian, Lithuanian, Latvian, Estonian, Khmer, Burmese, Sinhala, Nepali, Swahili, Afrikaans, Welsh, Irish, Icelandic, Albanian, Azerbaijani, Mongolian, Kazakh. RTL (Arabic, Hebrew, Persian, Urdu) supported. Switch languages from the topbar globe.

Open source

Dual-licensed MIT OR Apache-2.0. Yours to use, fork, and extend.

Group

Connectors

Status

Files

CSV, TSV, Parquet, JSON, JSONL / NDJSON, Excel (.xlsx), YAML, TOML, Fixed-width (mainframe / banking positional dumps), XML (slash-separated rowPath), Apache Avro (.avro / .ocf, pure-Rust)

Available

Geospatial files

GeoJSON, Shapefile, GeoPackage, KML, GPX, GML via the spatial extension

Available (lazy-loaded)

Lakehouse table formats

Apache Iceberg, Delta Lake, DuckLake

Available

Embedded databases

SQLite (read tables), DuckDB (read tables or run a query)

Available

Network relational DBs

PostgreSQL, MySQL, MariaDB, CockroachDB

Available (live CI for PG + MySQL)

Network relational DBs

SQL Server (TDS), Oracle (Instant Client at runtime), ClickHouse (HTTP API)

Available

Network relational DBs

IBM DB2, generic JDBC

Planned

Object storage

Amazon S3, Google Cloud Storage, Azure Blob, HTTP(S), MinIO, Cloudflare R2, Backblaze B2

Available (live CI for MinIO)

Cloud warehouses

MotherDuck, Snowflake (SQL API + PAT/JWT), BigQuery, Redshift (postgres ATTACH), Databricks SQL (Statement Execution + chunk follow), Azure Synapse (TDS), Teradata (ODBC, Windows / Linux), DuckDB Quack (May 2026 remote protocol - HTTP on :9494, SECRET-based token auth)

Available

Streaming

Apache Kafka / Redpanda (pure-Rust rskafka), NATS JetStream, GCP Pub/Sub (REST + auto-ack), RabbitMQ (lapin AMQP), AWS Kinesis (HTTP + SigV4 - no AWS SDK)

Available

Streaming

Pulsar, Event Hubs, multi-shard Kinesis

Planned

APIs and SaaS (REST)

Salesforce, HubSpot, Pipedrive, Zendesk, Intercom, Stripe, QuickBooks, Xero, Shopify, Notion, Airtable, Asana, Trello, ClickUp, Monday.com, GitHub, GitLab, Linear, Jira, Slack, Discord, Telegram, Twilio, Mailchimp, SendGrid, Segment - thin pre-configured wrappers over src.rest / src.graphql. src.rest takes a configurable API-key auth header name and offset pagination that stops on a body total_count

Available

APIs (protocols)

OData v4 (follows @odata.nextLink), SOAP / generic XML APIs (XML response parsing with namespace local-name match)

Available

NoSQL and search

MongoDB (official driver), Cassandra / ScyllaDB (CQL), Elasticsearch / OpenSearch (from+size + search_after), Redis (SCAN + GET), CouchDB (_all_docs), DynamoDB (HTTP + SigV4 - no AWS SDK; auto-unwraps typed attributes)

Available

Vector / AI databases

pgvector (postgres ATTACH), Qdrant (/points/scroll), Weaviate (/v1/objects), Milvus (/v1/vector/query)

Available

Vector / AI databases

Pinecone (no list-all-vectors API), Chroma, LanceDB

Preview

File transfer

FTP / FTPS (pure-Rust suppaftp) and SFTP (SSH, pure-Rust russh + russh-sftp on the ring backend; password or private-key auth, optional host-fingerprint pin) - one File Transfer component, pick the protocol. Glob filter, base64 content per file

Available

Mailbox

IMAP (rustls TLS, mail-parser) - basic auth today, OAuth (gmail / o365) on the roadmap

Available

Webhook listener

Binds 127.0.0.1:port, collects N inbound HTTP requests with a timeout, parses JSON-object / JSON-array bodies into rows

Available

Desktop

System clipboard (pure-Rust arboard, auto-detects JSON-array shape)

Available

Repos

Git (commit log or file tree from a local working copy; shells out to system git CLI)

Available

Group

Operations

Fields

Map (visual mapper: joins a main input to up to 3 lookup inputs with inner / left joins and per-output expressions + filter), Project / Select, Cast, Rename, Add / Drop / Reorder Column, Coalesce, UUID v4

Rows

Filter (visual or raw SQL, with reject port), Distinct, Sample, Top N / Limit, Sort, Skip, Top N per Group, Forward Fill, Backward Fill, Constant Fill

Aggregate

Group By, Rollup, Cube, Count, Window Aggregate, Cumulative, Approx Quantile (t-digest), Approx Count Distinct (HyperLogLog)

Join

Inner, Left, Right, Full Outer, Cross, Lookup, Semi, Anti, Spatial Join

Set operations

Union, Union All, Intersect, Except / Minus

Window

Row Number, Rank, Dense Rank, Lead, Lag, First Value, Last Value, NTile

Strings

Regex Replace, Regex Extract, Regex Match, Split, Concat, Trim, Case Change, Length, Substring, Format, Hash (md5 / sha1 / sha256), IP Parse, URL Parse, Text Similarity (Levenshtein / Jaro-Winkler / Jaccard), Base64, Pad, Text Match

Date / Time

Parse, Format, Extract Part, Date Diff / Add, Truncate, Timezone Convert, Time Bin, Current Timestamp, Epoch Convert

Numeric

Round, Modulo, Absolute, Logarithm, Power, Square Root, Bucketize, Z-Score, Clamp, Sign

JSON / nested

Parse, Stringify, Flatten, JSONPath Extract, Merge Objects, Array Aggregate

Array

Explode / Unnest, Collect List, Element At, Contains, Distinct, Length, Zip Arrays to Table (headings + row-arrays -> one column per heading)

Pivot / shape

Pivot, Unpivot, Denormalize, Normalize, Transpose

CDC / SCD

Incremental Load (watermark column; saves the high-water mark to workspace state and advances only on a fully successful run), Diff Detect, SCD Type 1, SCD Type 2 (valid_from / valid_to / is_current), Merge / Upsert (universal across embedded, network, warehouse and Mongo sinks, with optional delete propagation driven by a CDC change-type column), DuckLake CDC change-feed reader, Row Hash (md5 / sha1 / sha256 fingerprint), Audit Stamp (_loaded_at / _loaded_date / _source / _batch_id)

AI / Search

Vector Similarity Search (cosine / L2 / inner product over FLOAT[N] via vss), Full-Text Search (BM25 via fts), Embeddings (OpenAI-compatible /v1/embeddings), LLM Transform (per-row chat completion with {column} templates), Classify (LLM-backed, normalizes to UNKNOWN), Text Chunker (RAG-ready, pure local), PII Redact (regex - emails / phones / SSNs / cards), Semantic Dedupe (cosine over precomputed embeddings)

Geospatial

Spatial Distance (ST_Distance), Spatial Buffer (ST_Buffer), Spatial Intersects (ST_Intersects)

Debug

Log Rows, Assert (hard-fail on SQL predicate violation)

Component

Behavior

Not-Null Check

Pass rows with no nulls in the chosen columns

Range Check

Pass rows inside a numeric range (inclusive or exclusive)

Regex Match

Pass rows whose column fully matches a pattern

Uniqueness Check

Pass the first row per key; route duplicates to reject

Schema Validate

Reject rows where any expected column is null

Column Profile

Per-column stats (count, null %, distinct, min / max, quartiles) via SUMMARIZE

Describe

Column names + types of the input

Histogram

Value frequencies for one column, most-frequent first

Standardize

Trim + case-normalize + collapse inner whitespace, in place

Fuzzy Deduplicate

Keep the first row per near-duplicate cluster

Record Match

Self-join: emit pairs of rows above a similarity threshold

Address Cleanse

Address parsing / normalization (planned - needs external lib)

Capability

What it does

Inline SQL

Write a SELECT; the upstream node is exposed as input, result runs as a real materialized stage. A raw SQL mode runs verbatim SQL (a leading WITH / multiple CTEs / UNIONs) with no input-CTE wrapper

SQL Template

Parameterized SQL with ${context.var} substitution

SQL Routines

Reusable, named SQL saved in the workspace

dbt

Run a dbt project (or one inline model) as a node, against the pipeline's DuckDB. Wire several upstream sources in and the project reads them all via dbt sources, so one project models across Postgres, MySQL, files, and lakes at once. Powered by the dbt Fusion engine, fetched free at first launch (Apache dbt-core fallback); no Python setup.

Shell

Run any shell command; emits {stdout, stderr, exit_code, duration_ms}. Platform-aware default shell. Optional timeoutMs kills the child.

WebAssembly UDF

Per-row WASM transform via pure-Rust wasmi. Sandboxed (no fs / net / env). Works with any WASM toolchain (Rust, AssemblyScript, C, TinyGo).

JavaScript UDF

Per-row JS transform via pure-Rust boa interpreter. Sandboxed. Define a transform(row) function.

Python / Rust UDFs

Embedded-language stages

Group

Connectors

Status

Files

CSV, TSV, Parquet (ZSTD), JSON, JSONL / NDJSON, Excel (.xlsx), YAML, TOML, XML (configurable wrappers), Avro (schema inferred from first row). Parquet + CSV support Hive-partitioned writes

Available

Geospatial files

GeoJSON, GeoPackage, Shapefile, KML, GPX via GDAL

Available (lazy-loaded)

Lakehouse

Apache Iceberg (full table layout), DuckLake - modes: overwrite, append, truncate, upsert (set-based delete-by-key + re-insert), merge (partial-column MERGE INTO that preserves columns the source omits) with optional CDC delete propagation

Available

Embedded databases

SQLite, DuckDB - modes: overwrite, append, upsert (set-based delete-by-key + re-insert, no PK required), merge (partial-column MERGE INTO that preserves columns the source omits) with optional CDC delete propagation

Available

Network relational DBs

PostgreSQL, MySQL, MariaDB, CockroachDB - modes: overwrite, append, truncate, upsert (ON CONFLICT / ON DUPLICATE KEY) with optional CDC delete propagation

Available (live CI for PG + MySQL)

Network relational DBs

SQL Server / Azure Synapse (TDS, multi-row VALUES batched; auto-creates the table if absent; upsert via MERGE), Oracle (Instant Client; INSERT ALL, batched per statement; auto-creates the table if absent; upsert via MERGE), ClickHouse (HTTP JSONEachRow; upsert by pointing at a ReplacingMergeTree target table) - every MERGE sink supports CDC delete propagation (a delete-flag column removes matched rows)

Available (SQL Server + Oracle + MySQL upsert and delete propagation verified live in Docker)

Network relational DBs

IBM DB2, generic JDBC

Planned

Object storage

S3, GCS, Azure Blob via DuckDB httpfs (MinIO / R2 / B2 via endpoint)

Available

Cloud warehouses

MotherDuck, Snowflake (PAT or JWT RS256; upsert + delete propagation via MERGE), BigQuery, Redshift, Databricks SQL (upsert + delete propagation via MERGE), Azure Synapse, Teradata (ODBC), DuckDB Quack (concurrent writers to remote DuckDB via the May 2026 protocol)

Available (Snowflake MERGE verified live against the SQL-API emulator)

HTTP APIs

REST (POST/PUT/PATCH batched JSON-array; configurable API-key auth header name), Webhook (one POST per row), GraphQL mutations

Available

Email (SMTP)

Per-row SMTP send via pure-Rust lettre + rustls. Plain text v1; HTML + attachments follow.

Available

NoSQL

MongoDB (insert_many batched; upsert via replace_one on a key, plus delete propagation via delete_one), Cassandra / ScyllaDB (CQL), Elasticsearch / OpenSearch (_bulk NDJSON), Redis (pipelined SET)

Available

NoSQL

DynamoDB

Planned

Streaming

Kafka / Redpanda (rskafka), NATS JetStream, GCP Pub/Sub (REST + OAuth2), RabbitMQ (lapin)

Available

Streaming

Pulsar, Kinesis

Planned

Vector / AI databases

pgvector, Pinecone (/vectors/upsert), Qdrant (/points PUT), Weaviate (/v1/batch/objects), Milvus (/v1/vector/insert)

Available

Vector / AI databases

Chroma, LanceDB

Preview (need vendor SDK)

Component

What it does

Replicate / Tee

Send the same data to multiple downstream outputs

Merge Streams

Concatenate multiple input streams (UNION ALL)

Switch / Conditional Split

Route rows to case_1..N outputs by boolean (first match wins); default for unmatched

Wait / Delay

Sleep N ms / s / min / h before passing rows through

Throttle

Inter-stage delay derived from a rows-per-second target

Checkpoint

Pass rows through and also write a parquet snapshot to a path

Dead Letter Queue

Terminal sink for rejected rows (JSON / CSV / Parquet)

Run Pipeline

Inline-execute another pipeline file (ctl.runpipeline)

Run Job

Call a child pipeline (picked from the workspace) passing parent context variables; chain several to build a Master Job (ctl.runjob)

Parallelize

Run the downstream branches wired to its outputs concurrently; branches are unlimited (ctl.parallelize)

Iterate

Run a sub-pipeline N times with ${ITER_INDEX} substitution

For Each

Run a sub-pipeline once per input row with ${ITER_ITEM_<FIELD>} substitution

Try / Catch

Install a fallback sub-pipeline if the wrapped stage fails

Retry

Per-stage retry policy (configure on Advanced tab)

Log Message

Emit an info log line ({rows} = upstream count), pass rows through (ctl.log)

Warn

Emit a warning log line, pass rows through (ctl.warn)

Die / Fail

Stop the run with a message: always, only when the input has rows, or only when empty (ctl.die)

Schedule

Cron / interval / file-watch triggers via the orchestration crate

Field

What it does

Retry attempts

Total tries on failure (1 = no retry). Sleeps backoff * attempt ms between attempts.

Retry backoff (ms)

Inter-attempt sleep, linearly scaled by attempt index.

Memory limit (MB)

PRAGMA memory_limit applied to this stage only.

Log row count

Print the post-stage rowcount to the run output.

Capability

What it does

Run feedback

Streaming run events light nodes up stage by stage, with per-node row counts, real mid-query cancel, and run history.

Error traceback

A failed stage reports the exact compiled SQL plus the underlying DuckDB message, in both the Run view and the NDJSON run log, so any component's failure is debuggable.

Column lineage

A top-bar Lineage button shows, per node, each output column traced back to the source column(s) it derives from.

Dives + dashboards

Local-first, live-querying, shareable data views, stitched into multi-chart dashboards. Generate a chart from a plain-language question, export a dive to a self-contained HTML file, open standalone /dive/<id> and /dash/<id> share pages, and find everything in the top-bar Dives gallery.

Run logs

Every run writes component-level NDJSON to <workspace>/logs/<pipeline name>/runtime.log (start/finish per stage, row counts, durations, ctl.log / ctl.warn / ctl.die messages). Tail it straight into Splunk or Dynatrace.

Schedules

Cron, fixed-interval, and file-watch triggers, driven by an in-process scheduler.

Context variables

Per-environment variables; bind any field to one via a Manual / Context dropdown, or reference ${var} inline. Resolved at run time.

Workspace-relative paths

Built-in ${workspace} (alias ${projectroot}) resolves to the active workspace root, so source / sink paths can be written relative to it and a workspace folder stays portable when copied or moved. No context needed; works in the canvas, schema autodetect, and headless runs.

Run-time path placeholders

Built-in ${date}, ${time}, ${datetime}, ${timestamp}, and ${now} (UTC) stamp the current run time into any path. They resolve fresh on every run (canvas, schedule, headless runner, built bundle), and a sink's parent folder is created automatically, so a path like ${workspace}/exports/${date}/orders.parquet lands in a new dated folder each day. No context needed.

Cloud credentials

Saved S3 / GCS / Azure connections become DuckDB SECRETs; cloud reads / writes go through httpfs. S3-compatible endpoints (MinIO / R2 / B2) supported via ENDPOINT + URL_STYLE.

Workspace

Pipelines, connections, contexts, documents, and routines persist as plain JSON and Markdown files in a folder you choose.

Engine

Role

Status

DuckDB

Default execution engine: analytics, file formats, cloud reads, SQL pushdown. Tracking v1.5.3 (latest stable). A lock-free single-SELECT read (Engine::query) powers dives.

Working

Duckie AI Assistant

Local chat assistant via llama.cpp + Qwen 2.5 Coder 1.5B GGUF. Downloads ~1.1 GB; runs entirely offline once installed. Managed as a llama-server subprocess exposing an OpenAI-compatible API on 127.0.0.1.

Installable

SlothDB

Alternate embedded analytical engine (SouravRoy-ETL/slothdb), installed the same way and selectable per pipeline.

Installable

Native

In-process Rust streaming / incremental engine.

Planned

OS

Asset

How to run

Windows

Duckle-windows-x64.exe

Double-click. Unsigned binary - Windows SmartScreen will warn the first time; click "More info" -> "Run anyway".

macOS (Apple Silicon)

Duckle-macos-arm64

chmod +x Duckle-macos-arm64 && ./Duckle-macos-arm64. Right-click -> Open the first time to bypass Gatekeeper.

Linux (x86_64)

Duckle-linux-x64

chmod +x Duckle-linux-x64 && ./Duckle-linux-x64. Requires WebKitGTK 4.1 (libwebkit2gtk-4.1-0 on Debian / Ubuntu).

Engine

Size

Required?

What it powers

DuckDB CLI

~30 MB + extensions

Yes - cannot run pipelines without it

Every source / transform / sink that runs as SQL

Duckie AI Assistant

~1.1 GB (llama-server + Qwen 2.5 Coder 1.5B GGUF)

Optional

The chat sidebar that generates pipelines from natural language

Step

What you do

Where to look

1. Sources

Drag a source, point it at a file / DB / cloud URL / SaaS endpoint. Click Autodetect schema to read columns + a sample.

Sources reference

2. Transforms

Wire transforms to source output ports. Configure in the Properties panel. Preview tab shows live rows; Plan tab shows generated SQL.

Transforms reference

3. Data quality

Drop in a validator (Not-Null, Range, Regex, Uniqueness). Passing rows continue on the main port; failures route to the reject port.

Data quality reference

4. Sinks

Finish with a sink (file, DB, cloud, vector DB, message bus, email). Set write mode (overwrite, append, truncate, upsert).

Sinks reference

5. Run

Press Run to execute on DuckDB. Nodes light up stage by stage; Output + Console show row counts, timing, errors. Stop button kills mid-run.

Run feedback

6. Ask Duckie

For anything you can describe in English, the AI assistant can sketch a pipeline. Iterate by editing the graph or asking follow-ups.

Meet Duckie

7. Reuse

Save Connections, Context variables, and SQL Routines in the workspace; reference ${context.var} in any field. Everything persists as plain files.

Workspace and Git flow

8. Schedule

Attach a cron, interval, or file-watch trigger to run a pipeline automatically.

Schedules and triggers

Feature

What it does

Status snapshot

Current branch, ahead/behind counts, list of modified / staged / untracked / conflicted files

Stage all + commit

One-click git add -A && git commit -m "..." with your message

Push / Pull

git push and git pull --ff-only against origin. The button stays disabled when there's nothing to push

Branch list, switch, create

Lists local branches; click to switch; create new branches inline

Remote URL config

Add or change origin URL from inside the panel - auto-detects GitHub vs GitLab from the host

PAT-prompt fallback

First tries git push using your system credential helper (GitHub CLI, osxkeychain, manager-core). On a 401, prompts for a Personal Access Token, saves it AES-encrypted in <workspace>/.duckle/secrets/git.json (auto-gitignored), retries with the token injected into the HTTPS URL

CI build badge in topbar

Polls GitHub Actions or GitLab CI every 30 s for the latest pipeline on your current branch. Shows green / red / yellow / gray. Click to open the build in your browser

Provider

CI source

API

github.com

GitHub Actions

GET /repos/{owner}/{repo}/actions/runs

gitlab.com or self-hosted GitLab

GitLab CI

GET /api/v4/projects/{id}/pipelines

Other / bitbucket

(no CI badge for now)

-

Trigger type

Config

Example

Cron

Standard 5-field cron expression with optional timezone

0 2 * * * (every day at 2 AM)

Interval

every N {seconds, minutes, hours, days}

every 15 minutes

File watch

Watch a directory for new/changed files matching a glob

/inbox/*.csv

Manual

Run-on-demand only (the default)

-

Option

What it does

Target OS

Pick Windows, Linux, or macOS in the build dialog. The native OS always builds; a Linux server file can be cross-built from any host (the Linux engine is bundled for you), while a macOS file can only be produced on a Mac. Appending the payload makes the file unsigned, so do not codesign / Authenticode-sign it.

Context

Pick a context at build time; its non-secret variables are baked into the pipeline.

Secrets: Environment

Each secret becomes a ${ENV:KEY} placeholder, so nothing sensitive is written into the file. The runner resolves real environment variables first, then a secrets.env (KEY=VALUE lines) placed next to the file.

Secrets: Passphrase

Secrets are encrypted inside the file with AES-256-GCM, decrypted at run time from the DUCKLE_BUNDLE_PASSPHRASE environment variable.

Type

Stored fields

Used by

PostgreSQL / MySQL / etc.

host, port, user, password, database, ssl mode

src.postgres, snk.postgres, ...

Snowflake

account, user, role, warehouse, PAT or JWT private key

src.snowflake, snk.snowflake

S3 / GCS / Azure

access key, secret, region (or service-account JSON)

All cloud sources/sinks via httpfs

MotherDuck / Databricks / BigQuery

token, workspace URL

Respective sources/sinks

Generic REST / SaaS

base URL, auth scheme (Bearer / API key / Basic, with a configurable API-key header name), token, custom headers

All REST aliases

Setting

Where

Effect

Theme

Topbar sun/moon toggle

Light / dark, persisted to localStorage

Workspace

Topbar workspace pill -> Switch

Change the folder Duckle reads/writes to

Active engine

Topbar engine selector

DuckDB (default) or SlothDB - per-pipeline

Active context

Topbar context dropdown

Switches which context variables resolve at run time

AI Assistant baseURL

xf.ai.llm / xf.ai.embed / xf.ai.classify props

Point at any OpenAI-compatible endpoint (default: Duckie's local llama-server)

Per-stage retry

Properties panel -> Advanced tab

Total attempts + linear-scaled backoff per stage

Per-stage memory cap

Properties panel -> Advanced tab

PRAGMA memory_limit applied just to that stage

Per-stage materialize

Properties panel -> Basic tab

auto, view (lazy), memory (read once, table in RAM), or disk (read once, streamed via a temp Parquet file for huge intermediates)

DuckDB extensions

Pre-fetched at install; lazy-loaded for spatial

See First-launch extension pre-fetch

Env var RUST_LOG

Before launching the binary

RUST_LOG=debug duckle.exe to see verbose engine logs

Env var DUCKLE_DUCKDB_BIN

Before running engine tests

Points the integration test suite at a DuckDB CLI

Env var DUCKLE_CA_CERT

Before launching the binary

Path to a PEM bundle of extra CA certificates to trust (corporate proxy / private CA), added on top of the OS trust store and bundled roots

Env var DUCKLE_HTTPS_PROXY (or standard HTTPS_PROXY / HTTP_PROXY / ALL_PROXY)

Before launching the binary

Routes REST / cloud-API connectors and the in-app updater through an HTTP proxy, e.g. http://user:pass@proxy:8080. Use the standard vars to also cover engine / model downloads

Tip

Why

Use Parquet, not CSV, for intermediate steps

Columnar + compressed; DuckDB reads only the columns the next stage needs. CSV is fine for source / sink at the edges.

Push filters as early as possible

xf.filter early in the graph compiles to a WHERE that runs at scan time, not a post-scan filter.

Use the vss + fts indexes

Vector + full-text search hit DuckDB extensions directly. Faster than the alternative of pulling data out and indexing in Python.

Avoid per-row API calls when batch APIs exist

xf.ai.embed batches up to 100 inputs per request; snk.rest defaults to one batched request. Per-row patterns (xf.ai.llm, snk.webhook) are slower by design - use them when you actually need per-row behavior.

Cap heavy aggregates with the per-stage memory limit

Properties panel -> Advanced -> Memory limit (MB) prevents one big GROUP BY from blowing through all of RAM.

Use ctl.checkpoint for long-running pipelines

A checkpoint stage writes a Parquet snapshot to a path you choose, so a future run can resume from there with src.parquet.

Disable xf.debug.log in prod

Logging rows is per-row I/O; fine for dev, costly at scale.

Sort once at the end, not in the middle

xf.sort is a global sort; doing it once before the sink avoids re-sorting downstream.

Put an xf.dbt node behind its upstream, not first

When a dbt node has upstream stages, Duckle warms dbt's project parse in the background while those stages run, so dbt run reuses a warm cache instead of paying a cold parse. Set DUCKLE_DBT_PREWARM=0 to disable.

Symptom

Likely cause

Fix

Window opens but content shows "localhost refused to connect"

Release binary built without --features custom-protocol (the v0.0.7 bug)

Rebuild with cargo build --release --features custom-protocol per Build from source. The release workflow already passes this flag.

"DuckDB CLI not found" on Run

First-launch installer was skipped or interrupted

Open the engine setup modal from the toolbar; click Install on DuckDB

"Couldn't download Duckie AI Assistant (HTTP 404)"

Pinned llama.cpp build temporarily unavailable from upstream

Bump LLAMACPP_BUILD in apps/desktop/src/engine_manager.rs to a recent stable, rebuild

Linux: app won't launch, missing libwebkit

WebKitGTK 4.1 isn't installed

sudo apt install libwebkit2gtk-4.1-0 (Debian/Ubuntu) or your distro's equivalent

macOS: "App can't be opened because Apple cannot check it"

Gatekeeper, unsigned binary

Right-click the binary -> Open -> Open Anyway

Pipeline runs but a connector errors with "extension not loaded"

Lazy-loaded extension (e.g. spatial) downloaded mid-run and failed

Run duckdb :memory: -c "INSTALL spatial; LOAD spatial;" from a terminal to pre-install; relaunch Duckle

Chat panel says "AI engine not registered"

Old version of Duckle before AI shipped (pre-v0.0.10)

Update to latest release

Duckie generates a pipeline but Insert doesn't put anything on the canvas

Active pipeline tab has been closed; nothing to insert into

Open a pipeline (or create a new one) before clicking Insert

MotherDuck / Snowflake auth fails

Token expired, or PAT lacks the role you're trying to use

Regenerate in the vendor UI; paste into the Connection in Duckle

Postgres ATTACH says "could not connect"

Local SSL mode mismatch

Connection -> Advanced -> set SSL mode to disable for localhost / require for production

AI tests skip with no failure

DUCKLE_DUCKDB_BIN isn't set

export DUCKLE_DUCKDB_BIN=/path/to/duckdb before cargo test

TLS "UnknownIssuer" / "invalid peer certificate" behind a corporate proxy

A TLS-inspecting proxy (Zscaler, Netskope, ...) re-signs traffic with its own CA

Duckle trusts your OS certificate store on top of its bundled roots, so the proxy CA in the Windows / macOS / Linux store is honoured automatically. If the CA isn't in the store, point DUCKLE_CA_CERT at a PEM file containing it. Note: DuckDB's own extension fetch (extensions.duckdb.org) and cloud reads (S3 / GCS / Azure) run inside the DuckDB engine with its own TLS, so also allow / exempt extensions.duckdb.org from inspection.

REST / cloud calls fail with "Connection Failed" / timeout (os error 10060) behind a proxy

The network requires an HTTP proxy to reach the internet, and Duckle is connecting directly

Set HTTPS_PROXY (and HTTP_PROXY) to your proxy URL, e.g. http://user:pass@proxy:8080, before launching Duckle - REST / cloud connectors and the updater now route through it. Use DUCKLE_HTTPS_PROXY if you want a Duckle-only proxy without changing global env.

Trigger

GitHub Actions

GitLab CI

Push to main or feature branch

.github/workflows/ci.yml

.gitlab-ci.yml (test + desktop-build stages)

Pull request / merge request

.github/workflows/ci.yml

.gitlab-ci.yml (same stages, rules: gate on MR events)

Tag v*

.github/workflows/release.yml

.gitlab-ci.yml (release stage; uploads binaries to GitLab Releases)