create_pipeline
Create a data pipeline that ingests into Postgres, MongoDB, Snowflake, Databricks, SQL Server, object store, or vector destinations, with schema detection, field protection, and validation rules.
Instructions
Create a pipeline. TAP-FED PIPELINES: when a tap will feed the pipeline, create and test the tap first, call propose_pipeline_schema, confirm the column types with the user, and pass the confirmed list as schema_fields — no sample content is needed and the destination gets typed columns on the first run (not mongodb: it stores documents as-is, so pass a few preview rows as JSON via content_text with a .json filename instead). FOUR destination categories: STRUCTURED (postgres, mongodb, snowflake, databricks, sqlserver) — send schema_fields (tap-fed; not mongodb) or a TINY sample (via content_text) and the schema is auto-detected; snowflake additionally REQUIRES credentialsSecret (a Platform secret with account/user/privateKey or password — discover via list_platform_secrets), warehouse, and database; databricks additionally REQUIRES credentialsSecret (a Platform secret with host plus clientId/clientSecret or token) and database (the Unity Catalog name), plus warehouse (the SQL warehouse ID) unless the Databricks secret has a warehouse field — in that case omit warehouse and do NOT ask the human for it (list_platform_secrets / get_platform_secret_fields show the field names); sqlserver additionally REQUIRES credentialsSecret (a Platform secret with host plus the fields its authMethod needs) and database, schema defaults to dbo, and it takes a CSV, JSON or XML sample. OBJECTSTORE (objectstore — writes Parquet files, ORC files, or an Iceberg table to MinIO or AWS S3) — same shape as structured (CSV-only, sample required for schema detection), plus objectStore-specific knobs (bucket, prefix, fileFormat, partitionBy, writeMode; keyFields for iceberg+merge; provider+credentialsSecret for S3). VECTOR (pgvector, qdrant, weaviate, milvus, chroma) — no schema; pass ONLY pipeline + destination (and optionally filename for the file-extension hint), no sample at all. LIVE READ (a Live Read pipeline, destination scratch) — same sample-required shape as structured, nothing is landed; the run's rows come back on the status rollup (resultPreview) and via get_pipeline_result; results expire — use it when the user wants an answer now, a validation result, or a transformed view with no reason to keep the rows (see the KEEP-OR-READ-LIVE RULE). SAMPLE-SIZE RULE: content_text is the fallback for data without a tap (a file in hand, pasted rows); the sample exists ONLY for schema detection and every column lands as string — send the header row plus 3-5 representative rows, NEVER a full dataset. Composing hundreds of rows here wastes minutes of generation time; the real data arrives later via the tap or upload_data. Prefer content_text (plain text) over content (base64) — the server encodes it for you. FIELD PROTECTION: when the user asks to protect sensitive fields, pass protect (field name → {method: hmac|mask|redact|drop|encrypt}) so those fields are protected before any AI stage or destination sees them; confirm the fields with the user, never guess. CODEGEN SCRIPTS: an AI rule or AI transformation is turned into a Python script when the pipeline is saved and every run executes that stored script; the result's codegenScripts says per kind whether it is ready or pending (pending: the first run generates it) — relay a pending reason to the user. With a code repository configured the script is committed there; an entry's warning means the file was edited in the repository and was not overwritten — relay it as written. If this Datris instance's agent policy requires a person to approve this action, the call does NOT perform it: it returns status pending_approval with an approvalId. Tell the user it is waiting for approval in Datris, then poll get_approval with that id (pause with wait_seconds between polls) — do not re-issue the action.
Tool call template
Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {
"pipeline": "<pipeline>"
},
"name": "create_pipeline"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| role | No | Optional Snowflake role to assume (e.g. DATRIS_LOADER). Only applies to destination=snowflake; omit to use the service user's default role. | |
| table | No | Destination table or collection name (default: pipeline name). Ignored for objectstore (the destination is bucket+prefix, not a table). | |
| bucket | No | Object-store bucket name. Only applies to destination=objectstore. For MinIO this is optional (default: {environment}-data). For S3 (provider=s3) this is REQUIRED — there is no global default S3 bucket. | |
| header | No | Whether CSV has a header row (default: true) | |
| prefix | No | Object-store key prefix under the bucket (e.g. 'sales/daily'). REQUIRED for destination=objectstore. | |
| reason | No | Optional one-line reason for this change, recorded in the platform's audit log and shown to anyone asked to approve it. | |
| schema | No | Destination schema. Applies to destination=snowflake (default: PUBLIC), destination=databricks (default: default) and destination=sqlserver (default: dbo). Simple identifiers resolve case-insensitively; names with hyphens/spaces are quoted (double quotes on Snowflake, backticks on Databricks, brackets on SQL Server). | |
| catalog | No | OMIT BY DEFAULT. Catalogs are a user-chosen organizational convention — do NOT set a catalog unless the user has explicitly asked to group this pipeline under a named catalog. Assigning one for them puts the pipeline into a taxonomy they didn't ask for. When unset, the platform shows the pipeline as Uncataloged, which is the correct default. | |
| content | No | Base64-encoded sample data. Prefer content_text instead — it's the same thing without you having to base64-encode. Only use this form when you already have base64 in hand (e.g. relaying an attachment). OMIT for vector destinations — the file goes through upload_data after this call returns. | |
| protect | No | OMIT BY DEFAULT. Field name → protection policy, e.g. {"account_id": {"method": "hmac"}, "contact_email": {"method": "mask", "preserve": "domain"}}. Each policy is set as `protect` on the matching schema field (case-insensitive name match); a name not in the detected schema is an error and nothing is saved. For a JSON source (schema is the single `_json` field) each name is a top-level key and is added beside `_json`. Protected fields are rewritten before data quality, transformation, and every destination, so no AI stage sees their raw values. hmac = deterministic pseudonym, mask = asterisks (preserve last4/domain/year/first3 keeps that part), redact = fixed marker, drop = remove the column, encrypt = ciphertext reversible on the Datris server only, by an operator holding the protect:reveal capability, never by an agent. hmac/mask/redact/encrypt need string fields; a keyFields column may only use hmac. The raw ingest object is deleted once the protected copy exists. Only set this when the user asked to protect fields, and confirm with them which fields and methods — never decide from column names yourself. suggest_field_protection can propose policies; pass only the ones the user accepted. | |
| database | No | Destination database name (default: datris). Ignored for objectstore. REQUIRED for snowflake — there is no default Snowflake database; ask the user which one to load. REQUIRED for databricks — it names the Unity Catalog CATALOG; there is no default, ask the user. REQUIRED for sqlserver — the database on the user's server; there is no default, ask the user. | |
| endpoint | No | S3 endpoint URL override. Only applies to destination=objectstore + provider=s3. Must use https://. Leave unset for the AWS regional default. Ignored for provider=minio. | |
| filename | No | Filename (e.g., data.csv, report.json, orders.xml). REQUIRED for structured destinations and objectstore. Optional for vector destinations — only used as a fileExtension hint (defaults to txt). | |
| pipeline | Yes | Pipeline name | |
| provider | No | Object-store provider. Only applies to destination=objectstore. Default 'minio' (the platform's built-in store). Set to 's3' to write to AWS S3 — that requires bucket AND credentialsSecret. | |
| truncate | No | Optional. When true, the destination table/collection is truncated before each run, so only the latest run's data is kept. Only applies to postgres, mongodb, snowflake, databricks, and sqlserver destinations (on Databricks the replace is a single atomic Delta commit). Default false (append). Mutually useful with — but distinct from — keyFields: truncate wipes everything each run, keyFields upserts per natural key. | |
| delimiter | No | CSV delimiter (default: comma) | |
| keyFields | No | Optional natural-key columns used to dedupe / upsert rows on every run. Applies to postgres, mongodb, snowflake, databricks, and sqlserver destinations, and to objectstore when fileFormat=iceberg and writeMode=merge (required there; for objectstore it is DROPPED unless fileFormat=iceberg and writeMode=merge). Example: ['user_id', 'event_date'] — rows with the same (user_id, event_date) will replace the existing row instead of appending. On Postgres this triggers a staging + INSERT…ON CONFLICT path; on Mongo it uses upsertJSON; on Snowflake it uses MERGE via a temp staging table; on Databricks it uses MERGE directly from the staged file; on SQL Server it uses MERGE via a temp staging table; on an Iceberg table it is a MERGE INTO keyed on those columns. NOTE: on conflict, ALL non-key columns from the incoming row overwrite the existing row, including NULLs (true upsert semantics, not non-null merge). If your source emits partial rows, coalesce upstream. Omit to append on every run (default behavior). | |
| warehouse | No | Compute that runs the load. For destination=snowflake: the virtual warehouse NAME (e.g. DATRIS_WH). For destination=databricks: the SQL warehouse ID from the warehouse's Connection details — the trailing segment of the HTTP path /sql/1.0/warehouses/<id> — NOT the warehouse name. REQUIRED for snowflake. For databricks, optional when the Databricks secret has a `warehouse` field (the pipeline then uses it — do not ask the human); otherwise required. Ignored for other destinations. | |
| writeMode | No | Object-store write mode. Only applies to destination=objectstore. Default: append. 'merge' is only valid with fileFormat=iceberg and requires keyFields — rows whose key matches an existing row replace it, the rest are inserted. | |
| fileFormat | No | Object-store storage format. Only applies to destination=objectstore. 'parquet' and 'orc' write loose columnar files under the prefix; 'iceberg' writes a table at the prefix (snapshots, schema evolution, and writeMode=merge with keyFields). Default: parquet. | |
| destination | No | Destination type (default: postgres for CSV, mongodb for JSON/XML). 'scratch' = Live Read: hand the rows back, land nothing. Use it when nothing should be landed and the rows should come back to the caller (status rollup resultPreview + get_pipeline_result); table/database/keyFields are ignored for it. Use 'objectstore' for Parquet files, ORC files, or an Iceberg table (fileFormat=iceberg) in MinIO (default provider) or AWS S3 (set provider=s3). Use 'snowflake' to load the user's Snowflake account — requires credentialsSecret, warehouse, and database. Use 'databricks' to load a Unity Catalog managed Delta table in the user's Databricks workspace — requires credentialsSecret and database (Unity Catalog name), plus warehouse (SQL warehouse ID) unless the secret has a `warehouse` field. Use 'sqlserver' to load the user's Microsoft SQL Server or Azure SQL database — requires credentialsSecret and database; accepts CSV, JSON or XML samples (JSON/XML land as one document column). | |
| partitionBy | No | Optional partition columns for objectstore writes. Spark creates a directory structure based on the distinct values of these columns. Field names must be in the destination schema. Example: ['dt', 'region']. | |
| codegen_rule | No | Optional data quality validation rule as a plain-English instruction. Only add when the user explicitly requests validation. Datris will generate a Python validation script from this instruction and run it locally against all data. Pass the checks the user confirmed verbatim, character for character, as a numbered list, one check per line, each naming the column it covers; add nothing they did not confirm. Pair with `codegen_rule_on_failure` (`warn` | `error`) as the user chose. | |
| content_text | No | Plain-text sample data (the server base64-encodes it for you) — the fallback for data without a tap (a file in hand, pasted rows) for structured destinations (postgres, mongodb, snowflake, databricks, sqlserver) and objectstore; tap-fed pipelines pass schema_fields instead. For sqlserver a CSV, JSON or XML sample all work. Header row + 3-5 representative rows ONLY; never a full dataset (the sample is used solely for schema auto-detection). Ignored if content is also set. OMIT for vector destinations. | |
| authoritative | No | OMIT BY DEFAULT. Pass false only when the user says this pipeline lands a derived copy (a rollup, a replica, an index built from another dataset) rather than the system of record. A pipeline's single destination is the authoritative copy by default; lineage and find_data mark derived copies so agents cite the right one. | |
| schema_fields | No | The column types the user confirmed from propose_pipeline_schema, as [{name, type}]. Pass it for a tap-fed pipeline INSTEAD of content/content_text: the server builds the schema from these names and applies these types, so the destination gets typed columns on the first run. Names match the generated columns case-insensitively; every name must be a column. Types: string, boolean, int, bigint, float, double, date, timestamp. Composes with protect, protect_preset, codegen_rule and codegen_transform. Accepted for postgres, snowflake, databricks, sqlserver, objectstore and scratch; rejected for mongodb (documents are stored as-is — pass a JSON content_text sample) and vector destinations (no column schema). | |
| unity_catalog | No | OMIT BY DEFAULT. Pass true only when the user asks for Unity Catalog. Supported for destination=databricks and for destination=objectstore with fileFormat=iceberg; the server rejects it elsewhere. Object store + iceberg: after each successful write the table is registered in Unity Catalog as <catalog>.<schema>.<pipeline> — also pass unity_catalog_secret and unity_catalog_catalog (for Databricks Unity Catalog also pass unity_catalog_mode=managed with provider s3) (unity_catalog_schema optional, default "default"); the metastore needs external data access enabled, the principal EXTERNAL USE SCHEMA on the schema, and an external location covering the S3 path; a failed registration is a warning on the run, not a failed load. Databricks: after each successful load, pushes a table comment, provenance column comments, tags (datris_pipeline, datris_catalog, datris_dq_status, managed_by) and table properties (last run id, config version, lineage path) to Unity Catalog. The service principal needs GRANT APPLY TAG ON SCHEMA on the target schema, or tagging is reported as a warning on the run (the load still succeeds). Also publishes External Lineage (tap/upload → pipeline → table, the config key `lineage`, default on), which needs CREATE EXTERNAL METADATA on the metastore; a missing grant is a warning, not a failed load. The install may already turn this on by default for Databricks pipelines (DATRIS_UNITY_CATALOG_DEFAULT=enabled), so a Databricks pipeline created without it can still sync; GET /api/v1/pipelines/<name>/unity-catalog shows enabledBy (pipeline or default). Opting one pipeline out takes "unityCatalog": {"enabled": false} in its config. | |
| protect_exempt | No | OMIT BY DEFAULT. Field names the user chose to leave unprotected under protect_preset (saved as protection.presetExempt). Only with protect_preset, and only names the user confirmed. | |
| protect_preset | No | OMIT BY DEFAULT. Keep a field-protection preset enforced on this pipeline (saved as protection.preset; delimited/CSV sources only). Every field the preset classifies, now and in later saves or new columns, must carry `protect` or be listed in protect_exempt, otherwise the server refuses the save naming the field. Pass the protect map the user confirmed from suggest_field_protection with the same preset. Only set this when the user asked for the preset. | |
| codegen_transform | No | Optional transformation instruction as a plain-English description. Only add when the user explicitly requests transformation. Datris will generate a Python script from this instruction and run it locally to transform all data. Pass the transformations the user confirmed verbatim, character for character, as a numbered list, one per line, each naming the column it changes; add nothing they did not confirm. | |
| credentialsSecret | No | Name of an existing PLATFORM secret holding destination credentials. For objectstore + provider=s3: fields accessKey, secretKey, region (optionally sessionToken); REQUIRED unless Datris runs in AWS with an instance role. For snowflake: fields account, user, and privateKey (key-pair auth) or password; ALWAYS required. For databricks: fields host, plus clientId/clientSecret (service-principal OAuth) or token (personal access token); ALWAYS required. For sqlserver: field host (optional port), plus by authMethod — sql (default): user, password; entraServicePrincipal: clientId, clientSecret, tenantId; entraPassword: user, password; entraManagedIdentity: optional managedIdentityClientId; kerberos: principal, password; ALWAYS required. Discover candidates via list_platform_secrets and verify field shape via get_platform_secret_fields. The agent cannot create this secret — if none exists, ask the user to create one on Configuration → Secrets → Platform. | |
| deleteBeforeWrite | No | When true, delete existing objects under the prefix before writing. Only applies to destination=objectstore. Default: false. Distinct from writeMode=overwrite (which is Spark-level). | |
| unity_catalog_mode | No | Absent means register. Only with unity_catalog=true on destination=objectstore + fileFormat=iceberg (ignored otherwise). managed is the mode for Databricks Unity Catalog: the catalog chooses the table's location in the pipeline's bucket (the schema needs a MANAGED LOCATION there), every write commits through the catalog, provider must be s3 (minio only with a non-Databricks catalog secret carrying icebergRestPath), deleteBeforeWrite is rejected, and any refusal fails the run. rest commits every write through the Unity Catalog Iceberg REST catalog at the pipeline's prefix (catalogs that honour a requested location, not Databricks). Once a table is committed through the catalog the pipeline cannot switch to another mode. Pass rest only when the user asks to keep a non-Databricks catalog current. | |
| unity_catalog_schema | No | Only with unity_catalog=true on destination=objectstore + fileFormat=iceberg (ignored otherwise). The Unity Catalog schema the table is registered under. Default: default. | |
| unity_catalog_secret | No | Only with unity_catalog=true on destination=objectstore + fileFormat=iceberg (ignored otherwise). Name of an existing PLATFORM secret for the Unity Catalog workspace: field host, plus clientId/clientSecret (service-principal OAuth) or token. Usually the same secret a Databricks pipeline uses. Discover via list_platform_secrets; the agent cannot create it. | |
| unity_catalog_catalog | No | Only with unity_catalog=true on destination=objectstore + fileFormat=iceberg (ignored otherwise). The Unity Catalog catalog the table is registered under. No default — ask the user. | |
| codegen_rule_on_failure | No | Optional. What a row that fails the data-quality rule does: `error` (default) fails the run; `warn` logs the failing row and keeps loading. Only read together with `codegen_rule`. Pass what the user chose when confirming the checks. |