create_tap
Creates a tap that fetches external data into a Datris pipeline using AI-generated or custom Python scripts, or an HTTP endpoint, and can schedule recurring ingestion runs.
Instructions
Create a tap — a fetcher that pulls data from an external source and pushes it into a pipeline. Two kinds: a Python script the platform executes (default — provide a plain-English instruction to have AI generate the script, or supply your own script directly), or kind: "http" — a user-hosted HTTP endpoint (any language) that Datris POSTs the run context to on each run and which responds with the tap envelope; pass endpoint_url and see the tap-http-contract doc. HTTP taps run no code on the platform; AI script generation and the DATRIS_PLATFORM_* platform-data callback do NOT apply to them, so keep any tap whose fetch logic reads platform data as a Python tap. For a large columnar or tabular source, yield each batch (a pyarrow RecordBatch, a pandas DataFrame, or a list of dicts) instead of each row — the platform serialises a batch natively, several times faster — and read only the columns the pipeline needs (columns=[...]). Yielding one row at a time remains correct for small or API-paged sources. If the user wants the tap to feed a pipeline, pass target_pipeline now. Without it, run_tap will fetch but not persist (response will show persisted: false, persistedReason: "no_target_pipeline") — you'd then need to call update_tap to wire a pipeline. For a fetch-only tap (the user wants the rows back, not landed), target a Live Read (scratch) pipeline — see the KEEP-OR-READ-LIVE RULE in the server instructions. If the user mentioned ANY recurrence (nightly, daily, hourly, every morning, market open, etc.), pass cron_expression NOW — the platform's scheduler will run the tap on that cadence automatically. This is the canonical way to make a tap recurring; do NOT respond with shell commands or external schedulers for the user to run themselves. See the SCHEDULING RULE in the server instructions. AFTER creating, call test_tap to validate the script BEFORE any run_tap or before relying on a scheduled cron run — see the VALIDATION RULE. Setting a cron on a never-tested script is a guaranteed-bad nightly run waiting to happen. The platform enforces this: passing cron_expression for a script (or endpoint) that has not passed test_tap returns HTTP 409 with the remedy — create the tap without cron_expression, call test_tap until it succeeds, then update_tap with the cron. Replacing the script of a scheduled tap is refused the same way until re-tested — to replace the script of a SCHEDULED tap, clear the cron via update_tap (empty cron_expression) first, then call create_tap with the new script, test_tap, then update_tap with the cron. Scripts can read data already stored in Datris WITHOUT credentials: every run auto-injects DATRIS_PLATFORM_HOST, DATRIS_PLATFORM_PORT, DATRIS_POSTGRES_DATABASE, and DATRIS_MONGODB_DATABASE, and the script queries via POST http://{host}:{port}/api/v1/query/postgres with {"sql": ..., "database": <pg_db>, "limit": -1} (or /query/mongodb with query/database/collection/limit) — the platform executes it with its own credentials and returns {results, count}. NEVER request the platform's own DB credentials in a tap secret and NEVER hardcode a snapshot of platform data into the script — read it live. The tap secret carries only the external source's credentials. Large sources: fetch() may yield records (a generator / iterator) instead of returning a list — the platform streams yielded records to disk one at a time, so the run is not limited by memory; a script that builds millions of rows into a list is killed for memory (exit code -9). Read the source in chunks / pages and yield each row; never cap the rows or split the source for size unless the estimate below exceeds a budget and the user has chosen chunking. Stream the source, never download it: the runner's only writable area is a 512 MB in-memory scratch space shared with installed packages, so a script that saves a large file locally fails with No space left on device — read remote files directly (HTTP range reads via pyarrow + fsspec, pd.read_csv(url, chunksize=...), requests stream=True) and yield as you go. Size a large source BEFORE creating anything: staged bytes per record ≈ 26 bytes × column count (JSON keys repeat on every row) against PIPELINE_MAX_PAYLOAD_MB (the value in force is pipelineMaxPayloadMB from get_version; default 4096 MB, 0 = unlimited), and script time ≈ rows ÷ 15,000 per second emitting one record at a time or rows ÷ 90,000 per second emitting batches against TAP_RUN_TIMEOUT_SECONDS (tapRunTimeoutSeconds from get_version; default 3600 s) — call get_version for both values in force before you size, name the rate you used, show the user a rows, bytes/record, GB staged, minutes table, and if either estimate is over its budget let the user choose first: raise the variable, narrow the window, or chunk the range via run_tap params. Reading only the columns the destination needs cuts both numbers linearly. If this Datris instance's agent policy requires a person to approve this action, the call does NOT perform it: it returns status pending_approval with an approvalId. Tell the user it is waiting for approval in Datris, then poll get_approval with that id (pause with wait_seconds between polls) — do not re-issue the action.
Tool call template
Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {
"name": "<name>"
},
"name": "create_tap"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| kind | No | Tap implementation kind. 'python' (default): a script the platform executes. 'http': a user-hosted endpoint speaking the tap HTTP contract — requires `endpoint_url`; `instruction`, `script`, and `packages` do not apply and are rejected. | |
| name | Yes | Unique tap name. | |
| reason | No | Optional one-line reason for this change, recorded in the platform's audit log and shown to anyone asked to approve it. | |
| script | No | Raw Python source code with a fetch() function. Use this to provide your own script instead of AI generation. | |
| source | No | Where the data really comes from, as a short provider name or host, for lineage and provenance. Optional: when omitted it is derived from the endpoint host or the host the script references most. Never put credentials here. | |
| packages | No | Extra pip packages the script imports that aren't pre-installed. Required when `script` imports non-stdlib modules beyond the pre-installed set (requests, beautifulsoup4, pandas, lxml, feedparser, boto3, google-cloud-storage, azure-storage-blob, openpyxl, pyyaml, python-dateutil, pytz). Pass the PyPI package names as a list of strings. When `instruction` is used instead of `script`, the AI populates this automatically — if you pass it anyway, your value wins. | |
| tap_type | No | Tap type. 'structured' (default) returns rows of records. 'document' returns a list of {uri, filename, content (base64)} dicts; the platform stages each document and uses a ledger to skip files it has already processed. A document tap's target_pipeline MUST have an unstructuredAttributes source and a vector-store destination (qdrant, pgvector, weaviate, milvus, or chroma) — the server rejects save attempts that violate this. | |
| depends_on | No | Optional: names of other taps in the same catalog whose destination tables this tap reads. Omit to let the platform infer them from the script's queries; pass an empty list to declare that it has none. Used only to order and explain the catalog's enable order (catalog_enable_order); never enforced. | |
| instruction | No | Plain-English instruction for AI script generation. Describe the source and the data to fetch, in one or two sentences. If provided, AI generates the Python script. | |
| secret_name | No | Vault secret name containing API keys/credentials the script needs for the EXTERNAL source. Never for the platform's own databases — scripts reach platform data credential-free via the auto-injected DATRIS_PLATFORM_* env vars and the /api/v1/query/* callback. Must be a tap secret (see list_tap_secrets / create_tap_secret); a platform secret is refused by the server. | |
| endpoint_url | No | For kind 'http' only: absolute http(s) URL Datris POSTs {tap, params, state, testLimit} to on each run. The endpoint responds with the tap envelope {type, data, state?, logs?}. If the tap's secret has an `endpoint_token` field, it is sent as an Authorization: Bearer header — no other secret fields are ever forwarded. | |
| cron_expression | No | Quartz CRON expression for recurring runs: 6 fields — seconds minutes hours day-of-month month day-of-week, plus an optional year (e.g., '0 0 * * * ?' for hourly, '0 30 5 ? * MON-FRI' for weekdays 5:30am). A 5-field Unix cron such as '*/1 * * * *' is NOT a Datris schedule: the save is refused with HTTP 400 whose `error` quotes the 6-field equivalent to resend. SET THIS whenever the user describes a recurrence — nightly, daily, hourly, market open, etc. The platform's scheduler fires the tap on this cadence automatically; do not propose external schedulers or shell cron for the user to run themselves. | |
| target_pipeline | No | Name of the pipeline to push fetched data into |