Skip to main content
Glama
Datris MCP Server

Datris MCP Server

create_tap

Creates a tap that fetches external data into a Datris pipeline using AI-generated or custom Python scripts, or an HTTP endpoint, and can schedule recurring ingestion runs.

Instructions

Create a tap — a fetcher that pulls data from an external source and pushes it into a pipeline. Two kinds: a Python script the platform executes (default — provide a plain-English instruction to have AI generate the script, or supply your own script directly), or kind: "http" — a user-hosted HTTP endpoint (any language) that Datris POSTs the run context to on each run and which responds with the tap envelope; pass endpoint_url and see the tap-http-contract doc. HTTP taps run no code on the platform; AI script generation and the DATRIS_PLATFORM_* platform-data callback do NOT apply to them, so keep any tap whose fetch logic reads platform data as a Python tap. For a large columnar or tabular source, yield each batch (a pyarrow RecordBatch, a pandas DataFrame, or a list of dicts) instead of each row — the platform serialises a batch natively, several times faster — and read only the columns the pipeline needs (columns=[...]). Yielding one row at a time remains correct for small or API-paged sources. If the user wants the tap to feed a pipeline, pass target_pipeline now. Without it, run_tap will fetch but not persist (response will show persisted: false, persistedReason: "no_target_pipeline") — you'd then need to call update_tap to wire a pipeline. For a fetch-only tap (the user wants the rows back, not landed), target a Live Read (scratch) pipeline — see the KEEP-OR-READ-LIVE RULE in the server instructions. If the user mentioned ANY recurrence (nightly, daily, hourly, every morning, market open, etc.), pass cron_expression NOW — the platform's scheduler will run the tap on that cadence automatically. This is the canonical way to make a tap recurring; do NOT respond with shell commands or external schedulers for the user to run themselves. See the SCHEDULING RULE in the server instructions. AFTER creating, call test_tap to validate the script BEFORE any run_tap or before relying on a scheduled cron run — see the VALIDATION RULE. Setting a cron on a never-tested script is a guaranteed-bad nightly run waiting to happen. The platform enforces this: passing cron_expression for a script (or endpoint) that has not passed test_tap returns HTTP 409 with the remedy — create the tap without cron_expression, call test_tap until it succeeds, then update_tap with the cron. Replacing the script of a scheduled tap is refused the same way until re-tested — to replace the script of a SCHEDULED tap, clear the cron via update_tap (empty cron_expression) first, then call create_tap with the new script, test_tap, then update_tap with the cron. Scripts can read data already stored in Datris WITHOUT credentials: every run auto-injects DATRIS_PLATFORM_HOST, DATRIS_PLATFORM_PORT, DATRIS_POSTGRES_DATABASE, and DATRIS_MONGODB_DATABASE, and the script queries via POST http://{host}:{port}/api/v1/query/postgres with {"sql": ..., "database": <pg_db>, "limit": -1} (or /query/mongodb with query/database/collection/limit) — the platform executes it with its own credentials and returns {results, count}. NEVER request the platform's own DB credentials in a tap secret and NEVER hardcode a snapshot of platform data into the script — read it live. The tap secret carries only the external source's credentials. Large sources: fetch() may yield records (a generator / iterator) instead of returning a list — the platform streams yielded records to disk one at a time, so the run is not limited by memory; a script that builds millions of rows into a list is killed for memory (exit code -9). Read the source in chunks / pages and yield each row; never cap the rows or split the source for size unless the estimate below exceeds a budget and the user has chosen chunking. Stream the source, never download it: the runner's only writable area is a 512 MB in-memory scratch space shared with installed packages, so a script that saves a large file locally fails with No space left on device — read remote files directly (HTTP range reads via pyarrow + fsspec, pd.read_csv(url, chunksize=...), requests stream=True) and yield as you go. Size a large source BEFORE creating anything: staged bytes per record ≈ 26 bytes × column count (JSON keys repeat on every row) against PIPELINE_MAX_PAYLOAD_MB (the value in force is pipelineMaxPayloadMB from get_version; default 4096 MB, 0 = unlimited), and script time ≈ rows ÷ 15,000 per second emitting one record at a time or rows ÷ 90,000 per second emitting batches against TAP_RUN_TIMEOUT_SECONDS (tapRunTimeoutSeconds from get_version; default 3600 s) — call get_version for both values in force before you size, name the rate you used, show the user a rows, bytes/record, GB staged, minutes table, and if either estimate is over its budget let the user choose first: raise the variable, narrow the window, or chunk the range via run_tap params. Reading only the columns the destination needs cuts both numbers linearly. If this Datris instance's agent policy requires a person to approve this action, the call does NOT perform it: it returns status pending_approval with an approvalId. Tell the user it is waiting for approval in Datris, then poll get_approval with that id (pause with wait_seconds between polls) — do not re-issue the action.

Tool call template

Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {
      "name": "<name>"
    },
    "name": "create_tap"
  }
}

Input Schema

NameRequiredDescriptionDefault
kindNoTap implementation kind. 'python' (default): a script the platform executes. 'http': a user-hosted endpoint speaking the tap HTTP contract — requires `endpoint_url`; `instruction`, `script`, and `packages` do not apply and are rejected.
nameYesUnique tap name.
reasonNoOptional one-line reason for this change, recorded in the platform's audit log and shown to anyone asked to approve it.
scriptNoRaw Python source code with a fetch() function. Use this to provide your own script instead of AI generation.
sourceNoWhere the data really comes from, as a short provider name or host, for lineage and provenance. Optional: when omitted it is derived from the endpoint host or the host the script references most. Never put credentials here.
packagesNoExtra pip packages the script imports that aren't pre-installed. Required when `script` imports non-stdlib modules beyond the pre-installed set (requests, beautifulsoup4, pandas, lxml, feedparser, boto3, google-cloud-storage, azure-storage-blob, openpyxl, pyyaml, python-dateutil, pytz). Pass the PyPI package names as a list of strings. When `instruction` is used instead of `script`, the AI populates this automatically — if you pass it anyway, your value wins.
tap_typeNoTap type. 'structured' (default) returns rows of records. 'document' returns a list of {uri, filename, content (base64)} dicts; the platform stages each document and uses a ledger to skip files it has already processed. A document tap's target_pipeline MUST have an unstructuredAttributes source and a vector-store destination (qdrant, pgvector, weaviate, milvus, or chroma) — the server rejects save attempts that violate this.
depends_onNoOptional: names of other taps in the same catalog whose destination tables this tap reads. Omit to let the platform infer them from the script's queries; pass an empty list to declare that it has none. Used only to order and explain the catalog's enable order (catalog_enable_order); never enforced.
instructionNoPlain-English instruction for AI script generation. Describe the source and the data to fetch, in one or two sentences. If provided, AI generates the Python script.
secret_nameNoVault secret name containing API keys/credentials the script needs for the EXTERNAL source. Never for the platform's own databases — scripts reach platform data credential-free via the auto-injected DATRIS_PLATFORM_* env vars and the /api/v1/query/* callback. Must be a tap secret (see list_tap_secrets / create_tap_secret); a platform secret is refused by the server.
endpoint_urlNoFor kind 'http' only: absolute http(s) URL Datris POSTs {tap, params, state, testLimit} to on each run. The endpoint responds with the tap envelope {type, data, state?, logs?}. If the tap's secret has an `endpoint_token` field, it is sent as an Authorization: Bearer header — no other secret fields are ever forwarded.
cron_expressionNoQuartz CRON expression for recurring runs: 6 fields — seconds minutes hours day-of-month month day-of-week, plus an optional year (e.g., '0 0 * * * ?' for hourly, '0 30 5 ? * MON-FRI' for weekdays 5:30am). A 5-field Unix cron such as '*/1 * * * *' is NOT a Datris schedule: the save is refused with HTTP 400 whose `error` quotes the 6-field equivalent to resend. SET THIS whenever the user describes a recurrence — nightly, daily, hourly, market open, etc. The platform's scheduler fires the tap on this cadence automatically; do not propose external schedulers or shell cron for the user to run themselves.
target_pipelineNoName of the pipeline to push fetched data into

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observedv0.1.0

TDQS

A4.6/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations supplied, the description carries the full burden and does so unusually thoroughly: it discloses the HTTP 409 enforcement for cron-on-untested scripts, the refusal to replace a scheduled tap's script, the credential-free DATRIS_PLATFORM_* callback, the exit-code -9 memory kill, the 512 MB scratch limit, and the pending_approval flow with approvalId. These are exactly the non-obvious behaviors an agent would otherwise violate.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness2/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The first sentence is well front-loaded and the two-kind split comes early, but the description then runs to several hundred words covering sizing math, batching rates, memory limits, and approval polling — much of which restates the server instructions it explicitly points to (SCHEDULING RULE, VALIDATION RULE, KEEP-OR-READ-LIVE RULE). Valuable individually, but collectively this is oversized for a tool definition and buries the selection signal.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 13-parameter mutation tool with no annotations and no output schema, the description covers what's needed to call it correctly: creation modes, scheduling prerequisites, pipeline wiring, failure modes with status codes, and the approval path. Nothing essential to a correct first invocation is left to inference.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the baseline is 3; the description exceeds that by attaching consequences to specific parameters — target_pipeline's persisted:false / no_target_pipeline outcome, cron_expression's 6-field Quartz requirement and 409 behavior, the interaction of kind:"http" with instruction/script/packages, and secret_name's tap-vs-platform restriction. It stops short of 5 because most per-field syntax still lives in the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The opening sentence gives a specific verb and resource ('Create a tap') and immediately defines the resource functionally ('a fetcher that pulls data from an external source and pushes it into a pipeline'). It then splits the resource into its two concrete forms (python script vs kind: "http" endpoint), so an agent can distinguish it from siblings like update_tap or test_tap without opening the schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicit routing is provided for nearly every decision point: use instruction for AI generation vs script for your own code, pass target_pipeline now or run_tap won't persist, pass cron_expression now if any recurrence was mentioned, and call test_tap after creation before run_tap. Alternatives are named (update_tap, run_tap, test_tap, get_version, get_approval) along with the conditions that select them.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Deploy Server

Other Tools