Set up a scheduled SFTP or S3 pull into an existing data spec
setup_scheduled_pullEnd-to-end workflow for "pull files from this SFTP server / S3 bucket on a schedule" requests: reuses a matching connection if one already exists in the workspace (same hostname/username for sftp, same roleArn for aws_s3), otherwise creates one; tests it; then creates a trigger that feeds an already-analyzed data spec (see onboard_data_source) on the given frequency. Pass hostname for an sftp pull, or roleArn (+ s3Bucket, required) for an aws_s3 pull — exactly one of the two is expected. Use this instead of calling manage_connection + manage_trigger yourself for first-time setup. If the connection test fails (e.g. the sftp public key or the aws_s3 IAM role isn't set up yet on the customer's side), no trigger is created — ask the user to finish that setup and re-run this tool, which will reuse the same connection and pick up where it left off. This is for pulling a NEW file from an external source — for "run this on a schedule/after another job" where the spec queries tables already in the workspace (sourceType "tables"), use manage_trigger with type "schedule" or "spec_success" instead; there is no connection involved.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| dedupe | No | Required — ask the user rather than assuming a value; omitting it fails the call. Whether repeat pulls should skip files already loaded into this spec, matched by file name. Has real consequences: with dedupe true, a file that reappears under the same name (e.g. re-uploaded with corrected data) will be silently skipped; with dedupe false, an unchanged file left on the server will be reloaded every run. | |
| roleArn | No | aws_s3: the IAM role the customer will create/update. | |
| hostname | No | sftp: SFTP server hostname to pull from. | |
| preRules | No | Natural language: which files to pick up (e.g. "only *.csv under /outbound") | |
| s3Bucket | No | aws_s3: bucket to poll. Required when roleArn is given. | |
| s3Prefix | No | aws_s3 only. Optional key prefix; defaults to the whole bucket. | |
| specName | Yes | Already-analyzed data spec to load files into (see onboard_data_source) | |
| username | No | sftp only. Defaults to "sftpuser". | |
| frequency | Yes | Pull schedule. | |
| postRules | No | Natural language: what to do after a file loads (e.g. "rename with .done suffix") | |
| workspaceId | No | Workspace to act on. Defaults to your only workspace if you have exactly one. |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| dedupe | No | ||
| specId | No | ||
| enabled | No | ||
| message | Yes | ||
| s3Bucket | No | ||
| s3Prefix | No | ||
| specName | No | ||
| warnings | No | aws_s3 only. Advisory notes about the created trigger, e.g. the 5000-object S3 listing cap. | |
| frequency | No | ||
| triggerId | No | Present only once the connection test succeeded and a trigger was created. | |
| connectionId | No | ||
| testSucceeded | No | ||
| connectionType | No |