create_pipeline
Creates a SingleStore pipeline from a full CREATE PIPELINE statement for continuous loading from S3, Kafka, Azure, GCS, or filesystem. Pipeline is registered but not started; use start_pipeline.
Instructions
Create a new pipeline from a full CREATE PIPELINE statement.
Pipeline definitions vary a lot by source (S3, Kafka, Azure Blob, GCS,
filesystem, ...), format (CSV/JSON/Avro/Parquet) and optional transforms,
so this tool takes the complete statement text rather than trying to
model every variant as separate parameters. It only checks that the
statement actually starts with CREATE [OR REPLACE] PIPELINE before
running it. Creating a pipeline does not start it -- call start_pipeline
afterwards, or include FOREGROUND handling via start_pipeline.
Example create_pipeline_sql:
CREATE PIPELINE my_pipeline AS
LOAD DATA S3 's3://my-bucket/path/'
CONFIG '{"region": "us-east-1"}'
CREDENTIALS '{"aws_access_key_id": "...", "aws_secret_access_key": "..."}'
INTO TABLE my_table
FIELDS TERMINATED BY ',';
Args:
create_pipeline_sql: The full CREATE PIPELINE ... statement.
database: Database to create the pipeline in (defaults to the
connection's configured database).
Input Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| database | No | ||
| create_pipeline_sql | Yes |
Output Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
No arguments | |||