create_ingestion_job
Creates a data ingestion job to load data from cloud storage into watsonx.data tables, supporting multiple file formats and write modes.
Instructions
Create a data ingestion job to load data into watsonx.data.
Args:
job_id: Unique job identifier (e.g., "ingestion-1234567890")
catalog: Target catalog name
schema: Target schema name
table: Target table name
file_paths: Source file path (e.g., "s3://bucket-name/file.csv")
file_type: Source file type - "csv", "parquet", "json", "orc", "avro" (default: "csv")
bucket_name: S3 bucket name (extracted from file_paths if not provided)
bucket_type: Bucket type - "amazon_s3", "aws_s3", "minio", "ibm_cos", "ibm_ceph",
"adls_gen1", "adls_gen2", "google_cs", "ibm_storage_scale", "ozone" (default: "ibm_cos")
write_mode: Write mode - "append", "overwrite" (default: "append")
engine_id: Spark engine ID to use for ingestion
field_delimiter: CSV field delimiter (default: ",")
line_delimiter: CSV line delimiter (default: "") escape_character: CSV escape character (default: "") header: Whether CSV has header row (default: true) encoding: File encoding (default: "UTF-8") driver_memory: Spark driver memory (default: "2G") driver_cores: Spark driver cores (default: 1) executor_memory: Spark executor memory (default: "2G") executor_cores: Spark executor cores (default: 1) num_executors: Number of Spark executors (default: 1)
Returns:
Dict with job_id, status, and creation detailsInput Schema
| Name | Required | Description | Default |
|---|---|---|---|
| job_id | Yes | ||
| catalog | Yes | ||
| schema | Yes | ||
| table | Yes | ||
| file_paths | Yes | ||
| file_type | No | csv | |
| bucket_name | No | ||
| bucket_type | No | ibm_cos | |
| write_mode | No | append | |
| engine_id | No | ||
| field_delimiter | No | , | |
| line_delimiter | No | ||
| escape_character | No | \ | |
| header | No | ||
| encoding | No | UTF-8 | |
| driver_memory | No | 2G | |
| driver_cores | No | ||
| executor_memory | No | 2G | |
| executor_cores | No | ||
| num_executors | No |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
No arguments | |||