ds_complement_data
Backfill missing workflow data for specified date ranges or partitions, with options to start from selected tasks and control serial or parallel execution.
Instructions
Backfill (complement) workflow data for date range or single partition.
Recommended usage (v2.0.14):
Single partition: partition_date="2024-01-01" (clearer semantics, recommended)
Date range: start_date + end_date (multi-partition backfill)
From specific task: start_task_names + task_depend_type="TASK_POST"
Single task only: start_task_names + task_depend_type="TASK_ONLY"
⚠️ Important (v2.0.19):
Default mode: RUN_MODE_SERIAL (one partition at a time, safer)
Parallel mode: run_mode="RUN_MODE_PARALLEL" (multiple partitions concurrently, higher risk)
Before backfill: check dependency chain with ds_get_workflow → analyze upstream deps
Minimize scope: prefer start_task_names + TASK_POST to backfill from target task forward
Mandatory standard: don't backfill entire workflow unless full-chain rerun is explicitly needed
Args: project_name: Project name workflow_code: Process-definition code start_date: Start date yyyy-MM-dd (for range backfill) end_date: End date yyyy-MM-dd (for range backfill) partition_date: Single partition yyyy-MM-dd (alternative to start_date+end_date, recommended for clarity) start_task_names: Optional list of task names to start from (backfills these + their downstream tasks) task_depend_type: Dependency type (default TASK_POST: from start tasks forward; TASK_ONLY: start tasks only; TASK_PRE: start tasks + upstream) run_mode: RUN_MODE_SERIAL (default, one at a time) or RUN_MODE_PARALLEL (concurrent)
Serial ordering guarantee (v2.0.18): In RUN_MODE_SERIAL, the request is submitted using DS's continuous range fields complementStartDate / complementEndDate, so DS generates instances strictly in ascending day order. Discrete date-list formats (complementScheduleDateList / comma-separated) do not guarantee ordering and are only used as fallbacks. The chosen format is reported in the return value's "format" field ("date_range" / "comma_separated" / "json_list").
Examples: # Single partition (recommended) ds_complement_data(workflow_code=123, partition_date="2024-01-01")
# Date range (serial mode, safer)
ds_complement_data(workflow_code=123, start_date="2024-01-01", end_date="2024-01-31")
# From specific task forward (minimize scope)
ds_complement_data(workflow_code=123, partition_date="2024-01-01",
start_task_names=["ads_table"], task_depend_type="TASK_POST")
# Parallel mode (higher concurrency, more resource usage)
ds_complement_data(workflow_code=123, start_date="2024-01-01", end_date="2024-01-10",
run_mode="RUN_MODE_PARALLEL")
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| end_date | No | ||
| run_mode | No | RUN_MODE_SERIAL | |
| start_date | No | ||
| project_name | Yes | ||
| workflow_code | Yes | ||
| partition_date | No | ||
| start_task_names | No | ||
| task_depend_type | No | TASK_POST |