create_data_pipeline
Create a Data Pipeline in a Microsoft Fabric workspace with custom activities and dependencies. Orchestrate multi-stage workflows like Bronze → Silver → Gold for reliable data transformation.
Instructions
Create a Data Pipeline in a Fabric workspace with custom activities and dependencies.
Perfect for orchestrating medallion architecture workflows (Bronze → Silver → Gold).
Args:
pipeline_name: Name for the pipeline
pipeline_definition: Pipeline JSON definition with activities and dependencies
workspace: Workspace name or ID (optional, uses context if not provided)
description: Optional description for the pipeline
ctx: Context object
Returns:
Dictionary with pipeline details including ID and status
Example - Bronze to Silver to Gold cascade:
pipeline_definition = {
"properties": {
"activities": [
{
"name": "Bronze_Ingestion",
"type": "Notebook",
"typeProperties": {
"notebook": {"name": "bronze_ingest_notebook"}
},
"dependsOn": []
},
{
"name": "Silver_Transform",
"type": "Notebook",
"typeProperties": {
"notebook": {"name": "silver_transform_notebook"}
},
"dependsOn": [
{
"activity": "Bronze_Ingestion",
"dependencyConditions": ["Succeeded"]
}
]
},
{
"name": "Gold_Transform",
"type": "Notebook",
"typeProperties": {
"notebook": {"name": "gold_transform_notebook"}
},
"dependsOn": [
{
"activity": "Silver_Transform",
"dependencyConditions": ["Succeeded"]
}
]
}
]
}
}
create_data_pipeline(
pipeline_name="Medallion_ETL_Pipeline",
pipeline_definition=pipeline_definition,
workspace="PROD-Analytics",
description="Orchestrates Bronze → Silver → Gold transformations"
)
Input Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| workspace | No | ||
| description | No | ||
| pipeline_name | Yes | ||
| pipeline_definition | Yes |