refactor
Replace duplicate patterns in PySpark code with utility functions, extract common utilities, and generate optimized data pipelines.
Instructions
Refactor PySpark code and generate pipeline structures.
Modes:
patterns
Refactor code by replacing duplicate patterns with utility function calls.
Parameters: original_code (required), code_samples (required)
utilities
Extract common utility functions from code patterns.
Parameters: code_samples (required), patterns
pipeline
Generate optimized PySpark data pipeline code or project structure.
Parameters (pipeline): data_sources (required list), processing_requirements
(required), target_format, include_monitoring
Parameters (project): sql_content, workspace_name, workspace_path,
output_dir, include_glue_template, dialect, include_batch_processing,
include_visualization
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| mode | Yes | ||
| dialect | No | ||
| patterns | No | ||
| output_dir | No | ||
| sql_content | No | ||
| code_samples | No | ||
| data_sources | No | ||
| original_code | No | ||
| target_format | No | delta | |
| workspace_name | No | ||
| workspace_path | No | ||
| include_monitoring | No | ||
| include_glue_template | No | ||
| include_visualization | No | ||
| processing_requirements | No | ||
| include_batch_processing | No |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
No arguments | |||