refactor
Replace duplicate PySpark patterns with utility functions, extract common code, and generate optimized data pipeline structures from sources and processing requirements.
Instructions
Deprecated. Prefer review. Still registered this minor version.
Refactor PySpark code and generate pipeline structures.
Modes:
patterns
Refactor code by replacing duplicate patterns with utility function calls.
Parameters: original_code (required), code_samples (required)
utilities
Extract common utility functions from code patterns.
Parameters: code_samples (required), patterns
pipeline
Generate optimized PySpark data pipeline code or project structure.
Parameters (pipeline): data_sources (required list), processing_requirements
(required), target_format, include_monitoring
Parameters (project): sql_content, workspace_name, workspace_path,
output_dir, include_glue_template, dialect, include_batch_processing,
include_visualization
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| mode | Yes | ||
| dialect | No | ||
| patterns | No | ||
| output_dir | No | ||
| sql_content | No | ||
| code_samples | No | ||
| data_sources | No | ||
| original_code | No | ||
| target_format | No | delta | |
| workspace_name | No | ||
| workspace_path | No | ||
| include_monitoring | No | ||
| include_glue_template | No | ||
| include_visualization | No | ||
| processing_requirements | No | ||
| include_batch_processing | No |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
No arguments | |||