glue_s3
Analyze S3 data layouts for AWS Glue, generate optimization strategies, and consolidate small files using automated Glue jobs.
Instructions
Analyze and optimize S3 data layouts for AWS Glue.
Modes:
analyze
Analyze S3 data layout for optimization opportunities.
Parameters: s3_location (required), database_name (required),
table_name (required), data_format, query_patterns, data_size_gb
optimize
Generate comprehensive S3 optimization strategy.
Parameters: database_name (required), table_name (required),
s3_location (required), data_format, target_file_size_mb,
compression_type, enable_small_file_optimization, query_patterns
consolidate
Generate Glue job for small files consolidation.
Parameters: source_database (required), source_table (required),
target_database (required), target_table (required),
target_file_size_mb, consolidation_strategy
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| mode | Yes | ||
| table_name | No | ||
| data_format | No | parquet | |
| s3_location | No | ||
| data_size_gb | No | ||
| source_table | No | ||
| target_table | No | ||
| database_name | No | ||
| query_patterns | No | ||
| source_database | No | ||
| target_database | No | ||
| compression_type | No | snappy | |
| target_file_size_mb | No | ||
| consolidation_strategy | No | coalesce | |
| enable_small_file_optimization | No |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
No arguments | |||