Spark History MCP Server

{ "app_id": "application_1768320005356_0008", "skew_analysis": [ { "is_skewed": true, "skew_ratio": 0.0, "max_duration": 0.0, "median_duration": 0.0, "stage_id": 0 }, { "is_skewed": true, "skew_ratio": 0.0, "max_duration": 0.0, "median_duration": 0.0, "stage_id": 2 } ], "spill_analysis": [ { "has_spill": true, "total_disk_spill": 0, "total_memory_spill": 0, "stage_id": 0 }, { "has_spill": true, "total_disk_spill": 0, "total_memory_spill": 0, "stage_id": 2 }, { "has_spill": true, "total_disk_spill": 0, "total_memory_spill": 0, "stage_id": 5 } ], "resource_analysis": [], "partitioning_analysis": [], "join_analysis": [], "recommendations": [ { "category": "Code", "issue": "Hardcoded number of partitions.", "suggestion": "Replace `repartition(10)` with a dynamic approach based on the input data size or cluster configuration. Consider using `repartition(numPartitions=spark.sparkContext.defaultParallelism)` or calculate the number of partitions based on data size (e.g., 1 partition per 128MB of uncompressed data). This can be determined from the `spark.read.load().inputFiles()` if source is accessible.", "evidence": "Line: df = spark.createDataFrame(...).repartition(10)", "impact_level": "Medium" }, { "category": "Code", "issue": "Use of `collect()` on a grouped DataFrame.", "suggestion": "`collect()` brings the entire result set to the driver, which can lead to OutOfMemoryErrors, especially with skewed data. If you only need a sample of the results or aggregations use `take()` or aggregate the counts. If you need all results and memory isn't an issue, then this might be acceptable, but consider the size of the results from the `groupBy` operation before using `collect()`. Consider debugging with a `limit()` or `show()` before applying `collect()` to confirm memory usage, and the skew impacts.", "evidence": "Line: grouped.collect()", "impact_level": "Medium" } ] }