Spark History MCP Server

{ "app_id": "application_1768524288842_0019", "skew_analysis": [ { "is_skewed": true, "skew_ratio": -1.0, "max_duration": -1.0, "median_duration": -1.0, "stage_id": 0 } ], "spill_analysis": [ { "has_spill": true, "total_disk_spill": 0, "total_memory_spill": 0, "stage_id": 0 }, { "has_spill": true, "total_disk_spill": 0, "total_memory_spill": 0, "stage_id": 1 } ], "resource_analysis": [], "partitioning_analysis": [], "join_analysis": [], "recommendations": [ { "category": "Configuration", "issue": "Increase the number of shuffle partitions to reduce the amount of data handled by each task in Stage 0, potentially mitigating skew.", "suggestion": "Set spark.sql.shuffle.partitions to 400", "evidence": "Current: 200 (default)", "impact_level": "High" }, { "category": "Configuration", "issue": "Set the number of default partitions same as shuffle partition for better data distribution, especially after shuffles like in stage 0. This can lead to more balanced tasks.", "suggestion": "Set spark.default.parallelism to 400", "evidence": "Current: Not explicitly set, defaults to number of cores on the cluster", "impact_level": "High" }, { "category": "Configuration", "issue": "Switching to Kryo serialization can improve performance and reduce memory usage during shuffles, especially if the data being shuffled contains custom objects. This may alleviate pressure on GC and reduce shuffle spill.", "suggestion": "Set spark.serializer to org.apache.spark.serializer.KryoSerializer", "evidence": "Current: org.apache.spark.serializer.JavaSerializer", "impact_level": "High" }, { "category": "Configuration", "issue": "Enable the external shuffle service. While this is more impactful in dynamic allocation scenarios, it can provide some stability and resource management benefits by isolating the shuffle service from the executors, potentially reducing executor memory pressure.", "suggestion": "Set spark.shuffle.service.enabled to true", "evidence": "Current: false (assumed, default is false in client mode)", "impact_level": "High" }, { "category": "Configuration", "issue": "Increase the fraction of JVM memory used for Spark execution and storage. This can help reduce disk spilling during shuffles, especially in Stage 0. However, be cautious not to set too high a value, potentially leading to OOM issues. Monitor GC.", "suggestion": "Set spark.memory.fraction to 0.7", "evidence": "Current: 0.6 (default)", "impact_level": "High" }, { "category": "Configuration", "issue": "Enable G1GC garbage collector and tune its parameters. G1GC is generally more efficient for large heaps and can reduce GC pauses. `InitiatingHeapOccupancyPercent` controls when GC starts (lower values trigger GC earlier). `DisableExplicitGC` prevents full GC triggered by code.", "suggestion": "Set spark.driver.extraJavaOptions to -Djava.net.preferIPv6Addresses=false -XX:+UseG1GC -XX:InitiatingHeapOccupancyPercent=35 -XX:+DisableExplicitGC", "evidence": "Current: -Djava.net.preferIPv6Addresses=false -XX:+IgnoreUnrecognizedVMOptions ...", "impact_level": "High" }, { "category": "Configuration", "issue": "Enable G1GC garbage collector and tune its parameters on the executor. `InitiatingHeapOccupancyPercent` controls when GC starts (lower values trigger GC earlier). `DisableExplicitGC` prevents full GC triggered by code.", "suggestion": "Set spark.executor.extraJavaOptions to -Djava.net.preferIPv6Addresses=false -XX:+UseG1GC -XX:InitiatingHeapOccupancyPercent=35 -XX:+DisableExplicitGC", "evidence": "Current: -Djava.net.preferIPv6Addresses=false -XX:+IgnoreUnrecognizedVMOptions ...", "impact_level": "High" }, { "category": "Configuration", "issue": "Enable Adaptive Query Execution. AQE might help in optimizing the query execution plan based on runtime statistics, potentially reducing skew and improving overall performance in Stage 0 and Stage 1.", "suggestion": "Set spark.sql.adaptive.enabled to true", "evidence": "Current: false (default)", "impact_level": "High" }, { "category": "Configuration", "issue": "Reduce the max size in flight for shuffle data. If spilling is occurring, try reducing the amount of data fetched per round to give the executors more room to process data.", "suggestion": "Set spark.reducer.maxSizeInFlight to 24m", "evidence": "Current: 48m (default)", "impact_level": "High" }, { "category": "Code", "issue": "Using `groupByKey` instead of `reduceByKey` or `aggregateByKey`.", "suggestion": "`groupByKey` shuffles all values across the network. Consider using `reduceByKey` if you're simply summing or aggregating values (e.g., `rdd.reduceByKey(lambda a, b: a + b)`) or `aggregateByKey` for more complex aggregations.", "evidence": "Line: 15", "impact_level": "Medium" }, { "category": "Code", "issue": "Hardcoded number of partitions. `spark.sparkContext.parallelize(data)` uses default parallelism", "suggestion": "Consider using `spark.sparkContext.parallelize(data, numSlices=...)` to explicitly control the number of partitions based on cluster size and data volume. This can improve parallelism and performance.", "evidence": "Line: 11", "impact_level": "Medium" }, { "category": "Code", "issue": "Performing `collect()` on potentially large dataset.", "suggestion": "`collect()` brings all the data to the driver. For large datasets, this can cause OOM errors. If you only need a sample, use `take(n)` or `takeSample(withReplacement, num, seed)`. If you need to process the entire dataset, consider writing to a distributed storage system (e.g., Parquet on S3).", "evidence": "Line: 18", "impact_level": "Medium" } ] }