Spark History MCP Server

{ "app_id": "application_1768320005356_0009", "skew_analysis": [ { "is_skewed": true, "skew_ratio": 0.0, "max_duration": 0.0, "median_duration": 0.0, "stage_id": 0 }, { "is_skewed": true, "skew_ratio": 0.0, "max_duration": 0.0, "median_duration": 0.0, "stage_id": 2 } ], "spill_analysis": [ { "has_spill": true, "total_disk_spill": 0, "total_memory_spill": 0, "stage_id": 0 }, { "has_spill": true, "total_disk_spill": 0, "total_memory_spill": 0, "stage_id": 2 } ], "resource_analysis": [], "partitioning_analysis": [], "join_analysis": [], "recommendations": [ { "category": "Configuration", "issue": "Executor memory is very low. This is a client mode job, but 512MB is extremely limiting. Increasing it to a more reasonable value like 2GB to give executors more room to breathe, reduce spill and GC overhead.", "suggestion": "Set spark.executor.memory to 2g", "evidence": "Current: 512m", "impact_level": "High" }, { "category": "Configuration", "issue": "Spark is configured to use very little memory for execution. The current setting of 0.1 will starve execution. Increase to 0.6 to make more memory available to Spark's execution engine, reducing the likelihood of spills to disk.", "suggestion": "Set spark.memory.fraction to 0.6", "evidence": "Current: 0.1", "impact_level": "High" }, { "category": "Configuration", "issue": "RDD compression is enabled and good practice. No change needed.", "suggestion": "Set spark.rdd.compress to True", "evidence": "Current: True", "impact_level": "High" }, { "category": "Configuration", "issue": "The JavaSerializer is the default, which is slow and can lead to larger serialized objects. KryoSerializer is generally faster and more compact. Enable and register custom classes if needed for maximum benefit. Consider also tuning `spark.kryo.referenceTracking` to true to avoid OOMs.", "suggestion": "Set spark.serializer to org.apache.spark.serializer.KryoSerializer", "evidence": "Current: org.apache.spark.serializer.JavaSerializer", "impact_level": "High" }, { "category": "Configuration", "issue": "When using Kryo, enable registration required. Registering classes ahead of time will improve serialization performance. If not registered, kryo must serialize class names, this degrades performance.", "suggestion": "Set spark.kryo.registrationRequired to true", "evidence": "Current: false", "impact_level": "High" }, { "category": "Configuration", "issue": "Adding garbage collection tuning for better performance. The G1GC is generally a good option for Spark. This is only a starting point and will need to be further tuned based on profiling and monitoring of the application. `ParallelRefProcEnabled` improves performance of Kryo by parallelizing reference processing. ", "suggestion": "Set spark.executor.extraJavaOptions to -Djava.net.preferIPv6Addresses=false -XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/jdk.internal.ref=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.cs=ALL-UNNAMED --add-opens=java.base/sun.security.action=ALL-UNNAMED --add-opens=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED -Djdk.reflect.useDirectMethodHandle=false -XX:+UseG1GC -XX:MaxGCPauseMillis=100 -XX:+ParallelRefProcEnabled", "evidence": "Current: -Djava.net.preferIPv6Addresses=false -XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/jdk.internal.ref=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.cs=ALL-UNNAMED --add-opens=java.base/sun.security.action=ALL-UNNAMED --add-opens=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED -Djdk.reflect.useDirectMethodHandle=false", "impact_level": "High" }, { "category": "Configuration", "issue": "Same as executor, applies the same G1GC settings to the driver.", "suggestion": "Set spark.driver.extraJavaOptions to -Djava.net.preferIPv6Addresses=false -XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/jdk.internal.ref=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.cs=ALL-UNNAMED --add-opens=java.base/sun.security.action=ALL-UNNAMED --add-opens=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED -Djdk.reflect.useDirectMethodHandle=false -XX:+UseG1GC -XX:MaxGCPauseMillis=100 -XX:+ParallelRefProcEnabled", "evidence": "Current: -Djava.net.preferIPv6Addresses=false -XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/jdk.internal.ref=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.cs=ALL-UNNAMED --add-opens=java.base/sun.security.action=ALL-UNNAMED --add-opens=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED -Djdk.reflect.useDirectMethodHandle=false", "impact_level": "High" }, { "category": "Configuration", "issue": "Given limited memory, Spark will spill to disk, but this can be mitigated with the above recommendations, which reduces overall impact. Reducing spilling will increase overall performance. When memory is optimized, spilling is a negative performance implication.", "suggestion": "Set spark.shuffle.spill to false", "evidence": "Current: true", "impact_level": "High" }, { "category": "Code", "issue": "Creating very large strings in each row significantly increases the size of the data and exacerbates memory pressure, leading to potential spills.", "suggestion": "If possible, reduce the string length. Consider using a smaller, more efficient data structure or using a different data generation strategy if the string length is not crucial.", "evidence": "Line: df = spark.range(0, 2000000).withColumn(\"str\", F.expr(\"repeat('a', 1000)\"))", "impact_level": "Medium" }, { "category": "Code", "issue": "orderBy() on a large DataFrame with large string columns is likely to cause significant shuffle and spill to disk, especially with the limited executor memory.", "suggestion": "If only top N values are needed, use a topN based sort with broadcast join, and consider using a sampling-based approach if a complete sort is unnecessary. If a full sort is necessary, and the string column can't be reduced in size, consider increasing the executor memory. Also consider using `spark.sql.shuffle.partitions` config to scale shuffle", "evidence": "Line: sorted_df = df.orderBy(\"str\")", "impact_level": "Medium" }, { "category": "Code", "issue": "The `count()` action forces the execution of all preceding transformations, including the shuffle from the `orderBy()` operation, which can be expensive. Since a `count` is run after `orderBy` the shuffle is likely going to spill.", "suggestion": "Be mindful of unnecessary `count()` operations. If possible, avoid them and choose more efficient alternatives, like writing out a sample of the data to diagnose shuffle performance problems.", "evidence": "Line: print(f\"Count: {sorted_df.count()}\")", "impact_level": "Medium" } ] }