GC Problems in Spark Streaming Under Light Workloads

A Spark troubleshooting case.

Spark Executor OOM: GC overhead limit exceeded

The job is simple: receive data from Kafka, pass it through a tagging operator, and write to Hive. Executor memory is set to 2 GB, with roughly 1 GB of Spark UnifiedMemory. A tagging-rule table of about 350 MB is passed through a closure (this should have used a broadcast variable, but our ETL framework’s broadcast mechanism was not suitable, so we passed it through the closure instead).

Because the job is lightweight, we allocated relatively little memory. A peculiar pattern emerged: with very large data volumes, GC time was normal; with small volumes, excessive GC time caused an OOM. This puzzled me. To analyze memory usage, I first enabled executor heap dumps to find out what was keeping the garbage collector so busy.

1
-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/path

Opening it in the JVM tool produced this result:

There were three large byte arrays of similar sizes. One was referenced by TorrentBroadcast, as expected. The other two were not referenced by any concrete object; they existed only as DeserializedMemoryEntry objects in Spark’s memory model. Since the program was very lightweight and needed little memory, this was strange. With no clear lead, I added GC parameters to inspect the collection process more closely:

1
-XX:+PrintGCdetails

The following GC logs then appeared (I am writing this at home and cannot be bothered to connect to the VPN for the real logs):

1
[Full GC (Metadata GC Threshold) ...
2
[Full GC (Ergonomics) ...

I first increased MetaspaceSize to avoid Full GCs caused by the Metadata GC Threshold. Then I focused on Full GCs triggered by Ergonomics. The official documentation says:

1
Ergonomics is the process by which the Java Virtual Machine (JVM) and garbage collection tuning, such as behavior-based tuning, improve application performance.
2

3
Example:
4
    UseAdaptiveSizePolicy actions to meet  *** throughput goal ***
5
                       GC overhead (%)
6
    Young generation:       16.10         (attempted to grow)
7
    Tenured generation:      4.67         (attempted to grow)
8
    Tenuring threshold:    (attempted to decrease to balance GC costs) = 1

In other words, based on historical memory behavior, the JVM dynamically adjusts parameters when it is too far from its goals. As shown above, reaching the throughput goal requires balancing GC overhead in the young and old generations. The JVM lowers the tenuring threshold, making it easier for young-generation objects to enter the tenured generation.

After several observations, I found that large objects repeatedly entered the old generation and caused frequent Full GCs: these were the two unexplained byte arrays in the heap dump. Connecting this with the earlier observation that GC overhead was normal at high data volumes but excessive at low volumes, I realized that passing the rule table in a closure put a large task-binary object into the old generation whenever a new task was deployed. With small data volumes, tasks finished quickly, sometimes faster than a GC cycle. An executor continuously handling small tasks therefore kept performing Full GCs, eventually causing an OOM.

References

The Unspoken - The Why of GC Ergonomics