Before Flink 1.10, RocksDB memory use was unmanaged. Last year, the community reworked Flink’s integration using RocksDB’s existing memory-control features; see FLINK-7289.

Note: This article assumes some familiarity with Flink’s RocksDBStateBackend.

RocksDB Memory Usage

RocksDB memory falls into several categories; see the official documentation:

• Memtable (write buffer): Receives writes before they are flushed to disk when specified conditions are met.
• Block cache: Caches read data in blocks.
• Indexes and filters: Locate the block containing a requested key.
• Blocks pinned by iterators: Iteration initializes blocks at several RocksDB levels to read in batches rather than fetching individual records from disk.

All use native memory. In Flink, memtables, block caches, indexes, and Bloom filters usually dominate. Before Flink 1.10, these were configured per column family: each state had a complete allocation, so memory grew linearly with the number of states until OOM.

RocksDB Memory in Flink

RocksDB often serves online storage workloads with tens of gigabytes of memory per instance. In distributed engines such as Flink, each instance has fewer resources, making strict memory control necessary to avoid container OOM termination.

Flink 1.9

Flink 1.9 offered only these relatively coarse controls:

These parameters control only memtables and block caches per column family. Other native-memory categories and total RocksDB memory remain uncontrolled.

Flink 1.11

Compared with 1.9, version 1.11 coordinates RocksDB memory within a process through WriteBufferManager and Block Cache. This relies on several RocksDB features:

• WriteBufferManager can use a cache as its accounting container, controlling write buffers and block-cache memory together.
• Indexes and filters can reside in the block cache. Partitioned Index Filters help prevent data-cache entries from evicting them, preserving lookup performance.

Memory control requires enabling managed memory or setting state.backend.rocksdb.memory.fixed-per-slot. Flink calculates cache sizes from the supplied RocksDB memory budget using these parameters:

You may notice that Blocks pinned by iterators has no dedicated configuration. Flink does not explicitly budget this category, and RocksDB bugs prevent enabling strict capacity limits. Flink therefore calculates somewhat smaller allocations and leaves headroom. The calculations below come from RocksDBMemoryControllerUtils.java#L64:

1
  /**
2
   * Calculate the actual memory capacity of cache, which would be shared among rocksDB instance(s).
3
   * We introduce this method because:
4
   * a) We cannot create a strict capacity limit cache util FLINK-15532 resolved.
5
   * b) Regardless of the memory usage of blocks pinned by RocksDB iterators,
6
   * which is difficult to calculate and only happened when we iterator entries in RocksDBMapState, the overuse of memory is mainly occupied by at most half of the write buffer usage.
7
   * (see <a href="https://github.com/dataArtisans/frocksdb/blob/958f191d3f7276ae59b270f9db8390034d549ee0/include/rocksdb/write_buffer_manager.h#L51">the flush implementation of write buffer manager</a>).
8
   * Thus, we have four equations below:
9
   *   write_buffer_manager_memory = 1.5 * write_buffer_manager_capacity
10
   *   write_buffer_manager_memory = total_memory_size * write_buffer_ratio
11
   *   write_buffer_manager_memory + other_part = total_memory_size
12
   *   write_buffer_manager_capacity + other_part = cache_capacity
13
   * And we would deduce the formula:
14
   *   cache_capacity =  (3 - write_buffer_ratio) * total_memory_size / 3
15
   *   write_buffer_manager_capacity = 2 * total_memory_size * write_buffer_ratio / 3
16
   */

Problems in the New Version

The newer version controls RocksDB memory better, but internally we disabled managed memory and therefore this entire mechanism, mainly from a user’s perspective.

(1) Inflated apparent memory usage encourages users to add memory blindly.

In Flink 1.9, although limits were imprecise, monitoring reflected actual consumption and made over- or under-allocation clear. In 1.11, most managed memory is occupied at startup, with no intuitive monitoring to show whether the configuration is appropriate or how to tune write-buffer-ratio.

A user may see usage rise from 50% to 80% with the same total allocation, even though the job does not need more memory. When off-heap memory looks nearly full, many users increase the allocation, only to see it mostly occupied again.

(2) Memory control remains imprecise, and OOM incidents become more likely and frequent for many jobs.

We encountered several issues with managed memory, including:

• FLINK-24120: Each new savepoint slightly increased off-heap memory. Investigation traced this to a glibc arena leak.
• RocksDB’s block cache still cannot enforce precise memory limits; see FLINK-15532. Deterministic guarantees remain unavailable.

References

• Write Buffer Manager
• Block Cache
• Partitioned Index Filters
• glibc memory leak
• FLINK-24120: Document MALLOC_ARENA_MAX as workaround for glibc memory leak
• FLINK-15532: Enable strict capacity limit for memory usage for RocksDB
• FLINK-7289: Memory allocation of RocksDB can be problematic in container environments