Using Apache Flink as an example, this article explains why RocksDB is not an ideal streaming state store. In large production deployments, tuning and technical improvements still leave the combination short of expectations.

Background

RocksDB is an excellent key-value store. Years of Facebook development have made it stable and feature-rich enough for many mainstream workloads. It remains a natural embedded-KV choice, likely explaining Flink’s original selection for large keyed state. Newer stores such as Titan and TerarkDB address hardware and workload gaps, but are outside this discussion. Ordinary users may see few flaws, yet at the distributed-engine level and large production scale, RocksDB and streaming remain an imperfect combination.

RocksDB

RocksDB uses an LSM-tree. Append-style writes create files, and compaction removes duplicate, expired, and deleted key-value entries. SSTables sort entries by key, divide them into data blocks, and build index blocks from metadata for efficient reads.

SST files are organized into levels. Flushed data first enters L0. When compaction conditions are met, it moves to L1 and onward. Longer-lived data generally reaches higher levels.

Writing Data

Writes pass through serialization, an API call, the memtable, and persistence to SST files. By default, RocksDB maintains two configurable memtables. API writes synchronously enter the active memtable. When full or otherwise ready to flush, it becomes immutable, and a background thread sorts and deduplicates it into a new L0 SST file. If WAL is enabled, writes also synchronously enter the log.

Writes, including updates, are append-only in this model. Repeated writes to a key are merged during flush or compaction rather than updated in place. Deletes similarly appear as (Key -> DeleteType) records. The default memtable structure is a skip list.

Reading Data

SST characteristics differ by level:

• L0: Each file is internally sorted, but key ranges overlap across files. A key may appear in every L0 file.
• L1 through Ln: Compaction merges selected files into nonoverlapping key ranges. A key can occur in at most one file per level.

Data may reside in memtables, block cache, or SST files. There are two read types:

• Point lookup: Try memory first, then search SST levels. At L0, filter candidate files by key range. At later levels, binary-search to locate the relevant file.
• Range scan: Merge iterators over memtables, immutable memtables, L0 files, and higher-level files to produce ordered results.

Many optimizations apply, including per-file Bloom filters for point lookups and iterator read-ahead to reduce I/O. An SST file’s layout is shown below. Binary-searching the index block locates the required data block.

1
<beginning_of_file>
2
[data block 1]     // 具体的 KV 数据
3
[data block 2]    // 具体的 KV 数据
4
...
5
[data block N]    // 具体的 KV 数据
6
[meta block 1: filter block]            // Filter 信息，比如 bloom filter
7
[meta block 2: index block]              // data block 对应的 index，查询中通过对 index block 进行二分查找来定位到具体的 data block
8
... (compression/range deletion/stats block)
9
[meta block K: future extended block]
10
[metaindex block]
11
[Footer]
12
<end_of_file>

Compaction Strategies

Why compact? Merging files deduplicates keys and removes expired entries. Conceptually, read N files, reorganize them, and write the results again. Consider two extremes:

• Never compact: Files remain in L0 with overlapping ranges, so reads must inspect many files and perform poorly.
• Always compact: Merge every new SST with existing files, making writes extremely expensive.

Compaction balances read and write amplification, and should match the workload. RocksDB provides three strategies with complex trigger conditions described in the links below:

• Leveled compaction, the default: More frequent compaction, lower read amplification, higher write amplification.
• Universal compaction: Less frequent compaction, higher read amplification, lower write amplification.
• FIFO compaction: Almost no compaction, high read amplification, almost no write amplification.

Streaming Workloads and State Access

Consider how streaming state is used in Apache Flink.

Scenario 1: WordCount, counting each word within sixty-second windows.

For every record:

1. Locate its window from the word and timestamp.
2. Combine the word, window boundaries, and metadata such as KeyGroup into a RocksDB key and serialize it to byte[].
3. Read and deserialize the window’s intermediate result through RocksDB.
4. Update the result with the new word.
5. Serialize and write the updated result.

Scenario 2: Join stream A with stream B:

1
SELECT *
2
FROM a, b
3
WHERE a.id = b.id
4
AND a.time BETWEEN b.time - INTERVAL '4' HOUR AND b.time

For each A record, with B handled symmetrically:

1. Scan and deserialize B’s stored records.
2. Find matching records, join them, and emit downstream.
3. Read and deserialize A’s stored list, then append the new record.
4. Serialize and write A’s updated list.

For both windows and joins, state access concerns the time range relevant to current data. Windows access their own state; joins access the interval in their conditions. Unlike general web-service transactions, they do not operate equally across all historical data. This matches the observation that newer data is usually more valuable.

RocksDB as a State Store

With small state, spare resources can hide storage overhead. Large state or skew requires substantial overhead to maintain high streaming throughput.

Which Compaction Strategy?

Consider problems with leveled compaction:

Write amplification: Leveled compaction suits read-heavy workloads. Streaming records often produce multiple state reads and updates, with ratios near 1, as in tumbling-window aggregates. Frequent writes trigger compaction across levels. Every checkpoint also forces an L0 file, making default thresholds easy to reach.

Synchronized spikes: TaskManager CPU often spikes every four checkpoints, reducing throughput. The default L0->L1 trigger is four files, and tasks checkpoint around the same time, synchronizing compaction and competing with processing threads.

Traffic cycles: Higher traffic means more writes and compaction. Streaming traffic often differs severalfold between peaks and troughs. Compaction consumes the most resources precisely when processing needs them most, forcing larger allocations that sit underused off-peak.

Universal compaction improves matters somewhat but retains similar issues. Consider the minimal, apparently less useful FIFO compaction, described in the wiki:

1
FIFO compaction style is the simplest compaction strategy. It is suited for keeping event log
2
data with very low overhead (query log for example). It periodically deletes the old data, so it's
3
basically a TTL compaction style.
4

5
In FIFO compaction, all files are in level 0. When total size of the data exceeds configured size
6
 (CompactionOptionsFIFO::max_table_files_size), we delete the oldest table file. This means that
7
 write amplification of data is always 1 (in addition to WAL write amplification).

FIFO effectively maintains a stream of L0 SST files. Older data has lower priority and is more likely to expire, resembling streaming workloads. Too many L0 files hurt reads, though simple compaction options, Bloom filters, and caching can reduce file I/O.

I consider FIFO the closest fit for streaming among the three. However, it is not integrated with Flink’s semantics, including TTL alignment, and can lose data, so we do not recommend it to users. Custom RocksDB compaction APIs or source modifications could adapt it.

Embedded Storage and Distributed Computing

Embedded RocksDB instances are isolated per task, making a global view difficult. I compared embedded and distributed storage in the Hazelcast Jet article; here is another perspective.

For synchronized compaction, HBase-style jitter could spread operations over time and smooth output. Rescaling creates a related problem:

Suppose six key groups are initially assigned across three tasks as {1,2}, {3,4}, and {5,6}. Reducing parallelism to two reassigns them as {1,2,3} and {4,5,6}. This common streaming technique requires migration and merging among previously separate RocksDB instances. A single-machine store was not designed primarily for such rescaling, especially when recovery time is critical.

Resource Contention

RocksDB flush and compaction threads compete with processing threads for CPU. YARN or Kubernetes containers often have only a single-digit number of cores to balance other resources such as memory. Background storage work can therefore significantly affect processing. Low-priority compaction threads help little under continuous input, and falling too far behind causes write stalls that completely block task processing temporarily.

Other Issues

Other production problems and potential improvements include:

• Serialization: Read-Modify-Write requires deserialization on reads and serialization on writes. Complex user state, particularly in UDAFs, can make this a major bottleneck.
• Compression: Compression and decompression introduce similar overhead.
• Small files: Frequent updates generate small SST files.
• Time semantics: Rich semantics such as event time are absent.
• Retractions: Many deletes reduce scan and seek performance. ….

Summary

This article reviewed RocksDB and its mismatches with streaming workloads. Users can still choose RocksDBStateBackend for large state. I hope a better-fitting store, embedded or distributed, emerges as streaming adoption grows; someone is likely already building or preparing one.