If I had to name one particularly impressive feature of Flink as a stream-processing engine, State and StateBackend would be it.

Overview

In my view, a stateless distributed program running on Flink merely uses its distributed execution capabilities rather than truly taking advantage of the framework. Simple data import/export services that do not need fault tolerance are one example.

This overview introduces the existing types of Flink StateBackend and their components. Subsequent posts will explain each backend’s internals in increasing depth.

Components

A StateBackend usually consists of the following two parts:

• OperatorStateBackend
• KeyedStateBackend

OperatorStateBackend stores state in non-keyed scenarios, such as Kafka offsets. Common examples are ListState, UnionState, and BroadcastState. KeyedStateBackend stores keyed state, such as a user’s detailed activity within a 10-minute window. Common examples include ValueState, ListState, and MapState.

Both OperatorStateBackend and KeyedStateBackend fundamentally store state, so they share these capabilities:

• Restore state
• Modify state
• Persist state

The later discussions of individual StateBackends will follow these dimensions.

Types

KeyedStateBackend is generally accessed more frequently and holds more state than OperatorStateBackend, so it tends to receive more attention.

We usually refer to three types of StateBackend:

• MemoryStateBackend
• FsStateBackend
• RocksDBStateBackend

Their corresponding OperatorStateBackend and KeyedStateBackend implementations are:

StateBackend | OperatorStateBackend | KeyedStateBackend :-: | :-: | :-: | :-: | :-: MemoryStateBackend | DefaultOperatorStateBackend | HeapKeyedStateBackend FsStateBackend | DefaultOperatorStateBackend | HeapKeyedStateBackend RocksDBStateBackend | DefaultOperatorStateBackend | RocksDBKeyedStateBackend

You may now wonder how MemoryStateBackend differs from FsStateBackend. Later posts will explain this in detail. KeyedStateBackend defines how state is represented and serialized; StateBackend still controls where and how it is stored and how checkpoints operate. Thus both MemoryStateBackend and FsStateBackend keep state in memory, but differ in their checkpoint/savepoint behavior.

Here is a brief description of their use cases, consistent with the official blog. See the Ververica Blog for more details.

MemoryStateBackend

MemoryStateBackend is mainly for local debugging and can store small amounts of state, such as kilobytes of data.

FsStateBackend

FsStateBackend can be used in production and supports HA.

RocksDBStateBackend

RocksDBStateBackend is suitable for very large state and is the mainstream choice for many applications. It supports HA and incremental checkpoints.