In 2015, Flink’s authors published Apache Flink: Stream and Batch Processing in a Single Engine. Guided by that paper, this post explains how Flink designs and implements a unified batch/streaming architecture.

Introduction

In Flink, batch/streaming unification usually refers to these four areas, with Runtime denoting the execution implementation.

Data Exchange Model

Flink has a unified execution model for streaming and batch jobs.

Each task’s output is wrapped in an IntermediateResult. Internally, jobs are not explicitly classified as streaming or batch; different IntermediateResult types instead express two broad exchange models, PIPELINED and BLOCKING.

Before examining exchange details, why avoid distinguishing job types? What is the benefit?

Suppose, as shown above, a batch job’s output must initialize a streaming job. Is the combined job batch or streaming?

Conventional categories cannot describe it neatly. The usual industry solution splits it into two jobs, running batch first and streaming second. That works, but creates substantial operational costs:

1. External storage is needed to manage the batch output.
2. A scheduler must support dependencies between batch and streaming jobs.

To execute this as one job, the engine must not bind a job rigidly to batch or streaming. Can Flink support it? First examine data exchange, then revisit its feasibility.

Consider the design of PIPELINED exchange:

RecordWriter places data into buffers and routes it to partitions according to key-routing rules. Each partition wraps its data in a reader and queues it for the Netty server, which sends it downstream.

The same design works in BLOCKING mode. A partition writes data to files, while its reader retains file handles. Downstream tasks are scheduled after upstream tasks finish. A downstream Netty client’s Partition Request activates the partition and reader to retrieve the data.

Scheduling Model

There are LAZY and EAGER scheduling modes. By default, streaming uses EAGER and batch uses LAZY.

EAGER

This is straightforward: streaming follows an all-or-nothing design, with all tasks running or none.

LAZY

LAZY schedules upstream tasks first, then downstream tasks after upstream data is produced or execution finishes, somewhat like Spark stages.

Region Scheduling

Neither EAGER nor LAZY can execute the mixed batch/streaming job described earlier. The community therefore proposed Region Scheduling to unify scheduling. First, what is a region?

For a join with two massive inputs, both inputs must be ready before the join runs. Its two input edges should therefore use BLOCKING exchange. These edges divide the job into regions, shown by the dashed lines.

With Region Scheduling, the dark streaming portion becomes one region, while the light batch portion is divided into several regions. The light regions feed the dark region, so the scheduling rules prioritize the earliest upstream regions.

Summary

The data-exchange and scheduling models boil down to two points:

1 Run batch jobs with the PIPELINED model

PIPELINED streaming and BLOCKING batch processing are unsurprising. PIPELINED batch execution mainly benefits real-time analysis. Spark usually spills at shuffles or aggregations and schedules stages sequentially, greatly reducing immediacy. PIPELINED execution can improve performance.

How can a join use PIPELINED exchange without spilling? Flink does spill, but not at the two input boundaries. It transfers both inputs into the join operator and spills only when memory is insufficient. At massive scale this is fundamentally similar to Spark, but moderate datasets that fit in memory benefit substantially.

2 Integrate some scheduling-system capabilities

Region Scheduling does not care whether work inside a region is batch or streaming; it cares about dependencies between regions. This lets some workflows previously split into multiple jobs execute as one, including the mixed batch/streaming example above.

References

• Stream and Batch in a Single Engine
• Towards Flink 2.0: Unified Batch & Stream Processing
• A Deep Dive into Flink’s Network Stack
• Pipelined Region Scheduling