Liao Jiayi Oct 16, 2022 #Flink#Failover#Apache Flink

How Flink failover works, why it limits some applications, and alternative recovery approaches.

Liao Jiayi Feb 06, 2022 #Flink#Apache Flink

Flink's checkpoint semantics, their limitations, and production approaches to weakly consistent snapshots.

Liao Jiayi Jan 16, 2022 #Flink#RocksDB#Apache Flink

Why RocksDB, despite its strengths, is not an ideal match for large-scale streaming state in Apache Flink.

Liao Jiayi Jan 12, 2022 #Flink#Apache Flink

Notes on Hazelcast Jet's architecture, low-latency goals, and distributed state storage.

Liao Jiayi May 12, 2021 #Flink#Apache Flink

Custom state serializers can improve compatibility and performance in complex Flink applications.

Liao Jiayi Apr 17, 2021 #Apache Flink

How Flink improved control of RocksDB memory, and the operational problems we encountered with managed memory.

Liao Jiayi Sep 22, 2020 #Apache Flink

How ByteDance improved Flink recovery so a failed task does not interrupt the entire real-time recommendation pipeline.

Liao Jiayi Sep 13, 2020 #Stream processing#streaming#Apache Flink

Thoughts inspired by the SIGMOD paper Beyond Analytics: The Evolution of Streaming Processing Systems.

Liao Jiayi Aug 02, 2020 #Apache Flink

Guided by the 2015 paper Apache Flink: Stream and Batch Processing in a Single Engine, this article explores Flink’s unified architecture.

Liao Jiayi Jun 12, 2020 #Flink#network#Apache Flink

An introduction to network buffers in Flink.

Liao Jiayi Dec 02, 2019 #Flink#Apache Flink

Since the existing StateBackend implementations only override KeyedStateBackend, all Operator State uses DefaultOperatorStateBackend for snapshots.

Liao Jiayi Dec 01, 2019 #Flink#Apache Flink

How to build a State Backend in Flink from scratch.

Liao Jiayi Oct 21, 2019 #Flink#Apache Flink

Understanding the interactions between Flink and HDFS helps clarify the problems HDFS may introduce into Flink.

Liao Jiayi Sep 09, 2019 #Apache Flink

Apache Flink uses Akka for communication between its Dispatcher, JobMaster, ResourceManager, and TaskExecutor components.

Liao Jiayi Jul 23, 2019 #Database#Apache Flink

RocksDB is a popular lightweight embedded key-value database. Flink uses it in RocksDBStateBackend to store large state.

Liao Jiayi Jun 20, 2019 #Flink#Apache Flink

Flink’s backpressure mechanism is often praised, but a production problem led me to discover that it is not perfect.

Liao Jiayi May 13, 2019 #Flink#Apache Flink

A recent proposal on Flink’s development mailing list refactors StreamTask. Reading it reminded me how elegantly simple a good solution can be.

Liao Jiayi Mar 23, 2019 #Flink#Watermark#Apache Flink

An introduction to Apache Flink’s watermark mechanism.

Liao Jiayi Mar 18, 2019 #Flink#Apache Flink

Apache Flink schedules resources differently for streaming and batch data. This week’s brief post introduces the two modes.

Liao Jiayi Mar 10, 2019 #Flink#meetup#Apache Flink

Lessons from GrowingIO's migration from Spark to Flink for real-time analytics, presented at the Beijing Flink Meetup.

Liao Jiayi Nov 24, 2018 #IoT#Flink#Apache Flink

An English rendering of my Chinese translation of the data Artisans article on stream processing and Apache Flink in IoT.

Liao Jiayi Nov 04, 2018 #Flink#Checkpoint#Apache Flink

Translated from the Data Artisans blog, this article compares Savepoints and Checkpoints in Apache Flink.

Liao Jiayi Oct 25, 2018 #Flink#Checkpoint#Apache Flink

Three approaches to initializing Apache Flink state when no usable Checkpoint or Savepoint exists.

Liao Jiayi Jul 06, 2018 #Flink#Apache Flink

Some notes on the Table APIs, which I had not used before.

Liao Jiayi Mar 20, 2018 #Flink#Exactly-once#Apache Flink

An introduction to Exactly Once semantics in Flink.