Liao Jiayi Oct 16, 2022 #Flink#Failover#Apache Flink

How Flink failover works, why it limits some applications, and alternative recovery approaches.

Liao Jiayi Mar 13, 2022 #Feast#Streaming#Flink

A detailed look at Feast's architecture and core feature-store capabilities.

Liao Jiayi Feb 06, 2022 #Flink#Apache Flink

Flink's checkpoint semantics, their limitations, and production approaches to weakly consistent snapshots.

Liao Jiayi Jan 16, 2022 #Flink#RocksDB#Apache Flink

Why RocksDB, despite its strengths, is not an ideal match for large-scale streaming state in Apache Flink.

Liao Jiayi Jan 12, 2022 #Flink#Apache Flink

Notes on Hazelcast Jet's architecture, low-latency goals, and distributed state storage.

Liao Jiayi May 12, 2021 #Flink#Apache Flink

Custom state serializers can improve compatibility and performance in complex Flink applications.

Liao Jiayi Jun 12, 2020 #Flink#network#Apache Flink

An introduction to network buffers in Flink.

Liao Jiayi Mar 20, 2020 #Flink#StateBackend#Apache Flink StateBackend

RocksDBStateBackend stores large state in Flink.

Liao Jiayi Mar 04, 2020 #Flink#StateBackend#Apache Flink StateBackend

FsStateBackend may be the most commonly used StateBackend at many companies.

Liao Jiayi Feb 04, 2020 #Flink#StateBackend#Apache Flink StateBackend

DefaultOperatorStateBackend is a basic and relatively simple component of Flink's state backend.

Liao Jiayi Jan 31, 2020 #Flink#StateBackend#Apache Flink StateBackend

If I had to name one particularly impressive feature of Flink as a stream-processing engine, State and StateBackend would be it.

Liao Jiayi Dec 02, 2019 #Flink#Apache Flink

Since the existing StateBackend implementations only override KeyedStateBackend, all Operator State uses DefaultOperatorStateBackend for snapshots.

Liao Jiayi Dec 01, 2019 #Flink#Apache Flink

How to build a State Backend in Flink from scratch.

Liao Jiayi Oct 21, 2019 #Flink#Apache Flink

Understanding the interactions between Flink and HDFS helps clarify the problems HDFS may introduce into Flink.

Liao Jiayi Jun 20, 2019 #Flink#Apache Flink

Flink’s backpressure mechanism is often praised, but a production problem led me to discover that it is not perfect.

Liao Jiayi May 13, 2019 #Flink#Apache Flink

A recent proposal on Flink’s development mailing list refactors StreamTask. Reading it reminded me how elegantly simple a good solution can be.

Liao Jiayi Mar 23, 2019 #Flink#Watermark#Apache Flink

An introduction to Apache Flink’s watermark mechanism.

Liao Jiayi Mar 18, 2019 #Flink#Apache Flink

Apache Flink schedules resources differently for streaming and batch data. This week’s brief post introduces the two modes.

Liao Jiayi Mar 10, 2019 #Flink#meetup#Apache Flink

Lessons from GrowingIO's migration from Spark to Flink for real-time analytics, presented at the Beijing Flink Meetup.

Liao Jiayi Nov 24, 2018 #IoT#Flink#Apache Flink

An English rendering of my Chinese translation of the data Artisans article on stream processing and Apache Flink in IoT.

Liao Jiayi Nov 04, 2018 #Flink#Checkpoint#Apache Flink

Translated from the Data Artisans blog, this article compares Savepoints and Checkpoints in Apache Flink.

Liao Jiayi Oct 25, 2018 #Flink#Checkpoint#Apache Flink

Three approaches to initializing Apache Flink state when no usable Checkpoint or Savepoint exists.

Liao Jiayi Jul 06, 2018 #Flink#Apache Flink

Some notes on the Table APIs, which I had not used before.

Liao Jiayi Mar 20, 2018 #Flink#Exactly-once#Apache Flink

An introduction to Exactly Once semantics in Flink.