Notes on data infrastructure, AI systems, robotics, and the industries behind the technology.https://www.liaojiayi.com/enhttps://www.liaojiayi.com/en/blog/data-taste/https://www.liaojiayi.com/en/blog/data-taste/What does good data taste mean for large language models, from collection and selection to training, quality, and evaluation?Sun, 30 Aug 2026 00:00:00 GMThttps://www.liaojiayi.com/en/blog/kimi-data/https://www.liaojiayi.com/en/blog/kimi-data/Notes on the data-processing techniques in the Kimi K1.5, K2, and K3 technical reports.Sun, 23 Aug 2026 00:00:00 GMThttps://www.liaojiayi.com/en/blog/llm-text-dedup/https://www.liaojiayi.com/en/blog/llm-text-dedup/Deduplicating LLM documents well at scale is a complex problem.Sun, 23 Aug 2026 00:00:00 GMThttps://www.liaojiayi.com/en/blog/continual-learning/https://www.liaojiayi.com/en/blog/continual-learning/Continual learning lets models absorb feedback during use and keep improving, adding another path to intelligence beyond pretraining and post-training/RL.Sat, 15 Aug 2026 00:00:00 GMThttps://www.liaojiayi.com/en/blog/rec-llm/https://www.liaojiayi.com/en/blog/rec-llm/New wine bottles for familiar ideas: how can LLM techniques systematically reshape recommendation systems?Thu, 19 Mar 2026 00:00:00 GMThttps://www.liaojiayi.com/en/blog/papers/https://www.liaojiayi.com/en/blog/papers/Notes on papers I have read.Thu, 05 Feb 2026 00:00:00 GMThttps://www.liaojiayi.com/en/blog/rl/https://www.liaojiayi.com/en/blog/rl/Exploring reinforcement-learning concepts and infrastructure.Mon, 02 Feb 2026 00:00:00 GMThttps://www.liaojiayi.com/en/blog/sglang/https://www.liaojiayi.com/en/blog/sglang/SGLang notesWed, 28 Jan 2026 00:00:00 GMThttps://www.liaojiayi.com/en/blog/mit_robot/https://www.liaojiayi.com/en/blog/mit_robot/A few notesMon, 26 Jan 2026 00:00:00 GMThttps://www.liaojiayi.com/en/blog/lerobot/https://www.liaojiayi.com/en/blog/lerobot/Notes on LeRobot datasets, training, and policies.Sat, 24 Jan 2026 00:00:00 GMThttps://www.liaojiayi.com/en/blog/video-image-pipeline/https://www.liaojiayi.com/en/blog/video-image-pipeline/Image and Video Processing PipelinesTue, 20 Jan 2026 00:00:00 GMThttps://www.liaojiayi.com/en/blog/robot-libero/https://www.liaojiayi.com/en/blog/robot-libero/Notes from exploring the LIBERO robotics dataset.Sun, 18 Jan 2026 00:00:00 GMThttps://www.liaojiayi.com/en/blog/llm_eval/https://www.liaojiayi.com/en/blog/llm_eval/Inside a General-Purpose LLM Evaluation FrameworkSun, 28 Dec 2025 00:00:00 GMThttps://www.liaojiayi.com/en/blog/tensor/https://www.liaojiayi.com/en/blog/tensor/PyTorch Tensor TransformationsFri, 26 Dec 2025 00:00:00 GMThttps://www.liaojiayi.com/en/blog/vllm/https://www.liaojiayi.com/en/blog/vllm/KV caching is essential to vLLM inference. Growing contexts drive frequent architectural changes, including prefill/decode separation and new cache storage options.Sat, 08 Nov 2025 00:00:00 GMThttps://www.liaojiayi.com/en/blog/deepspeed/https://www.liaojiayi.com/en/blog/deepspeed/How ZeRO reduces redundant GPU memory through sharding, with a walkthrough of DeepSpeed's initialization and training flow.Tue, 22 Jul 2025 00:00:00 GMThttps://www.liaojiayi.com/en/blog/frequent-use/https://www.liaojiayi.com/en/blog/frequent-use/Frequently Used ToolsSat, 11 Jan 2025 00:00:00 GMThttps://www.liaojiayi.com/en/blog/flink-jm-tm/https://www.liaojiayi.com/en/blog/flink-jm-tm/How Flink failover works, why it limits some applications, and alternative recovery approaches.Sun, 16 Oct 2022 00:00:00 GMThttps://www.liaojiayi.com/en/blog/lake-iceberg/https://www.liaojiayi.com/en/blog/lake-iceberg/An exploration of Apache Iceberg's file organization, snapshots, deletes, schema evolution, and how its approach differs from Hudi.Sun, 29 May 2022 00:00:00 GMThttps://www.liaojiayi.com/en/blog/lake-hudi/https://www.liaojiayi.com/en/blog/lake-hudi/An overview of Hudi's design, timeline, table types, indexes, upserts, and incremental reads.Thu, 05 May 2022 00:00:00 GMThttps://www.liaojiayi.com/en/blog/feast/https://www.liaojiayi.com/en/blog/feast/A detailed look at Feast's architecture and core feature-store capabilities.Sun, 13 Mar 2022 00:00:00 GMThttps://www.liaojiayi.com/en/blog/ray-dataset/https://www.liaojiayi.com/en/blog/ray-dataset/Ray 1.8+ introduced Datasets to address the last mile before model training. These notes organize the concepts, based mainly on reading rather than extensive hands-on use.Thu, 10 Mar 2022 00:00:00 GMThttps://www.liaojiayi.com/en/blog/streaming-fault-tolerance/https://www.liaojiayi.com/en/blog/streaming-fault-tolerance/Flink's checkpoint semantics, their limitations, and production approaches to weakly consistent snapshots.Sun, 06 Feb 2022 00:00:00 GMThttps://www.liaojiayi.com/en/blog/why-not-rocksdb-in-streaming-state/https://www.liaojiayi.com/en/blog/why-not-rocksdb-in-streaming-state/Why RocksDB, despite its strengths, is not an ideal match for large-scale streaming state in Apache Flink.Sun, 16 Jan 2022 00:00:00 GMThttps://www.liaojiayi.com/en/blog/Hazelcast-jet/https://www.liaojiayi.com/en/blog/Hazelcast-jet/Notes on Hazelcast Jet's architecture, low-latency goals, and distributed state storage.Wed, 12 Jan 2022 00:00:00 GMThttps://www.liaojiayi.com/en/blog/state-serializer/https://www.liaojiayi.com/en/blog/state-serializer/Custom state serializers can improve compatibility and performance in complex Flink applications.Wed, 12 May 2021 00:00:00 GMThttps://www.liaojiayi.com/en/blog/rocket-memory/https://www.liaojiayi.com/en/blog/rocket-memory/How Flink improved control of RocksDB memory, and the operational problems we encountered with managed memory.Sat, 17 Apr 2021 00:00:00 GMThttps://www.liaojiayi.com/en/blog/work-priority/https://www.liaojiayi.com/en/blog/work-priority/Technical skills determine whether work can be done; workplace soft skills help determine whether it succeeds. Reflections on prioritization from my own experience.Wed, 17 Mar 2021 00:00:00 GMThttps://www.liaojiayi.com/en/blog/growth/https://www.liaojiayi.com/en/blog/growth/Reposted from joining-hypergrowth-startups. I strongly agree with much of it and have personally experienced how these traits affect individuals, teams, and entire companies.Thu, 28 Jan 2021 00:00:00 GMThttps://www.liaojiayi.com/en/blog/why-need-datalake/https://www.liaojiayi.com/en/blog/why-need-datalake/Growing interest in data lakes is driven by a qualitative change in user needs that traditional data warehouses and their components can no longer satisfy.Sun, 15 Nov 2020 00:00:00 GMThttps://www.liaojiayi.com/en/blog/single-task-failover/https://www.liaojiayi.com/en/blog/single-task-failover/How ByteDance improved Flink recovery so a failed task does not interrupt the entire real-time recommendation pipeline.Tue, 22 Sep 2020 00:00:00 GMThttps://www.liaojiayi.com/en/blog/stream-trend/https://www.liaojiayi.com/en/blog/stream-trend/Thoughts inspired by the SIGMOD paper Beyond Analytics: The Evolution of Streaming Processing Systems.Sun, 13 Sep 2020 00:00:00 GMThttps://www.liaojiayi.com/en/blog/recommendations/https://www.liaojiayi.com/en/blog/recommendations/Notes from a book I took a long time to finish.Wed, 12 Aug 2020 00:00:00 GMThttps://www.liaojiayi.com/en/blog/flink-unify/https://www.liaojiayi.com/en/blog/flink-unify/Guided by the 2015 paper Apache Flink: Stream and Batch Processing in a Single Engine, this article explores Flink’s unified architecture.Sun, 02 Aug 2020 00:00:00 GMThttps://www.liaojiayi.com/en/blog/sushimin/https://www.liaojiayi.com/en/blog/sushimin/I finished this book during a holiday trip to Guangzhou. Part memoir and part reflection on the lessons from different experiences, it is fairly candid and well worth reading.Mon, 20 Jul 2020 00:00:00 GMThttps://www.liaojiayi.com/en/blog/flink-network-buffer/https://www.liaojiayi.com/en/blog/flink-network-buffer/An introduction to network buffers in Flink.Fri, 12 Jun 2020 00:00:00 GMThttps://www.liaojiayi.com/en/blog/hudi/https://www.liaojiayi.com/en/blog/hudi/Notes on Apache Hudi, the open-source data lake framework led by Uber and shaped by its data-processing needs.Thu, 14 May 2020 00:00:00 GMThttps://www.liaojiayi.com/en/blog/flink-statebackend-rocksdbstatebackend/https://www.liaojiayi.com/en/blog/flink-statebackend-rocksdbstatebackend/RocksDBStateBackend stores large state in Flink.Fri, 20 Mar 2020 00:00:00 GMThttps://www.liaojiayi.com/en/blog/flink-statebackend-fsstatebackend/https://www.liaojiayi.com/en/blog/flink-statebackend-fsstatebackend/FsStateBackend may be the most commonly used StateBackend at many companies.Wed, 04 Mar 2020 00:00:00 GMThttps://www.liaojiayi.com/en/blog/flink-statebackend-operatorstatebackend/https://www.liaojiayi.com/en/blog/flink-statebackend-operatorstatebackend/DefaultOperatorStateBackend is a basic and relatively simple component of Flink's state backend.Tue, 04 Feb 2020 00:00:00 GMThttps://www.liaojiayi.com/en/blog/flink-statebackend-overview/https://www.liaojiayi.com/en/blog/flink-statebackend-overview/If I had to name one particularly impressive feature of Flink as a stream-processing engine, State and StateBackend would be it.Fri, 31 Jan 2020 00:00:00 GMThttps://www.liaojiayi.com/en/blog/ray/https://www.liaojiayi.com/en/blog/ray/Notes on Ray, a distributed framework designed for reinforcement learning and emerging AI workloads.Fri, 03 Jan 2020 00:00:00 GMThttps://www.liaojiayi.com/en/blog/blink-opt-1-count-distinct/https://www.liaojiayi.com/en/blog/blink-opt-1-count-distinct/Stream processing often encounters data skew. A common solution salts keys to redistribute records more evenly, then computes metrics through two-stage aggregation.Wed, 01 Jan 2020 00:00:00 GMThttps://www.liaojiayi.com/en/blog/calcite-paper/https://www.liaojiayi.com/en/blog/calcite-paper/I have been building a SQL engine and took the opportunity to reread the Calcite paper. Documentation on Calcite is frustratingly sparse.Wed, 01 Jan 2020 00:00:00 GMThttps://www.liaojiayi.com/en/blog/deep-dive-cep/https://www.liaojiayi.com/en/blog/deep-dive-cep/The architecture of rule-engine services and the implementation of complex event processing in Flink.Wed, 01 Jan 2020 00:00:00 GMThttps://www.liaojiayi.com/en/blog/flink-statebackend2/https://www.liaojiayi.com/en/blog/flink-statebackend2/Since the existing StateBackend implementations only override KeyedStateBackend, all Operator State uses DefaultOperatorStateBackend for snapshots.Mon, 02 Dec 2019 00:00:00 GMThttps://www.liaojiayi.com/en/blog/flink-statebackend/https://www.liaojiayi.com/en/blog/flink-statebackend/How to build a State Backend in Flink from scratch.Sun, 01 Dec 2019 00:00:00 GMThttps://www.liaojiayi.com/en/blog/flink-hdfs/https://www.liaojiayi.com/en/blog/flink-hdfs/Understanding the interactions between Flink and HDFS helps clarify the problems HDFS may introduce into Flink.Mon, 21 Oct 2019 00:00:00 GMThttps://www.liaojiayi.com/en/blog/kryo/https://www.liaojiayi.com/en/blog/kryo/Kryo can serialize Java classes that do not implement Serializable in various ways, but deserializing certain objects, such as those without a no-argument constructor, can produce this error.Sun, 22 Sep 2019 00:00:00 GMThttps://www.liaojiayi.com/en/blog/flink-rpc/https://www.liaojiayi.com/en/blog/flink-rpc/Apache Flink uses Akka for communication between its Dispatcher, JobMaster, ResourceManager, and TaskExecutor components.Mon, 09 Sep 2019 00:00:00 GMThttps://www.liaojiayi.com/en/blog/flink-cep/https://www.liaojiayi.com/en/blog/flink-cep/A source-code-oriented introduction to Flink CEP, revisiting my earlier notes while preparing an online walkthrough.Sun, 04 Aug 2019 00:00:00 GMThttps://www.liaojiayi.com/en/blog/rocksdb/https://www.liaojiayi.com/en/blog/rocksdb/RocksDB is a popular lightweight embedded key-value database. Flink uses it in RocksDBStateBackend to store large state.Tue, 23 Jul 2019 00:00:00 GMThttps://www.liaojiayi.com/en/blog/amazon-principles/https://www.liaojiayi.com/en/blog/amazon-principles/This was the first course I bought on Geek Time. These are simply my notes.Thu, 11 Jul 2019 00:00:00 GMThttps://www.liaojiayi.com/en/blog/flink-network-stack-opt/https://www.liaojiayi.com/en/blog/flink-network-stack-opt/Flink’s backpressure mechanism is often praised, but a production problem led me to discover that it is not perfect.Thu, 20 Jun 2019 00:00:00 GMThttps://www.liaojiayi.com/en/blog/disruptor/https://www.liaojiayi.com/en/blog/disruptor/Disruptor is a lock-free message queue introduced by LMAX in 2011. Its 45-minute presentation contains many ideas worth exploring.Sun, 16 Jun 2019 00:00:00 GMThttps://www.liaojiayi.com/en/blog/geek-linux/https://www.liaojiayi.com/en/blog/geek-linux/Summaries and reflections from the Geek Time course on Linux performance optimization.Mon, 27 May 2019 00:00:00 GMThttps://www.liaojiayi.com/en/blog/streamtask/https://www.liaojiayi.com/en/blog/streamtask/A recent proposal on Flink’s development mailing list refactors StreamTask. Reading it reminded me how elegantly simple a good solution can be.Mon, 13 May 2019 00:00:00 GMThttps://www.liaojiayi.com/en/blog/jvm-opt/https://www.liaojiayi.com/en/blog/jvm-opt/A Spark troubleshooting case involving a simple job that reads from Kafka, applies tagging operators, and writes to Hive.Fri, 12 Apr 2019 00:00:00 GMThttps://www.liaojiayi.com/en/blog/spark-hive-dynamic/https://www.liaojiayi.com/en/blog/spark-hive-dynamic/Hive tables are often partitioned by time and populated through SparkSQL dynamic-partition inserts. This reveals an opportunity for optimization.Fri, 12 Apr 2019 00:00:00 GMThttps://www.liaojiayi.com/en/blog/note-influence-without-authority/https://www.liaojiayi.com/en/blog/note-influence-without-authority/Influence Without AuthorityThu, 11 Apr 2019 00:00:00 GMThttps://www.liaojiayi.com/en/blog/hbase-tx/https://www.liaojiayi.com/en/blog/hbase-tx/A discussion of HBase’s transaction mechanisms.Tue, 02 Apr 2019 00:00:00 GMThttps://www.liaojiayi.com/en/blog/flink-watermark/https://www.liaojiayi.com/en/blog/flink-watermark/An introduction to Apache Flink’s watermark mechanism.Sat, 23 Mar 2019 00:00:00 GMThttps://www.liaojiayi.com/en/blog/Flink-Schedule-Mode/https://www.liaojiayi.com/en/blog/Flink-Schedule-Mode/Apache Flink schedules resources differently for streaming and batch data. This week’s brief post introduces the two modes.Mon, 18 Mar 2019 00:00:00 GMThttps://www.liaojiayi.com/en/blog/flink-meetup/https://www.liaojiayi.com/en/blog/flink-meetup/Lessons from GrowingIO's migration from Spark to Flink for real-time analytics, presented at the Beijing Flink Meetup.Sun, 10 Mar 2019 00:00:00 GMThttps://www.liaojiayi.com/en/blog/aerospike-evict/https://www.liaojiayi.com/en/blog/aerospike-evict/In real-time applications, eviction in systems such as Aerospike and Redis helps prevent storage failures caused by traffic spikes.Sun, 03 Mar 2019 00:00:00 GMThttps://www.liaojiayi.com/en/blog/CEP-In-Flink-4/https://www.liaojiayi.com/en/blog/CEP-In-Flink-4/Bottlenecks encountered when using the Apache Flink CEP module.Wed, 27 Feb 2019 00:00:00 GMThttps://www.liaojiayi.com/en/blog/CEP-In-Flink-3/https://www.liaojiayi.com/en/blog/CEP-In-Flink-3/How matched events are extracted in Apache Flink’s CEP module.Wed, 20 Feb 2019 00:00:00 GMThttps://www.liaojiayi.com/en/blog/photo-website/https://www.liaojiayi.com/en/blog/photo-website/An introduction to stock photography platforms including 500px, DFIC, and ZCOOL Hellorf.Mon, 18 Feb 2019 00:00:00 GMThttps://www.liaojiayi.com/en/blog/CEP-In-Flink-2/https://www.liaojiayi.com/en/blog/CEP-In-Flink-2/How Apache Flink matches CEP rules.Thu, 14 Feb 2019 00:00:00 GMThttps://www.liaojiayi.com/en/blog/Spark-WeakReference/https://www.liaojiayi.com/en/blog/Spark-WeakReference/A Stack Overflow question about when Spark reclaims Accumulator and Broadcast variables led me to explore how ContextCleaner uses WeakReference and a reference queue to clean up unused objects.Tue, 12 Feb 2019 00:00:00 GMThttps://www.liaojiayi.com/en/blog/scuba-junkie-1/https://www.liaojiayi.com/en/blog/scuba-junkie-1/A travelogue of a wonderful trip diving and earning my certification at Scuba Junkie.Mon, 04 Feb 2019 00:00:00 GMThttps://www.liaojiayi.com/en/blog/CEP-In-Flink-1/https://www.liaojiayi.com/en/blog/CEP-In-Flink-1/How Apache Flink parses CEP rules.Fri, 18 Jan 2019 00:00:00 GMThttps://www.liaojiayi.com/en/blog/deco/https://www.liaojiayi.com/en/blog/deco/I am grateful I made this mistake early in my diving experience, rather than later at greater depths and in harsher conditions.Tue, 01 Jan 2019 00:00:00 GMThttps://www.liaojiayi.com/en/blog/flink-IoT/https://www.liaojiayi.com/en/blog/flink-IoT/An English rendering of my Chinese translation of the data Artisans article on stream processing and Apache Flink in IoT.Sat, 24 Nov 2018 00:00:00 GMThttps://www.liaojiayi.com/en/blog/flink-savepoint-vs-checkpoint/https://www.liaojiayi.com/en/blog/flink-savepoint-vs-checkpoint/Translated from the Data Artisans blog, this article compares Savepoints and Checkpoints in Apache Flink.Sun, 04 Nov 2018 00:00:00 GMThttps://www.liaojiayi.com/en/blog/flink-checkpoint/https://www.liaojiayi.com/en/blog/flink-checkpoint/Three approaches to initializing Apache Flink state when no usable Checkpoint or Savepoint exists.Thu, 25 Oct 2018 00:00:00 GMThttps://www.liaojiayi.com/en/blog/MapFailed/https://www.liaojiayi.com/en/blog/MapFailed/An incident in which Map Failed caused a JVM out-of-memory error, with its causes and remedies.Thu, 11 Oct 2018 00:00:00 GMThttps://www.liaojiayi.com/en/blog/oracle-cep/https://www.liaojiayi.com/en/blog/oracle-cep/CEP is a major real-time processing capability. Mature systems such as Siddhi and EsperTech have their own syntax; incorporating CEP into SQL makes for an interesting alternative.Sat, 25 Aug 2018 00:00:00 GMThttps://www.liaojiayi.com/en/blog/calcite/https://www.liaojiayi.com/en/blog/calcite/Calcite, a widely used general-purpose SQL parsing framework.Fri, 20 Jul 2018 00:00:00 GMThttps://www.liaojiayi.com/en/blog/bitmap/https://www.liaojiayi.com/en/blog/bitmap/Notes on bitmap compression techniques and RoaringBitmap's implementation and performance.Fri, 06 Jul 2018 00:00:00 GMThttps://www.liaojiayi.com/en/blog/flink-1.9https://www.liaojiayi.com/en/blog/flink-1.9Some notes on the Table APIs, which I had not used before.Fri, 06 Jul 2018 00:00:00 GMThttps://www.liaojiayi.com/en/blog/aerospike-write/https://www.liaojiayi.com/en/blog/aerospike-write/Aerospike’s write mechanism and the problems it can encounter.Sun, 03 Jun 2018 00:00:00 GMThttps://www.liaojiayi.com/en/blog/hs-err-pid-log/https://www.liaojiayi.com/en/blog/hs-err-pid-log/A guide to JVM fatal-error logs, including thread states, native frames, and memory information.Wed, 11 Apr 2018 00:00:00 GMThttps://www.liaojiayi.com/en/blog/aerospike/https://www.liaojiayi.com/en/blog/aerospike/An introduction to Aerospike’s use cases, data model, and indexes, with guidance on choosing it based on its strengths and weaknesses.Tue, 20 Mar 2018 00:00:00 GMThttps://www.liaojiayi.com/en/blog/flink-exactly-once/https://www.liaojiayi.com/en/blog/flink-exactly-once/An introduction to Exactly Once semantics in Flink.Tue, 20 Mar 2018 00:00:00 GMThttps://www.liaojiayi.com/en/blog/sort-algorithms-md/https://www.liaojiayi.com/en/blog/sort-algorithms-md/Sorting is fundamental, but different algorithms embody different ideas worth studying. To reinforce my understanding, I am implementing the basic sorting algorithms by hand.Fri, 23 Sep 2016 00:00:00 GMThttps://www.liaojiayi.com/en/blog/programming-basic/https://www.liaojiayi.com/en/blog/programming-basic/Important fundamentals that do not always arise in daily work.Tue, 23 Aug 2016 00:00:00 GMThttps://www.liaojiayi.com/en/blog/leetcode/https://www.liaojiayi.com/en/blog/leetcode/Notes on LeetCode solutions.Fri, 22 Jul 2016 00:00:00 GMT