Data Lakes (2) - How Iceberg's Core Features Work

The previous article, Data Lakes (1) - How Hudi’s Core Features Work, explained Hudi’s concepts and primary-key indexes for upserts. Apache Iceberg is another widely used data lake framework. Though their original goals and designs differ, expanding requirements have made them increasingly similar from a user’s perspective.

• Data Lakes (1) - How Hudi’s Core Features Work
• Data Lakes (2) - How Iceberg’s Core Features Work

Apache Iceberg

The official website describes it as Apache Iceberg is an open table format for huge analytic datasets.. Its founders position Iceberg as an efficient storage format for large-scale analytics. Much as Hive wraps HDFS, it fundamentally addresses data-warehouse problems.

Iceberg initially evolved toward a faster, more usable warehouse format, supporting schema changes and file-level filtering. Integration with Flink and the addition of Delete, Update, and Merge semantics have broadened its use cases.

Background

Hive/Spark on HDFS has traditionally underpinned offline warehouses. Increasingly real-time requirements and rapid iteration have exposed limitations:

• No row-level updates: Changes require overwriting an entire Hive table at high cost.
• No read/write isolation: One user’s writes can affect another’s reads, especially streaming reads.
• No version rollback or snapshots without retaining substantial historical data.
• No incremental reads: Each scan reads a whole table or partition.
• Limited performance: Pruning stops at Hive partition granularity.
• No schema evolution.
• …..

Basic Concepts

As shown above, Iceberg organizes HDFS files into snapshots, manifest lists, manifests, and data files.

1. Snapshot: Each user commit, such as a writing Spark job, creates a snapshot.
2. Manifest List: Tracks all manifests in the snapshot.
3. Manifest: Tracks its data files.
4. Data File: Stores the data. Iceberg later introduced delete files for deletion records, at the same structural level as data files.

Core Features

Time Travel and Incremental Reads

Time travel lets users read data from historical points in time. For example, in Spark:

1
// time travel to October 26, 1986 at 01:21:00
2
spark.read
3
    .option("as-of-timestamp", "499162860000")
4
    .format("iceberg")
5
    .load("path/to/table")

This reads the Iceberg table at timestamp=499162860000. How does it work underneath?

Every write produces a snapshot. By retaining commit metadata such as timestamps, Iceberg can find the snapshot for a requested time and resolve its data files.

Incremental reads follow the same idea: find snapshots between the start and end timestamps and read their data files as input.

Fast Scan & Data Filtering

One reason for Hive’s limited query performance is coarse partition-level pushdown. Iceberg optimizes planning at finer granularity. When a query arrives:

1. Find the snapshot for the timestamp, defaulting to the latest.
2. Use query partition information to filter the snapshot’s manifests.
3. Extract data-file objects, containing only metadata, from those manifests.
4. Prune more finely using data-file properties, including column-level value counts, null counts, lower bounds, and upper bounds.

Implementing Deletes

Iceberg implements Delete to support row-level updates through Delete + Insert. This introduces two concepts:

• Delete File: Stores deletion records, either position deletes or equality deletes.
• Sequence Number: A property shared by data and delete files that orders inserts and deletes, preventing consistency problems.

position & equality delete

Iceberg introduces equality_ids, which users specify when creating a table to identify keys for future deletes. For example, deleting user data for GDPR can use user_id as equality_ids.

The two delete types use different file contents:

• Position delete: Three columns: file_path (the data file containing the row), pos (row position), and row (row data).
• Equality delete: The fields included in equality_ids.

Delete files are joined with data during reads. Position deletes are more efficient because they locate an exact file and require only row-position comparisons. During writes, Iceberg therefore keeps information about the active data file in memory to use position deletes whenever possible:

Consider inserts and deletes arriving in sequence. Suppose the writer closes a file after inserting a1 and b1 and opens another. The new c1 record and its position are then tracked in memory. A delete for user_id=c1 can locate the first row of fileA and create a position delete file. A delete for user_id=a1 has no in-memory location because its file is closed, so it creates an equality delete file.

Sequence Number

Read-time merging raises a question: if the same equality_id is inserted, deleted, and inserted again, how do we delete the first insertion while retaining the second?

Data and delete files receive sequence numbers in write order. A delete applies only to data files with earlier sequence numbers. Concurrent writes to the same record rely on Iceberg’s transaction mechanism: writers inspect metadata and sequence numbers, then retry optimistically if the result does not match expectations.

Schema Evolution

Schema evolution is one of Iceberg’s distinguishing features. It supports:

• Adding fields
• Dropping fields
• Renaming fields
• Modifying fields
• Reordering fields

Schema evolution also relies on the file hierarchy. Writes create snapshot -> manifest -> data-file layers, and reads resolve from the snapshot down to data files. Iceberg can therefore record schema information in manifests during writes and apply the appropriate conversions during reads.

Summary

This article introduced Iceberg’s concepts and mechanisms. Hudi checks indexes during writes to implement upserts. Iceberg instead relies on file organization and merge-on-read (MOR), making Hudi more read-friendly and Iceberg more write-friendly.

The next article will compare Hudi and Iceberg in more depth.