Hazelcast Jet - Low-Latency Stream Processing at the 99.99th Percentile

Original paper: Hazelcast Jet: Low-latency Stream Processing at the 99.99th Percentile. Hazelcast Jet is a stream-processing engine with a somewhat different focus from widely used systems such as Apache Flink and Spark Streaming. The following notes explore those differences.

Background

Stream processing has become increasingly popular, with different architectures serving IoT and big-data workloads. These systems generally emphasize application functionality rather than end-to-end latency and embedded deployment.

Hazelcast’s In-Memory Data Grid (IMDG) is a distributed in-memory object store. As customers increasingly used it for computation-heavy workloads, Hazelcast developed Jet to meet their low-latency streaming needs.

Architecture

The overall architecture, from the paper:

• Pipeline API: A high-level unified batch and streaming API.
• Core API: A low-level API for customizing DAGs and deeper tuning.
• Execution Engine: Executes operators.
• State Backend: Uses IMDG for streaming state.

Design Highlights

Graph construction:

1. Chain operators to reduce data copying.
2. Place operators with locality in mind wherever possible.
3. Insert exchange operators for nonlocal transfers; these can combine aggregates before transmission.

Tasklets and threads: Tasklets are Jet’s smallest processing units. Several share a cooperative thread and perform operations such as aggregation and joins. Blocking I/O, including network requests and disk access, runs on noncooperative threads. Sources and sinks typically interact with external systems and therefore use those threads. The tasklet/cooperative-thread model resembles an event loop.

JVM optimizations:

1. Reduce copying and transfers through operator chaining, local placement, and shared-memory communication between operators on the same node.

2. Reduce GC-induced jitter. Full GC pauses can destabilize latency, so Jet also uses its threading model to limit disruption:

   i. Cooperative threads and tasklets avoid unnecessary processing threads and CPU context switches. ii. The cooperative-thread count is slightly lower than the number of cores, reserving CPU capacity for GC.

State Management

Jet uses Hazelcast’s distributed IMDG object store and considers locality between operators and state during deployment. Suppose three nodes own these key spaces:

• Node 1: Key space (P1, P4, P7, P10)
• Node 2: Key space (P2, P5, P8, P11)
• Node 3: Key space (P3, P6, P9, P12)

After a key-hash exchange, each downstream operator owns a fixed key space. Jet places it on the corresponding node to maximize local state access. When an IMDG node fails, its state replicas are copied again and redistributed among nodes.

Fault Tolerance

Consistent snapshots use Chandy-Lamport. Sources trigger a snapshot and send barriers; downstream operators save state after receiving all barriers. Once every operator finishes, the saved state is committed to IMDG. Recovery uses the latest snapshot there. (Further details are not discussed.)

With transactional source and sink connectors, two-phase commit provides exactly-once semantics.

Discussion: Embedded Storage or a Distributed Storage Service?

Jet manages state through external IMDG rather than embedded storage. This choice deserves closer examination.

Flink currently uses embedded RocksDB, with clear advantages and disadvantages:

• Advantages
  1. The job manages the storage lifecycle, simplifying use and operations.
  2. Local state access supports high-throughput streaming.
  3. The storage engine’s checkpoint mechanism supports the synchronous checkpoint phase.
• Disadvantages
  1. Embedded storage lacks independent fault tolerance and relies on a distributed filesystem such as HDFS.
  2. Rescaling requires rebuilding storage and redistributing state, often a complex and slow process.
  3. Disk-backed storage imposes hardware requirements.

In production, these disadvantages grow with job count and state size. More jobs generate more checkpoint I/O and small HDFS files; larger state makes rescaling slower. Distributed storage services largely reverse these tradeoffs:

• Advantages
  1. Built-in distributed fault tolerance.
  2. More recovery and rescaling options, including remote state access and data migration.
  3. A broader view of stored data enables better layouts and resource utilization.
• Disadvantages
  1. Another distributed component to operate.
  2. Some performance loss: remote state access requires serialization and network transfer, both avoidable with embedded storage.
  3. Custom checkpointing is required. A distributed store does not know the streaming operators or their placement, so globally consistent snapshots must be coordinated from the computation side.

The choice depends on which drawbacks are acceptable. For smaller clusters, HDFS is unlikely to bottleneck and rescaling costs remain manageable, making embedded storage sensible. Large clusters that can operate another component and redesign checkpointing around it may prefer distributed storage.

Middle-ground options exist: optimize embedded-storage rescaling and merge small files before HDFS uploads, or use locality-aware scheduling, as in this paper, to reduce remote-access costs. There are many such techniques beyond this article’s scope.

Summary

Much of Jet’s fault tolerance and semantics resembles Flink. Its major difference is IMDG-based external state management, which provides a useful alternative. Jet also targets two compelling areas:

1. Low latency
2. Both embedded and distributed deployment

Flink and Spark Streaming are widely used but do not guarantee stable end-to-end latency, so they more often support offline or near-online computation than latency-sensitive consumer services such as monetary calculations. Supporting both embedded and distributed deployment is also unusual: siddhi-io, for example, is embedded but not distributed. In practice, highly latency- and correctness-sensitive online services often use embedded engines such as Siddhi or Drools, while near-online and offline reporting uses Flink or Spark Streaming.

The paper’s Pipeline API and cooperative-thread/tasklet model are already common in production. Low latency mostly comes from careful DAG-level details: shared memory within nodes, spare CPU for GC, and combine operators for preaggregation. Beyond IMDG, the overall architecture appears to offer relatively few new ideas.