Some thoughts on the future of stream processing, inspired by the SIGMOD paper Beyond Analytics: The Evolution of Streaming Processing Systems.

Interested readers should read the paper. Its first half covers the field’s development and many stream-processing concepts; I will skip those and begin with Section 4.

4.1 Emerging Applications

1. Cloud Application
2. Machine Learning
3. Streaming Graphs

First, cloud native is attracting enormous attention. In my experience, AWS and physical IDC servers have incomparable maintenance costs. As investment moves away from internet businesses and funding becomes harder, hardware costs may matter even more to startups. Two examples from my experience with AWS:

• Usage-based object storage: Similar to cold storage, it benefits from the provider’s global view and efficient allocation, making storage very cheap while bandwidth and CPU used by accesses are billed separately. This makes sense: keep historical detailed data in object storage and recent or hot data in HDFS or another OLAP store to substantially reduce storage costs.
• Cheap but unreliable instances: This surprised me at the time. AWS offers inexpensive instances with weak availability guarantees. They are unsuitable for long-running services, but with speculative execution and a shuffle service, they work well for batch jobs. Batch work is offline, and DAG dependencies naturally support recovery.

The second point is machine learning, which seems uncontroversial and needs no explanation.

The third concerns computation graphs. Spark and Flink usually compile user code or SQL into a DAG before scheduling it. This is a static graph: even iterative computation retains the same topology and formulas. Deep-learning workloads can change formulas and computation patterns during execution, changing the topology as well. These are dynamic graphs, as discussed in Ray.

4.2 The road ahead

Programming Models

Another point worth discussing: whether batch or streaming, distributed applications are usually written using either framework-specific concepts, such as Spark RDDs and Flink DataStreams, or SQL.

Code requires learning framework-specific concepts, while streaming SQL still has a high entry barrier, unlike batch SQL. FaaS and actor-model approaches are therefore appearing in big-data systems, including Stateful Functions from Flink’s founding company and Ray actors.

This seems a long-term trend. In my experience with business teams, algorithm engineers find distributed deployment, execution, and high availability painful and expensive to learn. They simply want their algorithms to run on remote servers. That is why I like Ray’s concise model: one annotation can run a function on a distributed service.

1
import ray
2
ray.init()
3

4
@ray.remote
5
def f(x):
6
    return x * x
7

8
futures = [f.remote(i) for i in range(4)]
9
print(ray.get(futures))

Transactions

Transaction support in streaming mainly means ACID guarantees for stateful services. Only with these guarantees can stream processing directly serve online business workloads.

Advanced State Backends

This is closely related to my work: transaction support, as mentioned above, and better state backends. No state store yet fits streaming perfectly. Flink uses RocksDB, whose update performance is problematic.

My ideal state backend would offer:

• Fully pluggable storage, whether embedded like local RocksDB or remote like HBase. Serialization should be independent of the storage medium, allowing users to switch between them.
• State queryable at any time, like a distributed database. State could support real-time analysis as well as output. Combined with application logic, real-time analysis, state-producing computation, and output would form a complete loop serving online businesses.

Loops & Cycles

This concerns a feedback loop and fits more naturally within Stateful Functions. Implementing it on existing big-data frameworks seems very difficult.

Elasticity & Reconfiguration

Cloud-native elasticity is relatively easy for stateless services with Kubernetes or other frameworks. For stateful services, redistributing state without interrupting online service remains difficult.

Current industry approaches mainly include:

• Automating stop, reconfigure, and restart. Redundant steps can be removed, but the overall cost remains close to manual operation.
• Running hot and standby jobs simultaneously, electing a leader through ZooKeeper. Service continues, but resource consumption doubles and code must handle issues such as aligned consumption offsets, making this difficult to use.

Dynamic Topologies

Requirements shape frameworks. Deep learning and neural networks will demand increasingly flexible scheduling, especially when robots react to their surroundings. Thousands of small tasks may need scheduling each second to process different sensors and environmental factors.

Shared Mutable State

One limitation of current big-data frameworks is the JobManager-TaskManager interaction model. TaskManagers cannot share resources or state; in Apache Flink they cannot even communicate directly. State X computed by Task A cannot be used by Task B. A common workaround duplicates A’s input to B and repeats the same computation to regenerate X.

Sharing state between tasks makes consistency difficult. Embedded state backends are not a good fit; Ray uses Plasma shared-memory storage.

Queryable State

Already discussed above; skipping this point.

State Versioning

This requirement seems manageable. If state versions are needed, existing snapshots and checkpoints can provide them.

Hardware Acceleration

Hardware acceleration.

Summary

In one sentence, stream processing continues to advance because users increasingly demand real-time results.

The paper’s scope extends beyond Apache Flink to streaming services such as Akka Streams, although much of its discussion concerns big data. In that field, stream processing is already a mature and widely used stack for analytics and live dashboards. Major internet companies have largely built the capability to move warehouses and reporting from offline to real time.

The next step is therefore from real-time analytics toward directly powering business applications, along two paths:

• Extend existing stream-processing models for more complex computation, greater speed, and better stability, without major architectural changes.
• Support business-facing deep-learning and neural-network workloads, which requires rethinking established concepts such as DAGs.