CEP In Flink (4) - Usage Bottlenecks

The previous three posts mainly explained how CEP works in Apache Flink. Although Flink optimizes CEP with an NFA and SharedBuffer, as a developer I still find many aspects of its stability, or rather its usability, hard to rely on.

Recap

Here are the previous three posts:

1. CEP In Flink (1) - Parsing CEP Rules
2. CEP In Flink (2) - Matching CEP Rules
3. CEP In Flink (3) - Extracting Matched Events

Bottlenecks and Limitations

All CEP details discussed here are based on release-1.6.

Pattern

There are two issues with Patterns:

• Multiple Patterns cannot be matched simultaneously
• Patterns cannot be modified dynamically
• A Pattern cannot end with NotFollowBy

The first two situations are very common in real applications. A company may run multiple campaigns at once, or dynamically adjust user outreach strategies to reactivate users promptly. The relevant community ticket is FLINK-7129.

Combining the third case with a timeout is also a common CEP requirement. Minor changes to the existing code would support NotFollowBy together with Timeout, but Flink’s rule parser has not yet lifted this restriction.

Audience Filtering

In some scenarios I know, CEP is used to select an audience, such as people who clicked a campaign link but did not participate. Implementing this in Flink CEP requires creating an NFA for every user. Imagine the memory pressure when the number of users reaches tens of millions.

EventTime Processing

CEP naturally needs to support EventTime in stream processing, which also means handling late data through watermarks. Flink stores the detailed late-arriving data in a Map<Long, List<IN>> structure. In other words, if the watermark is set to the current time minus 5 minutes, 5 minutes of data will be kept in memory. To me, this is another major source of memory pressure.