Aerospike (3) - Eviction
In real-time applications, eviction in systems such as Aerospike and Redis helps prevent storage failures caused by traffic spikes.
Translated from Chinese with AI · Read the original
In real-time applications, we often configure eviction in Aerospike, Redis, and similar systems to prevent storage failures caused by traffic spikes.
How It Works
As shown above, Aerospike distributes records with a TTL along the horizontal axis according to that TTL. Each bucket’s width is determined by the maximum TTL and evict-hist-buckets. Here:
bucket_width = max_ttl / evict-hist-buckets
evict-hist-buckets can be configured per namespace and changed dynamically without restarting the cluster. high-water-memory-pct or high-water-disk-pct determines when eviction begins. Once a threshold is reached, the nsup thread starts evicting from the histogram’s first bucket.
Limitations
Reality is not always so simple. Eviction is a tradeoff between retaining and discarding data, so an off-the-shelf strategy may never perfectly fit an application. Aerospike’s mechanism is imperfect: storage may exceed the configured HWM without any data being evicted. Since Aerospike 3.8, eviction has two characteristics:
- evict-hist-buckets is dynamically configurable, allowing users to control histogram granularity and eviction granularity.
- Eviction operates on whole buckets; it cannot remove only part of a bucket.
Another relevant parameter is evict-tenths-pct.
It determines how much data is evicted each time. Consider an extreme case: an uneven histogram puts every record in the first bucket. Surely it cannot evict that entire bucket? This is where evict-tenths-pct matters. Adjust evict-hist-buckets dynamically to make the histogram finer, reducing the first bucket’s record count below the evict-tenths-pct limit so eviction can proceed. Note that bucket objects themselves consume memory; see the links below for details.
This eviction mechanism introduces risks. Although its control is finer than Redis’s eviction strategies, extreme situations may require manual parameter adjustments or make eviction impossible. I had to tune it manually a few days ago after business traffic increased.
Tuning
Aerospike’s eviction mechanism mainly depends on those two parameters. If production problems arise, use the logs to adjust the strategy. Here are excerpts from the official logs:
Apr 07 2016 13:42 GMT: WARNING (nsup): (thr_nsup.c) {‘{’}test{‘}’} no records below eviction void-time 200346037 - insufficient histogram resolution?
void-time is the timestamp of the threshold bucket determined by evict-tenths-pct, measured from January 1, 2010: effectively Aerospike’s own timestamp. The threshold bucket here is bucket 1. As in the earlier example, adjust evict-hist-buckets.
Jan 30 2017 02:36 GMT: WARNING (nsup): (thr_nsup.c) {‘{’}test{‘}’} no records below eviction void-time 222541923 - threshold bucket 361, width 259 sec, count 686375 > target 530312 (0.5 pct)
Here the threshold bucket is 361 and each bucket spans 259 seconds. However, the first bucket contains 686375 records, exceeding the 0.5 percent limit set by evict-tenths-pct, so eviction cannot proceed.