Aerospike (2) - The Write Mechanism

I recently encountered several Aerospike problems. Reading logs, source code, and articles in the Aerospike community has given me a deeper understanding.

Storage Layout

The SSD namespace write flow is shown above. Suppose a MapOperation.PUT is in progress. Its path is:

1. Validate metadata according to MapOperation.PUT parameters and policies, such as whether the namespace and set exist, then write to the smallest write unit, block5.
2. A block defaults to 1024 bytes. It is added to the Write Queue only when full or when no more data is arriving.
3. Independent worker threads persist data from the Write Queue to SSD.
4. After writing, if memory or storage reaches the HWM (high-watermark-memory), eviction frees space by expiring some records early. See TTL and eviction for details.
5. After some records in a block are deleted, its usage level drops from 100 percent. Below defrag-lwm-pct, which defaults to 50 percent, defragmentation places blocks in the write queue to be rewritten.

Potential Risk

Several details of this mechanism create potential problems. See the Aerospike parameter list for tuning options.

• Writing a record larger than write-block-size fails. Adjust write-block-size per namespace. An excessively large block size can produce low utilization and frequent defragmentation; too small a size increases SSD flushes.
• Adding a block to the write queue may exceed its maximum size, producing “write fail: queue too deep: exceeds max %i”. Adjust max-write-cache; queue length = max-write-cache / write-block-size.
• For efficient reads, a record’s data must occupy contiguous memory. UPDATE, DELETE, and PUT therefore rewrite the whole record. Avoid large records that change frequently, as they hurt performance and can trigger the queue-too-deep limit.
• Frequent UPDATE, DELETE, and PUT operations reduce block utilization and trigger defragmentation, which interferes with normal writes. Lower the defragmentation low-water mark, defrag-lwm-pct, to reduce its frequency.
• Eviction expires data early, as discussed above. Adjust high-water-memory-pct and high-water-dist-pct to change the strategy.

Log Inspection

Logs are the easiest way to identify problems, but Aerospike’s abbreviations can be obscure. The info level seems sufficient for diagnosing most issues. Here are some important entries.

1
{test} /dev/xvdc: used-bytes 3695468032 free-wblocks 34690 write-q 0 write (445674,0.0) defrag-q 0 defrag-read (39134,0.0) defrag-write (5427,0.0)

This periodic log describes one node. For namespace test on SSD /dev/xvdc, 3695468032 bytes are used, 34690 free-wblocks remain, and write-queue length is 0. A total of 445674 blocks, including defragmented blocks, have been written, at a current rate of 0.0. The defrag queue is empty; 39134 blocks have been queued for defragmentation, currently at 0.0. Of these, 5427 actually underwent defragmentation, also currently at 0.0.

1
{test} breached eviction hwm (memory), memory sz:1792174976 (1792174976 + 0) hwm:1288490188, disk sz:3695922432 hwm:20128464896
2
no records below eviction void-time 266233878 - threshold bucket 4678, width 113 sec, count 298 > target 11 (0.5 pct)

This log concerns Aerospike eviction. Size has exceeded the HWM and triggered eviction, but count 298 > target 11. A bucket is the smallest eviction unit: at most 11 records can be evicted, while the nearest bucket contains 298, so eviction cannot proceed.