Bitmap - Performance and Implementation
Notes on bitmap compression techniques and RoaringBitmap's implementation and performance.
Translated from Chinese with AI · Read the original
Original paper: An Experimental Study of Bitmap Compression vs.
Inverted List Compression
Bitmaps are versatile: with the right usage, both space consumption and query latency can be excellent. I have recently studied them and am recording notes based on the paper above.
History
From basic bitmaps to RoaringBitmap, perhaps the best choice for many current workloads, the underlying ideas are not especially complicated, but their development spans more than a decade.
WAH(Word Aligned Hybrid)
This algorithm compresses only groups consisting entirely of zeros or ones. It divides bits into consecutive groups of 31 and encodes each into 32 bits, as shown below:

EWAH(Enhanced Word Aligned Hybrid)
This extends WAH with a metadata header. My understanding is that it helps queries and insertion more than storage compression. The header is structured as follows:

CONCISE(Compressed N Composble Integer Set)
Another WAH optimization handles odd bits. In WAH, a single set bit prevents the entire group from being compressed. This method records the position of that exceptional bit.

VALWAH(Variable-Aligned Length WAH)
A parameterized optimization relaxes WAH’s fixed 32-bit group size, which can represent up to 2^31 - 1 compressed groups, usually more than needed. Parameters vary by bitmap rather than following one fixed rule, which hurts query efficiency.
Roaring
Values are divided into buckets spanning 65,536 positions. Integers in a bucket share their upper 16 bits, which identify the bucket: [0, 65535] has upper bits 0, and [65536, 65536*2 - 1] has upper bits 1. Within a bucket, 16-bit short integers store the lower bits. Above 4,096 values, this representation offers no compression benefit.
Reading RoaringBitmap’s Source
HighLowContainer holds keys for the shared upper 16 bits and the containers storing the values. Since containers hold the actual data, most optimizations happen there. Examining add reveals their internal structures.
First, consider the frequently used binarySearch method. It returns an index when found and a negative insertion-position encoding otherwise, conveying both location and existence.
protected static int hybridUnsignedBinarySearch(final short[] array, final int begin, final int end, final short k) { int ikey = toIntUnsigned(k); // next line accelerates the possibly common case where the value would // be inserted at the end if ((end > 0) && (toIntUnsigned(array[end - 1]) < ikey)) { return -end - 1; } int low = begin; int high = end - 1; // 32 in the next line matches the size of a cache line while (low + 32 <= high) { final int middleIndex = (low + high) >>> 1; final int middleValue = toIntUnsigned(array[middleIndex]);
if (middleValue < ikey) { low = middleIndex + 1; } else if (middleValue > ikey) { high = middleIndex - 1; } else { return middleIndex; } } // we finish the job with a sequential search int x = low; for (; x <= high; ++x) { final int val = toIntUnsigned(array[x]); if (val >= ikey) { if (val == ikey) { return x; } break; } } return -(x + 1);}ArrayContainer
short[] content;
@Overridepublic Container add(final short x) { int loc = Util.unsignedBinarySearch(content, 0, cardinality, x); if (loc < 0) { // Transform the ArrayContainer to a BitmapContainer // when cardinality = DEFAULT_MAX_SIZE if (cardinality >= DEFAULT_MAX_SIZE) { BitmapContainer a = this.toBitmapContainer(); a.add(x); return a; } if (cardinality >= this.content.length) { increaseCapacity(); } // insertion : shift the elements > x by one position to // the right // and put x in it's appropriate place System.arraycopy(content, -loc - 1, content, -loc, cardinality + loc + 1); content[-loc - 1] = x; ++cardinality; } return this;}Relevant fields:
- content: A sorted short array simplifies insertion, deletion, updates, and lookup. HighLowContainer already holds the upper 16 bits, so only the lower 16 bits are needed here.
- DEFAULT_MAX_SIZE: Sorted-array insertion requires binary search and is relatively inefficient. The limit is 4,096; above it, the container converts to BitmapContainer.
The add flow:
- Binary-search content for x. If it exists, do nothing; otherwise continue.
- Check cardinality to decide whether to convert the container or expand capacity.
- Shift the suffix after loc one position and insert the value.
BitmapContainer
final long[] bitmap;
@Overridepublic Container add(final short i) { final int x = Util.toIntUnsigned(i); final long previous = bitmap[x / 64]; long newval = previous | (1L << x); bitmap[x / 64] = newval; if (USE_BRANCHLESS) { cardinality += (previous ^ newval) >>> x; } else if (previous != newval) { ++cardinality; } return this;}Relevant fields:
- bitmap: A container can represent 65,536 (2^16) integers. BitmapContainer groups the bits into 64-bit longs, forming a long array.
The add flow:
- Use x/64 to find the long-array position and read previous.
- Calculate newval as previous | (1L << x).
- Update cardinality. Dense values can produce runs of ones within a long. Calling runOptimize can convert these into a RunContainer.
RunContainer
RunContainer compresses consecutive ones: for example, 15, 16, 17, 18 becomes 15,3. Its key field is the short array valuesLength. Position 2n stores a value, and 2n+1 stores the following run length. For example, valuesLength = [1,3,15,2,88,4] represents 1,2,3,15,16,88,89,90,91 in the example.
private short[] valueslength;int nbrruns = 0;The add method:
@Overridepublic Container add(short k) { // TODO: it might be better and simpler to do return // toBitmapOrArrayContainer(getCardinality()).add(k) // but note that some unit tests use this method to build up test runcontainers without calling // runOptimize int index = unsignedInterleavedBinarySearch(valueslength, 0, nbrruns, k); if (index >= 0) { return this;// already there } index = -index - 2;// points to preceding value, possibly -1 if (index >= 0) {// possible match int offset = toIntUnsigned(k) - toIntUnsigned(getValue(index)); int le = toIntUnsigned(getLength(index)); if (offset <= le) { return this; } if (offset == le + 1) { // we may need to fuse if (index + 1 < nbrruns) { if (toIntUnsigned(getValue(index + 1)) == toIntUnsigned(k) + 1) { // indeed fusion is needed setLength(index, (short) (getValue(index + 1) + getLength(index + 1) - getValue(index))); recoverRoomAtIndex(index + 1); return this; } } incrementLength(index); return this; } if (index + 1 < nbrruns) { // we may need to fuse if (toIntUnsigned(getValue(index + 1)) == toIntUnsigned(k) + 1) { // indeed fusion is needed setValue(index + 1, k); setLength(index + 1, (short) (getLength(index + 1) + 1)); return this; } } } if (index == -1) { // we may need to extend the first run if (0 < nbrruns) { if (getValue(0) == k + 1) { incrementLength(0); decrementValue(0); return this; } } } makeRoomAtIndex(index + 1); setValue(index + 1, k); setLength(index + 1, (short) 0); return this;}The decision logic is fairly involved; a flowchart should make it clearer.
Comparing containers:
Container | Space Efficiency | Query Efficiency
- | :-: | -: ArrayContainer | Uncompressed, low | Binary search, low BitmapContainer | Uncompressed, low | Direct indexing, high RunContainer | Compressed, high | Sequential search, medium
