Roaring bitmaps

A better compressed bitset

Bitsets, also called bitmaps, are commonly used as fast data structures. Unfortunately, they can use too much memory. To compensate, we often use compressed bitmaps.

Roaring bitmaps are compressed bitmaps which tend to outperform conventional compressed bitmaps such as WAH, EWAH or Concise. In some instances, they can be hundreds of times faster and they often offer significantly better compression.

Roaring bitmaps are used by several important systems:

When should you use a bitmap?

Sets are a fundamental abstraction in software. They can be implemented in various ways, as hash sets, as trees, and so forth. In databases and search engines, sets are often an integral part of indexes. For example, we may need to maintain a set of all documents or rows (represented by numerical identifier) that satisfy some property. Besides adding or removing elements from the set, we need fast functions to compute the intersection, the union, the difference between sets, and so on.

To implement a set of integers, a particularly appealing strategy is the bitmap (also called bitset or bit vector). Using n bits, we can represent any set made of the integers from the range [0,n): it suffices to set the ith bit is set to one if integer i is present in the set. Commodity processors use words of W=32 or W=64 bits. By combining many such words, we can support large values of n. Intersections, unions and differences can then be implemented as bitwise AND, OR and ANDNOT operations. More complicated set functions can also be implemented as bitwise operations.

When the bitset approach is applicable, it can be orders of magnitude faster than other possible implementation of a set (e.g., as a hash set) while using several times less memory.

When should you use compressed bitmaps?

An uncompress BitSet can use a lot of memory. For example, if you take a BitSet and set the bit at position 1,000,000 to true and you have just over 100kB. That's over 100kB to store the position of one bit. This is wasteful even if you do not care about memory: suppose that you need to compute the intersection between this BitSet and another one that has a bit at position 1,000,001 to true, then you need to go through all these zeroes, whether you like it or not. That can become very wasteful.

This being said, there are definitively cases where attempting to use compressed bitmaps is wasteful. For example, if you have a small universe size. E.g., your bitmaps represent sets of integers from [0,n) where n is small (e.g., n=64 or n=128). If you are able to uncompressed BitSet and it does not blow up your memory usage, then compressed bitmaps are probably not useful to you. In fact, if you do not need compression, then a BitSet offers remarkable speed.

How does Roaring compares with the alternatives?

Most alternatives to Roaring are part of a larger family of compressed bitmaps that are run-length-encoded bitmaps. They identify long runs of 1s or 0s and they represent them with a marker word. If you have a local mix of 1s and 0, you use an uncompressed word.

There are many formats in this family:

There is a big problem with these formats however that can hurt you badly in some cases: there is no random access. If you want to check whether a given value is present in the set, you have to start from the beginning and "uncompress" the whole thing. This means that if you want to intersect a big set with a large set, you still have to uncompress the whole big set in the worst case...

Roaring solves this problem. It works in the following manner. It divides the data into chunks of 216 integers (e.g., [0, 216), [216, 2 x 216), ...). Within a chunk, it can use an uncompressed bitmap, a simple list of integers, or a list of runs. Whatever format it uses, they all allow you to check for the present of any one value quickly (e.g., with a binary search). The net result is that Roaring can compute many operations much faster that run-length-encoded formats like WAH, EWAH, Concise... Maybe surprisingly, Roaring also generally offers better compression ratios.

Publications

Technical Posts

Java Software

You can browse our API documentation online. You can download releases from the Maven repository or from GitHub. If your project depends on roaring, you can specify the dependency in the Maven "pom.xml" file:

<dependencies>
    <dependency>
      <groupId>org.roaringbitmap</groupId>
      <artifactId>RoaringBitmap</artifactId>
      <version>[0.5,)</version>
    </dependency>
 </dependencies>

where the version tag should refer to the version you need. Code sample:

import org.roaringbitmap.*;

    //...

    RoaringBitmap rr = RoaringBitmap.bitmapOf(1,2,3,1000);
    RoaringBitmap rr2 = new RoaringBitmap();
    for(int k = 4000; k<4255;++k) rr2.add(k);

    RoaringBitmap rror = RoaringBitmap.or(rr, rr2);

You can also work directly with memory-mapped bitmaps using the MutableRoaringBitmap and ImmutableRoaringBitmap classes.

Generally, our compressed bitmap Java software is hosted on GitHub.

C/C++ Software

The CRoaring library is a C/C++ library that provides an optimized implementation of Roaring for C/C++ programmers.

You can grab one of our releases. Please check the README file first. There is a C and a C++ interface.

Python software

We have a Python version wrapping our C/C++ code on GitHub.

Rust software

We have a Rust version wrapping our C/C++ code on GitHub.

Go Software

We have a Go version wrapping our C/C++ code on GitHub. You can browse the documentation online. You can grab a copy by typing:

go get github.com/RoaringBitmap/gocroaring

We also have a pure Go implementation. You can browse our API documentation online. You can download the code from GitHub. As usual, you can grab a copy by typing:

// dependencies
go get github.com/smartystreets/goconvey/convey
go get github.com/willf/bitset

//  main 
go get github.com/RoaringBitmap/roaring

Code sample (for the pure Go version):

rb1 := roaring.BitmapOf(1, 2, 3, 4, 5, 100, 1000)
rb2 := roaring.BitmapOf(3, 4, 1000)
fmt.Println("Cardinality: ", rb1.GetCardinality())
fmt.Println("Contains 3? ", rb1.Contains(3))
rb1.And(rb2)
rb3 := roaring.NewBitmap()
rb3.Add(1)
i := rb3.Iterator()
for i.HasNext() {
    fmt.Println(i.Next())
}
fmt.Println()

Various ports

In addition to the Java, C/C++, Python and Go versions described above, there are many other ports.

C++ izenelib by izenecloud
Go Roaring Bitmaps - compressed bitmaps in Go by Fernando Zandona
C Roaring bitmaps in C by Chris O'Hara
Python The Whoosh search engine uses Roaring (source code)
Java Apache Lucene has a Roaring bitmap implementation (source code)
Cython Roaring Bitmap in Cython by Andreas van Cranenburgh
Rust Roaring bitmap implementation for Rust by Nemo157
Haskell Roaring bitmaps in Haskell by Thomas Sutton
C# A .NET library for compressed bit set data structures
C# .NET Implementation of RoaringBitmap (C#)
OCaml Roaring bitmaps for OCaml

Funding

This work was supported by NSERC grant number 26143.