[lemire]

Two 32-bit operations were slow when Valkey's Roaring encoding of strings used them for BITCOUNT key start end and BITOP NOT. Profiles showed container_rank at 46% of a 1 KB BITCOUNT, and most of a BITOP NOT in roaring_bitmap_flip_closed itself.

Changes

• roaring_bitmap_range_cardinality(): in the first and last containers of the range, it subtracted two ranks. In a bitset container each rank is a popcount from the start of the container, so counting a range of 128 words could read up to 2048. The new container_range_cardinality() counts the values in [min, max]:
  • bitsets: popcount of the words of the range, or of the words outside it when that is shorter (the cardinality is stored);
  • arrays: the difference of two ranks, as before;
  • runs: the overlap of each run with the range.
• roaring_bitmap_flip(): each key of the range cost two binary searches (input and result) and an insertion into the result. It now visits the keys of the range and the containers of the input together, in one pass, and appends to the result. A full container flips to nothing and is skipped, instead of building and freeing an empty container. insert_flipped_container() and insert_fully_flipped_container() are removed: nothing else used them. roaring_bitmap_flip_inplace() is unchanged.

Results

Intel Xeon Gold 6548N, GCC 14 -O3 -march=native, best of 10 (range cardinality, 20,000 random ranges) or 20 (flip) runs:

	5.5.5	this PR
range cardinality, 8192 values, random 20% of 100M (bitsets)	256 ns	95 ns
range cardinality, 65536 values, same bitmap	310 ns	218 ns
range cardinality, 8192 values, random 0.1% of 100M (arrays)	83 ns	84 ns
range cardinality, 8192 values, runs of 300 every 1000	107 ns	94 ns
flip [0, 50M) of the first 50M values (full runs)	39.7 µs	2.2 µs
flip [0, 2^32) of 100k random 32-bit values	7,700 µs	3,500 µs
flip [0, 100M) of random 0.1% of 100M (arrays)	264 µs	191 µs
flip [0, 100M) of random 20% of 100M (bitsets)	3,910 µs	3,800 µs

In Valkey (server-side time per call), BITCOUNT over 1 KB of a dense bitmap goes from 2.45 µs to 0.88 µs, and BITOP NOT on a 50M-bit prefix from 54 µs to 9.4 µs.

Benchmark program

#include <stdio.h>
#include <stdlib.h>
#include <time.h>
#include "roaring.h"
static double now(void) { struct timespec t; clock_gettime(CLOCK_MONOTONIC, &t); return t.tv_sec * 1e9 + t.tv_nsec; }
static uint64_t s = 42;
static uint64_t rnd(void) { s ^= s << 13; s ^= s >> 7; s ^= s << 17; return s; }
static roaring_bitmap_t *random_bitmap(uint64_t universe, uint64_t n) {
    roaring_bitmap_t *r = roaring_bitmap_create();
    for (uint64_t i = 0; i < n; i++) roaring_bitmap_add(r, (uint32_t)(rnd() % universe));
    roaring_bitmap_run_optimize(r); return r;
}
static void bench_count(const char *name, const roaring_bitmap_t *r, uint64_t universe, uint64_t width) {
    enum { N = 20000 }; uint64_t *starts = malloc(N * 8), sink = 0; double best = 1e30;
    for (int i = 0; i < N; i++) starts[i] = rnd() % (universe - width);
    for (int k = 0; k < 10; k++) { double t = now(); for (int i = 0; i < N; i++) sink += roaring_bitmap_range_cardinality(r, starts[i], starts[i] + width); double d = (now() - t) / N; if (d < best) best = d; }
    printf("range_cardinality %-28s width %8llu: %8.1f ns (%llu)\n", name, (unsigned long long)width, best, (unsigned long long)(sink & 1)); free(starts);
}
static void bench_flip(const char *name, const roaring_bitmap_t *r, uint64_t end) {
    double best = 1e30; int reps = 0;
    for (int k = 0; k < 20; k++) { double t = now(); roaring_bitmap_t *f = roaring_bitmap_flip(r, 0, end); double d = now() - t; reps += roaring_bitmap_get_cardinality(f) & 1; roaring_bitmap_free(f); if (d < best) best = d; }
    printf("flip              %-28s            : %8.1f us (%d)\n", name, best / 1000, reps & 1);
}
int main(void) {
    const uint64_t U = 100000000;
    roaring_bitmap_t *dense = random_bitmap(U, U / 5);
    roaring_bitmap_t *sparse = random_bitmap(U, U / 1000);
    roaring_bitmap_t *runs = roaring_bitmap_create();
    for (uint64_t v = 0; v < U; v += 1000) roaring_bitmap_add_range(runs, v, v + 300);
    roaring_bitmap_run_optimize(runs);
    roaring_bitmap_t *prefix = roaring_bitmap_from_range(0, U / 2, 1);
    roaring_bitmap_t *hashed = random_bitmap((uint64_t)1 << 32, 100000);
    bench_count("random 20% (bitsets)", dense, U, 8192);
    bench_count("random 20% (bitsets)", dense, U, 65536);
    bench_count("random 0.1% (arrays)", sparse, U, 8192);
    bench_count("runs of 300 every 1000", runs, U, 8192);
    bench_flip("first 50M values", prefix, U / 2);
    bench_flip("100k random 32-bit values", hashed, (uint64_t)1 << 32);
    bench_flip("random 0.1% (arrays)", sparse, U);
    bench_flip("random 20% (bitsets)", dense, U);
    return 0;
}

Testing

• New test_range_cardinality_and_flip_by_container in toplevel_unit: checks range cardinality and flip against a plain bit array, for ranges ending inside and on the boundaries of chunks holding each kind of container (missing, array, bitset, full, one run, many runs).
• ctest: all 27 suites pass, also with ROARING_SANITIZE=ON.
• A separate randomized differential test (183,600 checks, mixed containers, with and without copy-on-write and shared containers) and flips of ranges up to 2^32 (each flipped twice, compared with the input), under AddressSanitizer and UBSan.
• clang-format 17 reports no violations.