[lemire]

Follow-up to #952. That PR removed the 1024-value gate in array_container_from_bitset, but it did not fully fix the regression it was meant to fix. It said removing the gate recovered gocroaring's FastOr. That three-way run was too noisy (±20–35%). A clean run (n=10, ±1–4%) against current master shows FastOr still 1.23× slower than with CRoaring 5.1.0 (9.0 → 11.1 ms) on AVX-512 hardware.

Cause

Since #868, bitset_extract_setbits_avx512_uint16 tests each word (if (v != 0)) to skip zero words. On the sparse input of array_container_from_bitset, zero and nonzero words often alternate; at about 1000 spread-out values, roughly 38% of words are zero. That branch is then mispredicted about once per value. (bitset_extract_intersection_setbits_avx512_uint16 already avoids this test, for the same reason.) The 5.1.0 kernel had no branch, and in FastOr, perf shows the decoder going from 2.7 ms (5.1.0) to 4.4 ms (master).

The benchmarks in #952 compared the two decoders of one version against each other, so they could not see the kernel itself getting slower.

Fix

Decoding every word without testing it (masked stores write nothing for v == 0) wins when most words are nonzero, but loses up to 2.3× to master on very sparse bitsets or runs. So the kernel now chooses by capacity, which array_container_from_bitset sets to the cardinality:

• ≥ 2048 values (two per word on average): decode every word, four at a time without capacity checks while room for 256 values remains.
• < 2048 values: test eight words at a time (VPTESTMQ) and visit only the nonzero ones. A run of zero words costs little, and a group costs about one misprediction.

Both stop once capacity values are written, instead of scanning all 1024 words; this is where the clustered layout gains most.

I also tried a single strategy with a per-group dense/sparse switch, and a "sticky" dense mode. Both stayed at 0.83–0.85× master on uniform 4096. This split is the only version I measured that is at least as fast as master in every cell.

Results

Xeon Gold 6548N (Emerald Rapids), pinned to one core, medians of 10 runs, microbenchmarks/bitset_to_array_bench. Each cell is ns, master → this PR:

card	uniform	clustered	runs
16	693 → 155 (4.47×)	510 → 23 (21.89×)	512 → 90 (5.71×)
64	1132 → 626 (1.81×)	542 → 26 (20.73×)	549 → 129 (4.24×)
128	1714 → 906 (1.89×)	565 → 31 (18.46×)	596 → 136 (4.37×)
256	2732 → 1121 (2.44×)	576 → 48 (11.98×)	707 → 160 (4.41×)
512	3969 → 1441 (2.76×)	578 → 83 (6.95×)	851 → 232 (3.67×)
768	4444 → 1804 (2.46×)	713 → 191 (3.73×)	1068 → 515 (2.07×)
1000	3889 → 1973 (1.97×)	724 → 225 (3.22×)	1190 → 678 (1.76×)
1024	3838 → 1991 (1.93×)	721 → 227 (3.18×)	1206 → 678 (1.78×)
2048	2256 → 1642 (1.37×)	786 → 268 (2.93×)	1834 → 1589 (1.15×)
4096	1605 → 1574 (1.02×)	891 → 445 (2.00×)	2934 → 1555 (1.89×)

roaring_bitmap_or_many (1024 chunks, 5 values per chunk per bitmap):

bitmaps	master (µs)	this PR (µs)	speedup
40	5855	4644	1.26×
100	8440	5904	1.43×
200	10382	8270	1.26×
400	12218	12144	1.01×
800	20271	20122	1.01×

gocroaring, against CRoaring 5.1.0

Full suite of 61 benchmarks, n=10, AVX-512:

• FastOr is now 3.6% faster than with 5.1.0 (9.02 → 8.69 ms). It was 42% slower with 5.4.0 and 23% slower with master.
• The suite as a whole is 9.3% faster.

On AVX2-only builds (-DCROARING_COMPILER_SUPPORTS_AVX512=0; this PR changes nothing there), master is 11% faster than 5.1.0, and FastOr is unchanged.

A few gocroaring benchmarks were 1–2.6% slower than with 5.1.0 even though the C code on their paths is identical, or differs only by the skew check from #913 in array_container_andnot. Rebuilt with -falign-functions=64 -falign-loops=64 -falign-jumps=64, all of them disappear into the noise but Intersect on a run and an array container: +2.8%, or 1.5 ns on a 55 ns cgo call. That path is identical C code in both versions, so I take it as a code-layout effect.

Tests

New avx512_extract_uint16_sparse_matches_scalar checks the decoder against the scalar loop on sparse bitsets with zero words:

• values spread out or in runs (so some words hold more than 32 values);
• cardinalities from 1 to 4096, on both sides of 2048;
• capacities that are exact or truncated, including truncation inside a group of eight.

The existing test only generates nonzero words, so it never exercised zero-word skipping. Removing the i += j truncation bookkeeping makes the new test fail.

All 27 tests pass, and the changed files are clean under clang-format 17.