Repository navigation
Always use the AVX-512 decoder in array_container_from_bitset - #952
Merged
Merged
Conversation
Since #868 (v5.1.1), array_container_from_bitset only uses the AVX-512 VBMI2 decoder when the bitset holds at least 1024 values, and the scalar loop below that. The vector decoder is in fact faster at every cardinality, so the gate only slows down the conversion of bitsets with fewer than 1024 values. Such bitsets are common: roaring_bitmap_or_many converts every container of its accumulator to a bitset, and roaring_bitmap_repair_after_lazy converts each one back to an array when the union is sparse. gocroaring's FastOr, over 200 sparse bitmaps, became 1.42x slower between CRoaring 5.1.0 and 5.4.0, and removing the gate alone recovers all of it. Add microbenchmarks/bitset_to_array_bench, which times the conversion through the dispatch and through each decoder forced, over cardinalities from 16 to 4096 and three layouts (uniform, clustered, runs), as well as roaring_bitmap_or_many over many sparse bitmaps. Inputs cycle across many containers so that the branch predictor cannot learn them.
Decode calls array_container_from_bitset and pays for allocation, so it is the before/after of the real conversion and not a lower bound on the forced kernels. DecodeScalar and DecodeAvx512 now write into one array of exactly the cardinality, the buffer the library allocates.
lemire
added a commit
that referenced
this pull request
Nov 4, 2026
…953) array_container_from_bitset decodes sparse bitsets (at most 4096 values) with bitset_extract_setbits_avx512_uint16. Since #868, that kernel tests each word ("if (v != 0)") to skip zero words. When zero and nonzero words alternate, as when about 1000 values are spread out, that branch is mispredicted about once per value. Even after #952, gocroaring's FastOr (roaring_bitmap_or_many over 200 sparse bitmaps) remained 1.23x slower than with CRoaring 5.1.0, whose kernel had no such branch. Neither extreme works across inputs: decoding every word without testing it is fastest when most words are nonzero, but wastes time on very sparse bitsets or runs. So the kernel now picks by the capacity, which array_container_from_bitset sets to the cardinality: - from 2048 values (two per word on average), it decodes every word with masked stores, four words at a time without capacity checks while there is room for 256 values; - below 2048 values, it tests eight words at a time (VPTESTMQ) and visits only the nonzero ones, so that a run of zero words costs little and a group costs about one misprediction. Both stop once "capacity" values are written, instead of scanning all the words. Against master on a Xeon Gold 6548N, with microbenchmarks/bitset_to_array_bench (medians of 10 runs), the conversion is 1.02x to 22x faster at every cardinality from 16 to 4096 in the uniform, clustered and runs layouts, and roaring_bitmap_or_many is 1.01x to 1.43x faster. gocroaring's FastOr is now 3.6% faster than with CRoaring 5.1.0. Add a test that checks the decoder against the scalar loop on sparse bitsets with zero words, spread out or in runs, on both sides of 2048 values, with truncated capacities: the existing test only had nonzero words.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Since #868 (v5.1.1),
array_container_from_bitsetuses the AVX-512 VBMI2 decoder only when the bitset holds at least 1024 values, and falls back to the scalar loop below that. The scalar loop is slower at every cardinality, so this gate only slows down bitsets with fewer than 1024 values.That case is common.
roaring_bitmap_or_many(lazy union with bitset conversion) turns every container of its accumulator into a bitset, androaring_bitmap_repair_after_lazyturns each one back into an array when the union is sparse.This came up while updating gocroaring from CRoaring 5.1.0 to 5.4.0.
FastOr(roaring_bitmap_or_manyover 200 sparse bitmaps, about 1000 values per container) became 1.42× slower (9.25 ms → 13.14 ms, n=15, ±1%).perfshowed 40% of the time in the scalarbitset_extract_setbits_uint16, where 5.1.0 ranbitset_extract_setbits_avx512_uint16. Removing the gate, and changing nothing else, bringsFastOrback to the 5.1.0 time. None of the other 60 gocroaring benchmarks moved.Fix
cardinality >= 1024condition, so the AVX-512 decoder is used whenever the hardware supports it.convert.candbitset_util.cthat claimed the scalar loop was as fast or faster on sparse input.Benchmarks (so this does not regress again)
This PR adds
microbenchmarks/bitset_to_array_bench:Decode: the dispatchingarray_container_from_bitset.DecodeScalarandDecodeAvx512: each decoder forced.OrMany:roaring_bitmap_or_manyover many sparse bitmaps whose union stays sparse.The
Decode*benchmarks cover cardinalities from 16 to 4096 (including 1000, 1023 and 1024, around the old gate) and three layouts: uniform, clustered and runs.Decodeincludes allocation, so it is the before/after ofarray_container_from_bitset. CompareDecodeScalarwithDecodeAvx512for the decoders themselves: on x64 with AVX-512 the vector decoder is faster at every cardinality.Inputs cycle across 64 bitsets, and across 1024 chunks for
OrMany. A first version used only 16 chunks; there the branch predictor learned the input and hid most of the regression.Results
Measured on a Xeon Gold 6548N (Emerald Rapids), pinned to one core, medians of 5 runs. Times are in ns.
Scalar vs. AVX-512 decoder (forced), uniform layout:
The AVX-512 decoder also wins at every cardinality in the clustered layout (1.36× to 5.2×) and the runs layout (1.27× to 1.54×).
Decode(the dispatch), before → after this PR:OrMany(1024 chunks, 5 values per chunk per bitmap), before → after, 10 samples, spread about 3%:All 27 tests pass, and all changed files are clean under clang-format 17. Non-x64 builds are unaffected; the new benchmark builds and runs on arm64 macOS, where
DecodeAvx512is compiled out.Not in this PR
The array/array union in
src/containers/mixed_union.c(around line 237) also converts a sparse bitset back to an array, and calls the scalarbitset_extract_setbits_uint16unconditionally, as it did before #868. It could likely use the AVX-512 decoder too. That is not a regression, so I left it for a separate change.