Repository navigation
[BUG FIX] fix shrink to fit shared containers and realloc - #959
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
shrink_to_fit: leave shared containers alone, answer bytes, never lose an array to realloc
roaring_bitmap_shrink_to_fitandroaring64_bitmap_shrink_to_fitcould reallocate memory that another bitmap wasreading, reported a mix of units, and could free or drop an array when
reallocmisbehaved.(bug fix) Shrinking a copy-on-write copy no longer moves the arrays its source reads
container_shrink_to_fitunwrapped a shared container and reallocated its array in place, though every bitmapsharing it reads that array. The README's thread-safety section allows passing copy-on-write copies to different
threads, so a thread reading the source could read freed memory while another shrank the copy. A shared container is
now shrunk only when this bitmap is its sole holder (reference count 1), so a copy whose partners were freed still
gives its slack back on its next shrink.
(bug fix)
shrink_to_fitanswers bytesarray_container_shrink_to_fitanswered the values saved andrun_container_shrink_to_fitthe runs saved, thoughtheir headers, both bitmap functions and the C++
shrinkToFitpromise bytes. Both now answer bytes, so the 32-bit and64-bit totals no longer mix units.
(bug fix) Shrinking to nothing frees instead of calling
realloc(p, 0)realloc(p, 0). glibc frees the block and answers NULL, and theNULL branch then freed it a second time. It is now freed, as an empty array container already was.
now freed, as the tree's node arrays already were.
(bug fix) A failed shrinking
reallockeeps the arrayThe array and run containers freed their array and kept their count, and the 64-bit container arrays and the tree's
node arrays were overwritten with NULL. Each now keeps its larger block and answers no bytes for it. The 64-bit
bitmap and the tree still take the smaller capacity, which the larger block holds, so a later growth reallocates from
the block. The container, tree and 64-bit array shrinks and
bitset_trimgo through one internal helper,croaring_realloc_shrinkinmemory.h, which frees at 0 bytes and keeps the block whenreallocfails.(bug fix)
Roaring64Map::shrinkToFitcounts the arrays of the empty bitmaps it erasesIt added a fixed 88 bytes for each empty 32-bit bitmap it erased, though an emptied bitmap can still hold its key and
container arrays. It now adds what that bitmap's own shrink gives back, plus the size of the map's value type.
(perf) Shrinking a 32-bit bitmap that already fits allocates nothing
ra_shrink_to_fitallocated, copied and freed the key and container arrays even when they already fit, so everyshrink of a fitted 32-bit bitmap did that work to save no bytes. It now returns at once.
(test)
tests/threads_unit.cpp: a copy-on-write copy shrunk on one thread while another reads the source, for thethread sanitizer job.
tests/test_memory_hook.h: helpers installing a memory hook with the C library's aligned functions, reset in eachtest's cmocka teardown, and an allocator counting the blocks and bytes it holds and failing chosen allocations: the
shrink tests compare each answer with the bytes the allocator got back.
🤖 Written and verified with the aid of Claude Code; I have reviewed every line.