Repository navigation
Conversation
Also add total_bytes_allocated and num_allocations methods on memory pools. Unlike the existing memory pool types, a ProxyMemoryPool is non-static so must be kept alive while objects that use memory allocated through it remain alive. This required adding some extra book-keeping.
There was a problem hiding this comment.
🟡 Changes recommended
Critical pool-ownership gaps in properties getters and derived readers, streams, and batches allow live native consumers to outlast the owning proxy, causing use-after-free.
6 open issues
Imported batches can outlive the file reader's pool proxy · New BufferReader does not retain the buffer's pool proxy · New ResizableBuffer stream drops pool ownership and omits it from Finish() · New ParquetFileWriter properties getter fails to retain the pool proxy · New ReaderProperties.MemoryPool drops the pool proxy reference · New WriterProperties.MemoryPool drops the pool proxy reference · New
What changed in this PR
Adds a managed ProxyMemoryPool wrapper for tracking native allocations independently per writer, implementing part 1 of #729. The PR also exposes allocation counters and adds pool-lifetime retention to dependent objects.
Changes:
- Adds disposable proxy pools and native bindings for cumulative allocation statistics.
- Retains pool references in properties, readers, writers, and buffers.
- Tests proxy statistics, nested pools, independent writer accounting, and lifetime behavior.
| File | Description |
|---|---|
| csharp/WriterPropertiesBuilder.cs | Passes the configured pool to built properties. |
| csharp/WriterProperties.cs | Retains the pool alongside the native properties handle. |
| csharp/ReaderProperties.cs | Retains the configured reader pool. |
| csharp/PublicAPI.Unshipped.txt | Declares the proxy pool and allocation counters. |
| csharp/ProxyMemoryPool.cs | Implements disposable proxy ownership using a safe handle. |
| csharp/ParquetFileWriter.cs | Retains the configured pool through native writer release. |
| csharp/ParquetFileReader.cs | Retains the configured pool through native reader release. |
| csharp/MemoryPool.cs | Adds allocation counters and lifetime-retention helpers. |
| csharp/IO/ResizableBuffer.cs | Passes pool ownership to the buffer wrapper. |
| csharp/IO/BufferOutputStream.cs | Passes its stored pool to finished buffers. |
| csharp/IO/Buffer.cs | Retains the pool through native buffer release. |
| csharp/Arrow/FileWriter.cs | Retains the configured pool for Arrow writers. |
| csharp/Arrow/FileReader.cs | Retains the configured pool for Arrow readers. |
| csharp.test/TestProxyMemoryPool.cs | Tests accounting, ownership, and disposal behavior. |
| cpp/MemoryPool.cpp | Exports proxy creation, deletion, and allocation counters. |
Review effort: Balanced
This allow allocations to outlive the managed pool without risk of use-after-free.
This ensures that the pool isn't freed before use.
adamreeve
force-pushed
the
proxy-mem-pool
branch
from
November 3, 2026 01:22
9123a5a to
0390d44
Compare
There was a problem hiding this comment.
🟡 Changes recommended
A critical use-after-free risk remains because both buffer constructors pass the pool’s raw pointer to native code without keeping memoryPool alive through construction; both need GC.KeepAlive(memoryPool).
1 open issue
6 resolved since last review
WriterProperties.MemoryPool drops the pool proxy reference ReaderProperties.MemoryPool drops the pool proxy reference ParquetFileWriter properties getter fails to retain the pool proxy ResizableBuffer stream drops pool ownership and omits it from Finish() BufferReader does not retain the buffer's pool proxy Imported batches can outlive the file reader's pool proxy
Review effort: Balanced
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Adds a wrapper for the Arrow C++
ProxyMemoryPoolclass, and also exposes the nativetotal_bytes_allocatedandnum_allocationsmethods on memory pools.This implements part 1 of #729.
Unlike the existing memory pool types, a
ProxyMemoryPoolis non-static so must be kept alive while objects that use memory allocated through it remain alive. This required keeping references to the memory pool in things that depend on it.In order to prevent use-after free when native memory allocations are freed, a native
RefCountedProxyMemoryPoolclass was added that deletes itself only after all allocations made through it have been freed.