Repository navigation
Conversation
…stripes Expose streaming batches from complete ORC files or selected stripes. Reuse native decoding through a stateless stripe reader factory and preserve row counts for zero-column projections. Add validation, lifecycle, range-reading and memory regression coverage. Generated-by: OpenAI Codex Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
|
|
Generate shared fixture data with Python integers and isolate NumPy integer arguments in a marked test. Cover whole-file and selected-stripe iterator lifetimes across all five codecs. Generated-by: OpenAI Codex Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
Merge upstream main at d24791f. Retain batch iteration and the removal of legacy ORC writer argument-order compatibility. Generated-by: OpenAI Codex Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
Remove a redundant column-list copy and reduce duplicate test combinations while preserving argument snapshots, codec coverage, and both iterator lifecycle paths. Generated-by: OpenAI Codex Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
| batches.reset(); // Early stop must not read the rest or another stripe. | ||
| ASSERT_EQ(reads, tracked->num_reads()); | ||
| } | ||
| for (const auto& range : tracked->get_read_ranges()) { |
There was a problem hiding this comment.
ResetStats() is outside the loop and the final check only excludes stripes 1 and 3. As a result, reading stripe 0 while selecting stripe 2, or vice versa, would still pass. Could we reset inside each iteration and compare every read range against all stripes except the one currently selected?
There was a problem hiding this comment.
ResetStats() now runs inside each iteration, and every read range is checked against all stripes except the selected one. This catches unintended reads between stripes 0 and 2 as well. The updated test passes for NONE and ZLIB, including the early-close assertion.
Use optional column selection, strengthen per-stripe read-range assertions, and explicitly reject mixed Python column selectors. Generated-by: OpenAI Codex Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
dmitry-chirkov-dremio
left a comment
There was a problem hiding this comment.
LGTM.
Selection and reader lifetime are covered well.
Memory numbers hold up in local tests too.
…tion Validate mixed column selectors in the shared ORC helper, remove redundant checks and test diagnostics, and cover all three read APIs. Generated-by: OpenAI Codex Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
Trigger a new CI run after unrelated infrastructure failures. Generated-by: OpenAI Codex Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
|
The three CI failures appear unrelated to this PR:
Once the RTools issue is resolved on main, I'll update this branch and check CI again. |
|
The R failure is currently tracked here: The other two seem to be just flaky transient errors. |
Merge upstream main to include the RTools 40 CI fix while preserving the ORC batch reader changes. Generated-by: OpenAI Codex Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
|
I’ve merged main into this branch to pick up the Windows RTools 40 download fix in #55378. I also verified the corrected archive URL in a local Windows 11 ARM VM: a 1 KiB range request returned HTTP 206 with the expected |
|
These two seem to be just flaky transient errors, too. |
|
My approval still stands. |
raulcd
left a comment
There was a problem hiding this comment.
My concerns were solved with the latest changes. I am not an expert on this area but I don't have more concerns
rok
left a comment
There was a problem hiding this comment.
I did a quick 🤖 assisted review. Nothing seems critical in Python and I don't know orc well enough to comment on the orc logic itself :)
| # None selects a whole-file reader. Zero-column projection needs | ||
| # individual stripe readers to preserve row counts. | ||
| if stripes is None and (columns is None or len(columns) != 0): | ||
| stripe_indices = [None] |
There was a problem hiding this comment.
I think we can drop the whole-file branch and the None sentinel, and just use range(self.nstripes()) when stripes is None.
liborc batches never cross stripe boundaries, so the output is the same either way: a dataset scan (which uses GetRecordBatchReader) of a file with 4 stripes of 16384 rows at batch_size=65536 yields [16384, 16384, 16384, 16384].
That removes the sentinel, the names / selected_names duplication, the zero-column special case and the GetRecordBatchReader declaration in _orc.pxd. It also means the batch_size clamp suggested in GetStripeReader covers every path.
There was a problem hiding this comment.
I kept the whole-file reader for stripes=None when reading all columns or using a non-empty projection. Each liborc RowReader initialization walks the file's stripe metadata, so creating one per stripe would make that work quadratic for a full scan. The whole-file path also preserves invalid-column validation for empty files. I capped native batch capacity on both paths.
| return Status::OK(); | ||
| } | ||
|
|
||
| if (schema_->num_fields() == 0) { |
There was a problem hiding this comment.
Preserving the row count is the right behaviour (it matches Parquet), but it makes columns=[] mean different things across the API. On 26.0.1:
>>> f.read(columns=[]).num_rows
0
>>> f.read_stripe(0, columns=[]).num_rows
0while iter_batches(columns=[]) keeps the rows. The docs say column selection "follows read_stripe", and test_iter_batches_projection has to special-case [] because of this.
Should we apply the same short-circuit in ReadBatch so all three agree?
There was a problem hiding this comment.
Thanks for pointing this out. iter_batches(columns=[]) preserves the row count, while ORCFile.read(columns=[]) and read_stripe(i, columns=[]) currently return zero rows. I clarified that the column selection syntax matches read_stripe. Since #54921 proposes keeping the existing read APIs' behavior unchanged, would you be comfortable addressing their zero-column row counts in a separate follow-up?
| self.reader.get().GetStripeReader( | ||
| stripe, batch_size, selected_names)) | ||
| while True: | ||
| if self.source_handle.get().closed(): |
There was a problem hiding this comment.
I'd remove this check, together with the source_handle member and test_iter_batches_closed_source_with_buffered_rows.
Reading from a closed source already raises as soon as liborc needs I/O (read_stripe after close raises OSError today). This only adds an error for rows liborc has already buffered, which is harmless, and no other pyarrow reader does it (ParquetFile.iter_batches doesn't). It also raises ValueError where the real I/O path raises OSError, which is why test_iter_batches_errors_after_output has to accept three exception types.
(Agree with your reply to the earlier thread though: .closed is already required at open, so this isn't a new prerequisite.)
There was a problem hiding this comment.
Removed the per-batch closed-source check, source_handle and the buffered-rows test. The ownership and early-close tests remain. The read-failure test now covers closing the source after the first batch and then attempting to read a later stripe on the explicit-stripe path.
| ... batches.close() | ||
| Closing the iterator does not close a file object supplied by the caller. | ||
| Returned batches remain valid. ``batch_size`` bounds the number of output rows, |
There was a problem hiding this comment.
Nit: this paragraph could be shorter. I'd keep "closing the iterator does not close the source" and one sentence saying batch_size limits rows rather than memory, and drop the rest.
There was a problem hiding this comment.
Shortened the paragraph to retain the iterator and source ownership behavior, and the distinction between the batch row limit and total memory.
…y iteration Generated-by: OpenAI Codex Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
Rationale for this change
ORCFile.read_stripe()materializes an entire stripe as one record batch. Dataset already scans ORC in batches; this adds native batching with explicit stripe selection toORCFile, supporting incremental consumption and early termination.Closes #54921.
What changes are included in this PR?
ORCFile.iter_batches(batch_size=65536, stripes=None, columns=None)with column projection.GetStripeReaderAPI with explicit optional column selection that decodes directly into batches without changing the reader's Seek position.Are these changes tested?
Validated locally on macOS arm64 with matching Release C++/Python builds, Python 3.13.7 and ORC 2.2.1.
44 passed; one existing timezone-database test restricted to ORC < 2.0 was skipped.
From
python/:134 passed with NumPy 2.5.4. In an isolated environment without NumPy, the
iter_batchesand mixed-column-selection tests had 113 passes; the marked NumPy integer-input test was skipped. Coverage includes five codecs, stripe order and duplicates, batch limits, nested/zero-column projections, invalid arguments, interleaved iterators and read failures. Both environments cover owner lifetime, early close and resource release for whole-file and selected-stripe iteration.The C++ memory regression uses four int64 columns and batches of 256 rows. For single stripes of 262144 and 1048576 rows, it checks that streaming Arrow allocation peaks stay below one quarter of whole-stripe peaks. A local malloc probe on a three-row file with a batch size of 8000000 observed no malloc calls of 1 MB or more for whole-file, selected-stripe and zero-column iteration. NONE/ZLIB range tests verify that consuming the first batch and closing early does not read unselected stripe data.
Changed-file format/lint checks and
git diff --checkpassed.Are there any user-facing changes?
Yes. The new API yields
RecordBatchobjects.stripes=Nonereads all stripes; explicit sequences preserve order and duplicates, and empty sequences yield no batches.columns=[]preserves row counts. Closing the iterator releases its reader without closing the input; returned batches remain valid.batch_sizelimits output rows, not total process memory. ORC buffers and dictionaries require additional memory, and small batches can reduce throughput. Reading the same stripes does not inherently reduce I/O.Was AI used for this PR?
PR code and description written by:
Reviewed before submission by:
The human contributor directed the design and reviewed the proposal and PR text. OpenAI Codex assisted with code, tests, documentation and static review.