Repository navigation
Plan and execute common RowFn batches in one dispatch - #10675
connortsui20 wants to merge 4 commits into
Conversation
Merging this PR will degrade performance by 3.79%4 warnings, 1 note
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing Footnotes
|
55d9af7 to
178dcd8
Compare
178dcd8 to
5cbf00a
Compare
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.997x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.998x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.008x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.994x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (1.000x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.998x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.997x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.997x ➖, 0↑ 1↓)
duckdb / parquet / ns (0.984x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.995x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.997x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.000x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.997x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.996x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.993x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.994x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.997x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.005x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.004x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.922x ➖, 6↑ 2↓)
datafusion / parquet / ns (0.965x ➖, 1↑ 2↓)
duckdb / vortex-file-compressed / ns (0.961x ➖, 1↑ 0↓)
duckdb / parquet / ns (0.753x ➖, 10↑ 1↓)
|
Keep the owned decoder as the required extension point and default borrowed decoding to it. Native primitive inputs share their canonical buffers directly. Retain decoded storage after the input handle is dropped. Extracted from #9979. Signed-off-by: Robert Kruszewski <github@robertk.io> Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Inspect borrowed inputs once for lengths, dtypes, constant classification, and strict validity. Pair collected inputs with a separate execution plan while retaining the existing dispatch and retry paths. Extracted from #9979. Signed-off-by: Robert Kruszewski <github@robertk.io> Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Let the packed Boolean collectors consume the batch validity when constructing their result. Keep custom output validation before masking and attach validity directly to other canonical outputs. Extracted from #9979. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Finish eligible batches in the planning visit and retain the planned retry and valid-row paths for the remaining partially valid batches. Decode direct Boolean inputs through private borrowed tuple adapters and keep collector selection outside the row loop. Extracted from #9979. Enforce the Boolean validity-length invariant in release builds before unchecked output construction. Signed-off-by: Robert Kruszewski <github@robertk.io> Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
5cbf00a to
7c47d4b
Compare
Summary
Rebased onto
developatd02df8998010cb7ae5dc52d13f8dc333a02986e7. #10676 is open again and remains based on this PR.This draft still contains the borrowed-decoding, batch-inspection, and validity commits from #10672, #10673, and #10674 because the fusion implementation depends on them. Closing their standalone PRs did not remove those implementation dependencies.
Draft pending fresh performance evaluation. Fusion improved the migrated comparison path in controlled runs, but the complete stack also had ARM and AVX-512 regressions. The independent internal candidates are #10741 and #10742.
Extracts fused execution from #9979. A private visitor plans a concrete signature and finishes batches that need no retry in the same dispatch. Remaining partially valid batches retain dense retry, valid-row execution, and filtered fallback.
Changes
Keeps borrowed tuple decoding, direct Boolean collection, constant masks, and collector call boundaries inside the framework. The supported RowFn authoring API is unchanged. The packed collector checks validity length in release builds before unchecked output construction.
Retains coverage for nullable execution and output validation. The dispatch tests now reach a real second dispatch through partially valid deferred execution, and new coverage checks single-dispatch execution and rejection of a mismatched validity length.
The rebase preserves the execution changes and adapts added tests to the removal of
array_session. Source range-diff review shows only test-session setup changes relative to the original patches. Tests, formatting, and benchmarks were not rerun on the rebased head. Earlier native measurements remain historical evidence.