[XiangpengHao]

Summary

• Stacked on feat: make the dictionary size cap configurable and raise its default to 2 MiB #10766 (the dictionary cap, raised to 2 MiB); its commit is this branch's first until it lands.

A proof of concept, not meant to merge as is. It shows what Vortex gains by reading its files through t4, a cache of an object store in memory and on local NVMe, and asks which parts, if any, you'd want.

t4 keeps an object store's files in 1 MiB blocks, in memory and in cache files on local SSDs, which it reads with direct I/O. On a miss it fetches from the store, racing straggling GETs. The integration is thin: t4 exposes "open a file once, then read ranges of it", and Vortex wraps that as a VortexReadAt. Vortex's crates use t4 only with the new t4 feature on (the benchmark binary always does), and t4 never depends on Vortex.

On Vortex's own datafusion-bench (TPC-H SF10 and ClickBench q20–q34), on three AWS instance types:

• Local SSDs: t4 reading from its SSD tier takes 9–16% less time than Vortex reading the same files from the same SSDs, with 6–15% less CPU.
• Remote: reading straight from S3, t4 cuts TPC-H's slowest runs by 18%. It doesn't speed up ClickBench (×1.01), where it fetches 1.48× the bytes.
• Memory: t4's memory tier ties Vortex reading from the page cache.

The design and every run are written up in t4's doc/09.

Changes

1. feat: make the dictionary size cap configurable and raise its default to 2 MiB is feat: make the dictionary size cap configurable and raise its default to 2 MiB #10766. The benchmark files were written with it.
2. feat: read Vortex files through t4 (vortex-io t4 feature) adds vortex_io::t4::T4ReadAt, a VortexReadAt over t4's CachedFile.
   • t4 answers each read with one buffer whose address agrees with the file offset modulo 4 KiB, so Vortex's aligned segments take it without a copy.
   • Vortex's coalescing of nearby reads stays its own (with_coalesce_config, the object-store config by default).
   • vortex-io re-exports T4Caching, T4CachingBuilder and CachedFile, so the cache is built with the same t4 Vortex links. vortex-file and vortex forward the feature.
   • Tests: vortex-io unit tests cover each read's bytes and alignment, a fetch then a hit, a batch, and the size. vortex-file/tests/t4_read_at.rs writes a Vortex file of over 2 MiB to an in-memory store and scans it back through T4ReadAt twice: fetched, then from t4's tiers, with Vortex's coalescing and without. It compares the arrays read with those written.
3. feat: VortexFormat takes a reader factory; datafusion-bench reads through t4 adds VortexFormat::with_vortex_reader_factory. The factory is used for schema and statistics inference and handed to every VortexSource.
   • This is a generic hook for any reader, not only t4, and could go in first, on its own.
   • It has a test that a listing table infers its schema and scans through the factory.
   • datafusion-bench's --opt t4-policy=... puts t4 in front of the benchmark's store through this hook.
4. chore: format VortexFormat's reader factory; drop a lint expectation the t4 test never meets: nightly rustfmt, and an unmet clippy expectation.
5. feat(datafusion-bench): measure tiers apart; take t4 by git: bench options, used for the numbers below:
   • evict-page-cache=on: drops the data files' pages (posix_fadvise(DONTNEED)) before every run. Vortex then reads from the SSDs in a warm process, as t4 does, without paying a fresh process's ~400 ms first run.
   • cpu-log=<path>: each run's process CPU time, and the bytes the md RAID read.
   • get-log=<path>: each run's GETs: count, bytes, the slowest.
   • rest-runs=<n>,rest-ms=<ms>: rests before runs 1 to n (see "The SSD burst" below).
   • t4-policy, t4-memory-gib, t4-disk, t4-disk-gib, t4-coalesce: t4's tiers. t4-policy=memory|disk|none pins every block to one tier, so a run measures one tier alone.
   • The runner now checks every run's row count against the first, not only the first run's, since a cache serves the later runs.

API Changes

Additive only:

• vortex-io's t4 feature and T4ReadAt, forwarded by vortex-file and vortex.
• VortexFormat::with_vortex_reader_factory.

Nothing changes with the feature off.

Results

Local SSDs, 28 Oct 2026

All of each box's instance-store SSDs as one md RAID0 (XFS), on the box's defaults. Vortex reads its files there with the page cache emptied before every run. t4 reads its cache file on the same RAID, filled from S3 by a first run and pinned to its disk tier.

Each query runs in one process per reader: a first run (not counted), five runs each after a 2 s rest, then twelve back to back. The figure is the median of the last five, summed over the suite, t4's over Vortex's (below 1, t4 is faster).

Box (single socket)	TPC-H time	slowest runs	CPU	ClickBench time	slowest runs	CPU
i3en.2xlarge: 4 cores, 2 SSDs, 1.4 GB/s	×0.84	×0.85	×0.93	×0.91	×0.91	×0.94
i7ie.12xlarge: 24 cores, 4 SSDs, 12.7 GB/s	×0.87	×0.85	×0.86	×0.88	×0.90	×0.89
i7i.24xlarge: 48 cores, 6 SSDs, 26.9 GB/s	×0.87	×0.77	×0.88	×0.86	×0.86	×0.85

t4 from the SSDs against Vortex from memory (the page cache): ×0.95 to ×1.05.

Remote and memory, 27 Oct 2026

An i7i.24xlarge in us-east-1, the data in S3 in the same region, each query in a process per reader, five runs. Geometric means over the suite of each query's median and slowest run. t4's merging of waiting GETs was still off then; it is on by default now.

t4 against	TPC-H	ClickBench q20–q34
Vortex straight to S3	slowest ×0.82, median ×0.81	slowest ×1.01, median ×1.03
Vortex from the page cache (t4's memory tier)	slowest ×0.96, median ×0.99	slowest ×0.94, median ×0.96

The slowest runs include each process's first, which also reads the footers and opens connections. Without them, the remote row holds: TPC-H slowest ×0.80, median ×0.82; ClickBench slowest ×0.97, median ×1.05.

• t4 removes S3's stragglers. The slowest GET over the suite drops from 823 to 256 ms on TPC-H, and from 3,244 to 314 ms on ClickBench, where 707 of S3's GETs took over 1 s and none of t4's did.
• On ClickBench, t4 fetches 1.48× the bytes, whole 1 MiB blocks around Vortex's ~1.9 MB reads. That is why its scans there aren't faster from S3.

Things to know when reading these

• The SSD burst. AWS's instance-store SSDs read several times faster than their published rate for a moment after a rest, then hold it. An i3en.2xlarge's two drives read 6.3 GB/s for about 0.4 s, then exactly AWS's 1.3 GB/s and 170K IOPS. That's why the table reports back-to-back runs after the burst is spent. The rested runs ran but aren't reported.
• Readahead. Vortex reads with buffered pread at the kernel's default readahead. On an md RAID0 that default is twice the stripe (2 MiB on the i3en), which doubles the bytes Vortex reads there (TPC-H q6: 825 MB a run, against the 400 MB its reads ask for). t4's direct reads don't read ahead. These numbers keep the defaults, as a user would run them.
• t4 is pinned to its disk tier here, with nothing kept in memory. A query that reads a table more than once (TPC-H q15, q18, q21) reads it from the SSDs each time, while Vortex's later passes hit the page cache. This counts against t4.
• ClickBench q23, a top-10 query, reads different blocks run to run. t4's tier holds what its first run fetched, so its later runs fetch the rest from S3, up to 0.8 GB a run. This counts against t4 too, and t4 is still faster on it.
• Two boxes of one type can differ by 6–20% in CPU speed. Every ratio here compares readers on the same box.

Dependencies

t4 is unpublished, so it is taken by git, pinned at XiangpengHao/t4@0cb2fbf. It brings bbtree, vstd and five Verus crates into the lockfile, and vstd is pinned to an exact version. That needs settling before anything merges: t4 should keep Verus out of a normal build.

How to reproduce

Build the bench (t4 is on in its vortex dependency):

cargo build --release -p datafusion-bench --bin datafusion-bench

Set up the data:

• The files are the bench's own (TPC-H SF10 in 16 files, ClickBench partitioned in 100), written with feat: make the dictionary size cap configurable and raise its default to 2 MiB #10766's 2 MiB cap.
• Vortex reads a local copy under vortex-bench/data on the RAID. t4 reads the same files from S3 through remote-data-dir.
• Make the RAID from the instance-store drives (device names vary by box):

sudo mdadm --create /dev/md0 --level=0 --raid-devices=4 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1
sudo mkfs.xfs -f -K /dev/md0 && sudo mkdir -p /mnt/raid && sudo mount -o noatime /dev/md0 /mnt/raid

Run one process per query and reader, after sync; echo 3 | sudo tee /proc/sys/vm/drop_caches. Here TPC-H q6, 18 runs: a first, five rested, twelve back to back.

B=target/release/datafusion-bench
REST=rest-runs=5,rest-ms=2000,cpu-log=cpu.jsonl
# Vortex from memory (the page cache)
$B tpch --formats vortex -q 6 -i 18 --opt scale-factor=10.0,$REST
# Vortex from the SSDs, its files' pages evicted before every run
$B tpch --formats vortex -q 6 -i 18 --opt scale-factor=10.0,$REST,evict-page-cache=on
# t4 from its disk tier on the same SSDs, filled from S3 by the first run
$B tpch --formats vortex -q 6 -i 18 --opt scale-factor=10.0,$REST,remote-data-dir=s3://<bucket>/tpch/10.0/,t4-policy=disk,t4-memory-gib=64,t4-disk=/mnt/raid,t4-disk-gib=64,get-log=gets.jsonl

ClickBench is the same with clickbench ... --opt flavor=partitioned,... and remote-data-dir=s3://<bucket>/clickbench_partitioned/. On the i3en.2xlarge (64 GiB), t4-memory-gib=8; with t4-policy=disk the memory tier holds nothing. Report the median of the last five runs' wall_ms in cpu.jsonl.

What I'd like to know

• Is the reader-factory hook on VortexFormat welcome on its own? It lets any VortexReadAt serve a DataFusion table.
• Would an optional t4 feature in vortex-io be acceptable, or would you rather the adapter live in a separate crate?
• Not done yet: DuckDB (a T4FileSystem), random access by URL, and t4's default (unpinned) placement across memory and SSD.

[codspeed]

Merging this PR will degrade performance by 3.15%

2 warnings

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚠️ Unable to measure the execution of 1 benchmark

This benchmark measured no execution time, so its result is not comparable and counts as unchanged. This usually happens when the compiler removes the code under test.

Prevent compiler optimizations

See affected benchmarks

⚡ 1 improved benchmark
❌ 1 regressed benchmark
✅ 2272 untouched benchmarks
⏩ 370 skipped benchmarks¹
🗄️ 30 archived benchmarks run²

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌	CPU Simulation	filter_powerlaw_by_correlated_runs[100000]	51.2 µs	61 µs	-15.96%
⚡	Walltime	lt_i64_nullable_avx512	8.1 µs	7.3 µs	+11.63%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing XiangpengHao:t4-read-at (9cbee84) with develop (40e0b1c)}

Footnotes

1. 370 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

2. 30 benchmarks were run, but are now archived. If they were deleted in another branch, consider rebasing to remove them from the report. Instead if they were added back, click here to restore them. ↩