Repository navigation
Conversation
A table scan that selects (nearly) all field columns of a device previously deserialized the device's TimeSeriesMetadata column by column, re-walking the device metadata index once per leaf node. Add a batch path that reads the whole device's metadata in a single pass via TsFileSequenceReader.readDeviceMetadata, populating the per-measurement cache and building the aligned metadata through the existing modification/loader logic. The batch path is taken when the selected field columns reach a configurable fraction (device_metadata_batch_scan_ratio, default 0.8) of the table's total field columns.
The batch path read a device's TimeseriesMetadata via TsFileSequenceReader.readDeviceMetadata, which deserializes only the header and statistics and leaves chunkMetadataList null. The downstream DiskAlignedChunkMetadataLoader dereferences that list in getCopiedChunkMetadataList, so a wide-table scan NPE'd at query time, and the null-chunk entries also polluted the shared per-measurement LRU cache (breaking later narrow queries on the same device). Switch to getDeviceTimeseriesMetadata, which deserializes chunk metadata too, and adapt the unit test to the list-based API, asserting that chunk metadata flows through to the result.
Remove the unreachable time-column-only branch in the batch loader (the batch gate already requires a non-empty selection), and validate device_metadata_batch_scan_ratio so a non-positive value falls back to the default instead of routing every table scan through the batch path. Document the valid range in the properties template.
Remove the ignoreNotExists gate so the batch path is also taken for files with a device-level time index, where previously only file-level-index files (or recovery) used it. When the batch read finds the device is missing from the file, replicate the per-column path: skip silently when ignoreNotExists is set, otherwise throw CorruptedTsFileException to surface the time-index/metadata inconsistency.
Remove the device_metadata_batch_scan_ratio config parameter and hardcode the threshold at 0.6 (previously configurable, default 0.8) so the batch path covers a wider set of scans without requiring configuration.
Serialize concurrent reads of the same device+file in getDeviceTimeSeriesMetadata (matching the per-column get) so threads don't each deserialize the whole device and race on the shared reader. Also attach the missing device id to the CorruptedTsFileException thrown when a device is absent from a device-level time index file.
Count every value column (FIELD + ATTRIBUTE) as the batch scan ratio denominator instead of only FIELD columns, matching the selected-column numerator. Cache empty (count == 0) series as the null placeholder in the batch read, matching the per-column get, so later per-column reads don't re-read them. Add a unit test for the absent-device (empty map) path.
Cover the loadAlignedTimeSeriesMetadataBatch empty-map branch: a device absent from a device-level time index file throws CorruptedTsFileException, while with a file-level time index (or during recovery) it is skipped.
Drive AlignedSeriesScanUtil with all value columns selected and a matching total field column count so the batch path is taken, and assert the aligned pages are still read correctly.
The batch read offers no benefit for a single-column device (one leaf either way) and it bypasses the per-column cache's debug logging, which IoTDBDebugQueryIT.tableTest relies on. Require at least two value columns before taking the batch path.
getDeviceTimeSeriesMetadata always deserialized the device's whole metadata index and then re-populated the cache, never reading it, so repeated wide-table scans were slower than the per-column path that it populates. Check the per-measurement cache first, read the device only when one of the requested measurements is missing, and cache just those measurements (plus the null placeholders for absent ones). Align the absent-device behavior with the per-column get(): skip the device when the file's bloom filter proves it is missing, otherwise raise the read IOException so the existing wrapper records the suspicious-disk metric. Require a null device offset for the batch path instead of relying on the zero sentinel used by external scans, and extract the aligned metadata assembly shared by both paths. Count only FIELD columns in the ratio denominator to match the smaller numerator, and extend the tests to cover the cache hit, the bloom-filter decisions, the gating boundary and the batch path actually being taken.
shuwenwei
reviewed
Nov 12, 2026
shuwenwei
reviewed
Nov 12, 2026
The batch metadata scan bypasses TimeSeriesMetadataCache.get(..., debug=true) and does not emit equivalent hit/miss diagnostics, so a DEBUG SELECT on a wide table lost its metadata logs. Make the batch gate take the debug and offset-addressed state explicitly so DEBUG queries keep the per-column path, and cover both conditions in the gating unit test.
getDeviceTimeSeriesMetadata read the whole device metadata index and only then discovered the device was absent, while the per-column get() can skip that read by probing the file's bloom filter first. Probe the cached time column entry before the read, keep the same missing-device skip/throw rules, and account the bloom filter cache/disk/time statistics like get().
JackieTien97
left a comment
JackieTien97
left a comment
Contributor
There was a problem hiding this comment.
Thanks for the optimization. I checked out 3fba980 and ran the PR's tests locally (FileLoaderUtilsTest, TimeSeriesMetadataCacheTest, AlignedSeriesScanBatchTest); all pass. I also added a throwaway probe test (not pushed) that writes a real aligned device with 700 columns (three measurement leaf groups) and compares getDeviceTimeSeriesMetadata against the per-column get(): all 701 series (time column plus 700 fields) matched, including statistics and chunk-metadata counts. So I found no wrong-result issue. The comments below are mostly about performance, diagnostics and metrics.
Main points:
- [P1] The batch path only counts as a cache hit when every selected sensor is in the LRU. If any selected column is absent from a file and that absence can't be cached as a placeholder (
ignoreNotExists=trueormay_cache_nonexist_series=false), every query re-reads the whole device. The per-column path doesn't (TimeSeriesMetadataCacheL362). - [P1] The missing-device error message describes the opposite inconsistency (L491).
- [P2] Metadata IO size and load counts are no longer comparable with the per-column path (L483, L370).
- [P2] The I/O saving only exists when the selected columns span more than one measurement leaf group (more than 256 series by default), and the gate doesn't reflect that (
FileLoaderUtilsL447). Could the same batching be done inside the existingget()miss path instead of a parallel path (L342)? - The PR description is out of date. It still describes a configurable
device_metadata_batch_scan_ratio(default 0.8), anIoTDBConfigchange, anignoreNotExistsgate andreadDeviceMetadata. The code hard-codes 0.6 with no switch, has no such gate, and usesgetDeviceTimeseriesMetadata. Could you update it and add benchmark numbers for a wide device (e.g. 800 columns, cold and warm cache) compared with master? That would make it easier to judge the 0.6 threshold.
shuwenwei
reviewed
Nov 13, 2026
Reuse cached measurements and batch-read only the missing ones, probing the per-measurement bloom filter first so absent columns do not force a whole-device read; the all-or-nothing cache check is gone. Count load statistics per series so they stay comparable with the per-column path, and document the batch read's missing IO-size accounting. Report a dedicated error naming the device and the TsFile when the resource records a device that the file metadata lacks (en + zh), share the load statistics recording between both paths, and pass the external flag through the bloom filter lookup. Only batch when the table has more field columns than the measurement index fan-out, so small devices stay on the per-column path. Add a real 700-column batch-vs-per-column comparison, a 300-column end-to-end batch scan spanning two leaf groups, and gate boundary cases.
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved batch-path issues can trigger unnecessary device reads and absent-device I/O errors, while regression tests leave sparse all-null rows and multi-leaf field values unverified.
5 open issues
What changed in this PR
This PR reduces repeated metadata-index traversal in DataNode wide-table queries by loading a device’s metadata in one batch when most field columns are selected.
Changes:
- Adds cache- and bloom-filter-aware batch loading with per-series query statistics.
- Gates batching by table width and a 60% projection threshold, excluding DEBUG and offset-addressed scans.
- Shares aligned metadata assembly, adds localized missing-device errors, and introduces regression tests.
| File | Description |
|---|---|
iotdb-core/datanode/src/test/java/org/apache/iotdb/db/storageengine/dataregion/read/reader/series/AlignedSeriesScanBatchTest.java |
Adds a real-file wide-device batch scan regression. |
iotdb-core/datanode/src/test/java/org/apache/iotdb/db/storageengine/buffer/TimeSeriesMetadataCacheTest.java |
Tests batch loading, caching, missing devices, and metadata equivalence. |
iotdb-core/datanode/src/test/java/org/apache/iotdb/db/queryengine/execution/operator/source/FileLoaderUtilsTest.java |
Tests batch eligibility boundaries and exclusions. |
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/buffer/TimeSeriesMetadataCache.java |
Implements cache-aware device metadata loading and statistics accounting. |
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/planner/plan/parameter/SeriesScanOptions.java |
Carries the table’s total field-column count. |
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/planner/DataNodeTableOperatorGenerator.java |
Supplies field counts for eligible table scans. |
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/source/relational/ExternalTsFileSeriesScanUtil.java |
Keeps external offset-addressed scans on the per-column path. |
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/source/FileLoaderUtils.java |
Selects the batch path and shares aligned metadata assembly. |
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/source/AlignedSeriesScanUtil.java |
Passes the field count into metadata loading. |
iotdb-core/datanode/src/main/i18n/zh/org/apache/iotdb/db/i18n/StorageEngineMessages.java |
Adds the Chinese missing-device error message. |
iotdb-core/datanode/src/main/i18n/en/org/apache/iotdb/db/i18n/StorageEngineMessages.java |
Adds the English error naming the device and TsFile. |
Review effort: Max
Skip the device read when the time column is known absent, so a bloom filter false positive on a field cannot force an absent-device read. Fall back to per-column lookups for a small remainder (one or two measurements), which reads a single leaf group instead of the whole device, and count the cold-call cache probe and lock wait in the disk lookup time. Strengthen the wide-scan tests: assert timestamps and every field value, compare a sparse batch scan against the per-column path, cover an all-null projection under ignoreAllNullRows, and add cache regressions for the filtered time column and the single-miss fallback.
shuwenwei
reviewed
Nov 13, 2026
TsFileSequenceReader is an external apache/tsfile dependency and has no batch API that accepts a measurement set, so the batch path still reads the whole device and filters afterwards. Note the constraint and the existing mitigations (bloom filter, per-column fallback) at the call site; a truly selective reader needs an upstream change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Wide-table scans currently load a device's
TimeseriesMetadatacolumn by column: each cache miss walks the device metadata index and deserializes the measurement leaf group that contains the requested column. The index traversal is independent of the column, so a scan selecting almost all columns pays it once per column. In the motivating workload (one device with ~700 columns, ~20 written per day, 12 files) the columns absent from each file each triggered a full index walk plus leaf-group deserialization, which dominated query latency.This PR adds a device-level batch path for table scans that select most of the table's field columns.
Content 1 — cache-aware batch read
TimeSeriesMetadataCache.getDeviceTimeSeriesMetadata(...):TsFileSequenceReader#getDeviceTimeseriesMetadata(chunk metadata included), and caches only the requested measurements;LoadTimeSeriesMetadataFromCacheCount/FromDiskCount), soEXPLAIN ANALYZEstays comparable with the per-column path. The batch read's byte count cannot be reported becausegetDeviceTimeseriesMetadatahas no IO-size recorder.Content 2 — when the batch path is used
Only for
DeviceTableScanNodescans (not tree-view, external, aggregation, or offset-addressed scans), when:max_degree_of_index_node(a device whose series fit in a single measurement leaf group already reads them all in one per-column lookup);DEVICE_METADATA_BATCH_SCAN_RATIO) of the table's field columns;The 0.6 threshold is a hard-coded conservative default; the earlier
device_metadata_batch_scan_ratioconfig was intentionally removed, so there is currently no runtime off switch.Behavior
get(). The error now names both the device and the TsFile.Tests
TimeSeriesMetadataCacheTest.testBatchMetadataMatchesPerColumn: writes a real 700-column aligned TsFile and compares the batch result with per-columnget()for every series, the time column and an absent column (statistics and chunk-metadata counts).AlignedSeriesScanBatchTest: writes a real 300-column device (two measurement leaf groups) and verifies the end-to-end scan takes the batch path (no per-column cache lookups) and reads the expected rows.FileLoaderUtilsTest: covers the gating boundary (fan-out, ratio, DEBUG, offset-addressed).Known follow-ups
TsFileSequenceReader#getDeviceTimeseriesMetadatareads the device metadata region and then re-reads each child leaf (about 2x IO). A measurement-selective batch API in apache/tsfile would let the batch path visit only the leaf groups that overlap the requested measurements; that requires an upstream change and could also make the ratio gate unnecessary. The measurement-levelreadTimeseriesMetadataAPI is already group-selective and is what the per-column path and the small-remainder fallback use.