Skip to content

[NIDX-01 workstream] Replace bounded read-only index access #14002

Closed
5 of 5 issues completed
Closed

Description

@hanahmily

[NIDX-01 workstream] Replace bounded read-only index access

Parent: #13990

Tracking parent — do not apply Backlog. The TDD workflow selects leaf issues only. This workstream closes after all five ordered children merge.

Why this workstream is split

A single issue previously combined generation open, segment count, deletion masks, stored chunks, dictionaries/postings, doc values, search-after, fixtures, and six production callers. That requires roughly the complete ICE read path and cannot freeze one behavioral seam plus RED/e2e tests in one TDD contract turn.

The children are sliced by fixture complexity against live callers, not by dormant format layers. Every child:

  • adds at most one new ICE read capability;
  • switches a named production caller in the same merge;
  • owns independently generated compatibility-writer fixture bytes and literal expected values;
  • exposes one BanyanDB-owned behavioral seam;
  • writes no index bytes; and
  • is independently reversible without converting a directory.

End-state boundary

After all children merge, these direct Bluge reads are native:

  • inverted.ReadOnlyDocCount;
  • Measure and Stream migration verification counts;
  • Measure index-mode copy and series-union source walks;
  • metadata/schema/reader.WalkShard and WalkDocs; and
  • Property repair tuple pagination/tree construction.

The complete bounded surface is committed-generation open, visible count, match-all stored walk, exact term/OR, deletion masks, required scalar decoding, and ascending Property repair search-after. Writers, arbitrary queries, and lifecycle behavior remain outside NIDX-01.

Ordered TDD leaves

  1. #14008 — NIDX-01A: baseline single-segment read-only count
  2. #14009 — NIDX-01B: committed generations and deletion masks, blocked by [NIDX-01A] Cut over baseline read-only document count #14008
  3. #14010 — NIDX-01C: stored document walks for migration sources, blocked by [NIDX-01B] Count committed generations with deletions #14009
  4. #14011 — NIDX-01D: exact-term schema walks, blocked by [NIDX-01C] Walk stored documents for migration source reads #14010
  5. #14012 — NIDX-01E: Property repair search-after, blocked by [NIDX-01D] Cut over exact-term schema walks #14011

Only the oldest unblocked open leaf may carry Backlog. Do not label this parent or multiple leaves. An open pull request does not unblock its successor; the preceding leaf must merge first.

Completion criteria

  • All five children are merged and closed.
  • Every named production caller uses the native reader.
  • Historical fixture, corruption, concurrency, bounds, and immutability tests pass.
  • Source checks prove every named caller reaches only the native read boundary.
  • No child merged a test-only parser, unused native API, or format-only foundation.

Lexical non-regression

Repository changes for every implementation leaf before the final removal may delete existing references but must add zero new case-insensitive bluge tokens and zero matching tracked paths. The gate includes imports and aliases, function/type/variable names, filenames and runtime names, strings, comments and messages, tests, fixture/provenance data, scripts, configuration, and generated assets. Compatibility evidence uses neutral legacy oracle or compatibility writer labels plus an immutable revision or content hash rather than adding a retired module name.

This lexical gate applies to repository changes, not to issue or archived-design prose that names the dependency in order to specify its removal.

Design

BDB-NIDX-SPEC-001 revision 0.2 — NIDX-01 behavior

Activity

  1. added this to the BanyanDB - 0.12.0 milestone on Aug 25, 2026
  2. changed the title [-][NIDX-01] Replace the Property shard index with the native implementation[/-] [+][NIDX-01] Replace bounded read-only index access with the native reader[/+] on Aug 25, 2026
  3. changed the title [-][NIDX-01] Replace bounded read-only index access with the native reader[/-] [+][NIDX-01 workstream] Replace bounded read-only index access[/+] on Aug 25, 2026
  4. hanahmily commented on Sep 11, 2026

    @hanahmily
    ContributorAuthor

    All five ordered leaves are merged and closed, which is this workstream's stated closing condition.

    leaf PR commit
    #14008 — NIDX-01A baseline read-only document count apache/skywalking-banyandb#1309 ba66af94
    #14009 — NIDX-01B committed generations with deletions apache/skywalking-banyandb#1316 eeb3b927
    #14010 — NIDX-01C stored document walks for migration sources apache/skywalking-banyandb#1318 411a3941
    #14011 — NIDX-01D exact-term schema walks apache/skywalking-banyandb#1325 59805944
    #14012 — NIDX-01E Property repair search-after apache/skywalking-banyandb#1335 31b32ca2

    The bounded read-only surface is on main at 31b32ca2: committed-generation open, visible count, match-all stored walk, exact term/OR, deletion masks, required scalar decoding, and ascending Property repair search-after. The native reader package imports no retired index dependency and reads a shard directory directly.

    Closing this unblocks #14007 (NIDX-02) for just-in-time decomposition.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions