Skip to content

[NIDX-01A] Cut over baseline read-only document count #14008

Closed

Description

@hanahmily

[NIDX-01A] Cut over baseline read-only document count

Parent workstream: #14002

Summary

Replace inverted.ReadOnlyDocCount for the smallest useful ICE v3 case: one committed segment with no deletion mask. This first TDD slice owns a compatibility corpus generated through BanyanDB's existing store boundary and the minimum bounded container reader. It is intentionally narrower than NIDX-01B's multi-generation fixture.

Pre-agreed test seam

Test only the existing production boundary:

func inverted.ReadOnlyDocCount(path string) (int64, error)

Callers continue to rely on that signature. Footer, offset, mmap, and section decoder types stay private and must not be boundary-tested. The production function must reach only the native reader in this merge.

Independent fixture

Before production implementation, generate and check in an ICE v3/snapshot v3 directory through BanyanDB's existing compatibility writer boundary:

  • one committed segment;
  • numeric document IDs 11 and 12;
  • no deletion record; and
  • arbitrary non-zero bytes in every reserved CRC32 field.

The declared visible count is the literal 2. Check in both the fixture bytes and a provenance manifest containing the neutral writer boundary, immutable go.mod content hash, generator command, logical documents, file hashes, and expected count. Production code must not invoke the generator or directly import the retired writer.

Requirements

R1. ReadOnlyDocCount(fixture) returns (2, nil) without creating an index-local runtime file or changing any path, byte, size, hash, mtime, or directory entry.

R2. The same call returns 2 while the compatibility writer remains open and the committed generation is visible.

R3. A copy with a truncated footer or an out-of-bounds section offset returns an error matching the native corruption sentinel selected by the boundary test. It must not panic, hang, or allocate beyond the configured section limit.

R4. Snapshot v3 segmentSize is the historical segment.Size() runtime accounting value, not the persisted file length. The valid corpus records 1292 while its .seg file is 577 bytes and must still count 2. Reserved CRC32 bytes likewise do not affect open, count, or corruption classification.

RED and end-to-end contract

  • Boundary RED: the checked-in fixture returns 2, while a malformed copy returns the typed corruption sentinel through ReadOnlyDocCount.
  • E2E RED: create/commit numeric IDs 11 and 12 through BanyanDB's compatibility writer boundary, keep that writer open, call the production ReadOnlyDocCount, verify 2, then verify an inventory of the directory is unchanged and the malformed-copy case is typed.
  • The contract author must make the e2e command fail before implementation; a test that only re-proves legacy count behavior is not a valid RED contract.

Acceptance criteria

  • The fixture, provenance manifest, boundary test, and e2e test are checked in before production behavior is authored by the TDD run.
  • go test ./pkg/index/inverted/... -run 'Test(ReadOnlyDocCountSingleSegment|E2EReadOnlyDocCount)' -count=1 passes, including the explicit accounting-value-versus-file-length compatibility case.
  • A positive dependency assertion proves ReadOnlyDocCount reaches only the native reader package and that package stays within its explicit standard-library/BanyanDB/bitmap allowlist.
  • make test-ci PKG=./pkg/index/inverted/..., make build, and make pre-push pass.

Scope

Packages: pkg/index/inverted and its private native ICE reader.

Out of scope: multiple segments, deletion masks, generation fallback, stored fields, dictionaries, postings queries, doc values, sort, writers, merge, and every other caller.

Compatibility and rollback

This slice writes no index bytes. Roll back by restoring the legacy implementation of ReadOnlyDocCount; no drain, replay, conversion, or directory change is required.

Lexical non-regression

Repository changes for every implementation leaf before the final removal may delete existing references but must add zero new case-insensitive bluge tokens and zero matching tracked paths. The gate includes imports and aliases, function/type/variable names, filenames and runtime names, strings, comments and messages, tests, fixture/provenance data, scripts, configuration, and generated assets. Compatibility evidence uses neutral legacy oracle or compatibility writer labels plus an immutable revision or content hash rather than adding a retired module name.

This lexical gate applies to repository changes, not to issue or archived-design prose that names the dependency in order to specify its removal.

Design

BDB-NIDX-SPEC-001 revision 0.2 — NIDX-01

Activity

  1. added this to the BanyanDB - 0.12.0 milestone on Aug 25, 2026
  2. added
    BacklogA list of non-trivial things that could benefit from a serious contribution.
    acceptedAccepted proposal/SWIP
    TBDTo be decided later, need more discussion or input.
    bugSomething isn't working and you are sure it's a bug!
    and removed
    BacklogA list of non-trivial things that could benefit from a serious contribution.
    acceptedAccepted proposal/SWIP
    TBDTo be decided later, need more discussion or input.
    on Aug 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    acceptedAccepted proposal/SWIPbugSomething isn't working and you are sure it's a bug!bydb file compatible changedatabaseBanyanDB - SkyWalking native database

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions