Skip to content

[NIDX-02 workstream] Replace the Property shard index #14007

Description

@hanahmily

[NIDX-02 workstream] Replace the Property shard index

Parent: #13990
Blocked by: #14002

Tracking parent — do not apply Backlog. Decompose this workstream into ordered TDD leaves only after every NIDX-01 leaf has merged and the actual native reader seam is available on main.

End-state boundary

banyand/property/db.newShard selects the native implementation for existing and new Property shards. The completed workstream owns compatible Property writes, exact/boolean/range queries, explicit sort, callbacks, snapshot publication/recovery, merge-time expiry, reference-safe GC, repair integration, backup, and same-file rollback.

The retained legacy Property constructor is an explicit rollback choice during the compatibility window, not a second active writer.

Cutover and writer ownership

The implementation choice is Property-database-wide during NIDX-02, not inferred independently for each shard.

  • The Property database establishes exclusive writer ownership before scanning its directory or opening any shard store.
  • While it owns the database, startup resolves exactly one implementation from an explicit Property-wide rollout selection; property/db.newShard uses that selection for every existing and newly created shard.
  • A process that cannot establish ownership fails before either implementation opens a shard writer.
  • The native index neither creates nor requires an index-local PID file, lock file, or writer-origin marker. Such runtime files are outside ICE/snapshot compatibility.
  • Selection is not inferred from ICE segments, snapshot manifests, CRC32 values, or other writer-origin heuristics; those files remain implementation-neutral during the same-file rollback window.
  • Rollback stops admission, drains durable callbacks, closes every active shard writer, changes the Property-wide selection, and then reopens the same files with the retained legacy constructor.
  • Simultaneous native and legacy writers for the Property database are prohibited.

Required behavior

  • Legacy-created shards open without rewrite and retain documents, repeated stored values, deletions, and explicit order.
  • Replace/upsert leaves one visible latest document; delete removes it from every supported query and scan.
  • Native output is ICE v3/snapshot v3 accepted by the pinned rollback binary for query, mutation, merge, and restart.
  • Publication and callbacks are durable; every crash cut selects a complete old or new generation.
  • Merge, expiry, and GC retain only data reachable from protected snapshots.
  • CRC32 fields remain present but are never calculated or validated.

Just-in-time decomposition gate

When #14002 closes:

  1. inspect the merged native reader and measured NIDX-01 run reports;
  2. identify the smallest live Property caller for each new writer/lifecycle capability;
  3. file leaf issues in dependency order so issue numbers ascend;
  4. give each leaf one pre-agreed seam, literal fixture results, one RED command, one e2e command, and focused package suites; and
  5. apply Backlog only to the first unblocked leaf.

Do not pre-file format-only writer, snapshot, merge, or GC tickets. Each leaf must activate a live Property behavior in its merge.

Completion criteria

All future leaves merge; the complete property/db.newShard role is native; the two-binary compatibility and crash matrices pass; and the workstream closes before NIDX-03 decomposition begins.

Lexical non-regression

Repository changes for every implementation leaf before the final removal may delete existing references but must add zero new case-insensitive bluge tokens and zero matching tracked paths. The gate includes imports and aliases, function/type/variable names, filenames and runtime names, strings, comments and messages, tests, fixture/provenance data, scripts, configuration, and generated assets. Compatibility evidence uses neutral legacy oracle or compatibility writer labels plus an immutable revision or content hash rather than adding a retired module name.

This lexical gate applies to repository changes, not to issue or archived-design prose that names the dependency in order to specify its removal.

Design

BDB-NIDX-SPEC-001 revision 0.2 — NIDX-02 behavior

Activity

  1. added this to the BanyanDB - 0.12.0 milestone on Aug 25, 2026
  2. changed the title [-][NIDX-02] Replace the Property shard index with the native implementation[/-] [+][NIDX-02 workstream] Replace the Property shard index[/+] on Aug 25, 2026
  3. hanahmily commented on Sep 3, 2026

    @hanahmily
    ContributorAuthor

    Thanks for raising the writer-ownership and protected-snapshot concerns.

    I checked the current Property shard model. There is no persisted per-shard metadata identifying the implementation that created or owns it: the shard directory provides only the group and shard ID, while ICE/snapshot metadata remains implementation-neutral so both implementations can reopen the same files during the rollback window. The native index will also neither create nor require an index-local PID or lock file; those runtime files are outside the compatibility format.

    I updated the issue body to make the cutover contract explicit. NIDX-02 now requires the Property database to establish exclusive ownership before scanning or opening shard stores, resolve one Property-wide implementation selection, and make property/db.newShard use only that selection for every existing and new shard. Rollback must stop admission, drain callbacks, close all writers, change the selection, and reopen the same files. Simultaneous native and legacy writers are prohibited.

    The merge/expiry/GC part was already covered by the design: merge operates from a pinned source snapshot, expiry clones deletion state, replacement publication precedes collection, and a file remains protected while referenced by a retained manifest, active reader, or backup. CRC32 remains present only for layout compatibility and is not calculated, validated, or used for generation selection or GC.

  4. hanahmily commented on Sep 11, 2026

    @hanahmily
    ContributorAuthor

    #14002 is closed and the native reader is on main at 31b32ca2, so this workstream is decomposed.

    Ordered TDD leaves

    1. #14073 — NIDX-02A: encode Property documents as native ICE v3 segment bytes
    2. #14074 — NIDX-02B: drive the native encoder through the segment plugin contract, blocked by [NIDX-02A] Encode Property documents as native ICE v3 segment bytes #14073
    3. #14075 — NIDX-02C: cut the Property database over to the native writer, blocked by [NIDX-02B] Drive the native encoder through the segment plugin contract #14074

    Only the oldest unblocked open leaf may carry Backlog. Do not label this parent or multiple leaves. An open pull request does not unblock its successor; the preceding leaf must merge first.

    Two notes on the split

    The writer is separated from the cutover on purpose. A writer-only ticket would normally be format-only and therefore forbidden, but nativeice.Open reads a shard directory directly and the package imports no retired dependency, so #14073 has two real consumers already on main — the merged reader and the pinned compatibility reader. That makes it vertical without touching newShard. The dangerous diff, where native bytes first go under a live Property writer, is isolated in #14075.

    The dependency-policy question is isolated in #14075. Registration through the aliased index package is the only place in the ladder that cannot avoid the retired token in an added line; #14073 and #14074 reach everything they need through the neutral segment alias. So the exemption decision gates only the last leaf and does not hold up the encoder.

    #14074 is provisional. Its consumer is the index library's lifecycle rather than a live Property caller. If that is judged not vertical, fold it into #14075 and close it as superseded; that decision should be made before it becomes the oldest unblocked leaf.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions