[Feature] Replace Bluge with a native ICE-compatible inverted index #13990

Summary

Replace runtime Bluge/ICE with a minimal BanyanDB-owned inverted index while preserving ICE v3 and snapshot v3. BanyanDB owns index behavior; this is not a new-format migration.

The behavioral and compatibility authority remains BDB-NIDX-SPEC-001 revision 0.2, introduced by skywalking-banyandb#1306 and refined by skywalking-banyandb#1307. The tracker hierarchy below refines implementation granularity so each automated TDD run owns one observable seam and one mergeable live-caller replacement.

Boundary

Public pkg/index/ contracts remain stable for callers. Native implementation types are internal; no Bluge, ICE, or bluge_segment_api type may escape. The five top-level NIDX issues are tracking workstreams, not implementation candidates:

1. bounded direct read-only operations;
2. Property shard index;
3. per-segment series sidx;
4. Stream element idx; and
5. remaining administration plus dependency removal.

TDD execution model

The TDD workflow selects the lowest-number open issue carrying both database and Backlog. Therefore:

• never apply Backlog to this umbrella or a workstream parent;
• only the oldest unblocked leaf may carry Backlog;
• a successor is not unblocked until its predecessor PR merges;
• create downstream leaves just in time, after their blocker has taught us the real seam; and
• every leaf must switch a named production/CLI caller in its merge—no fixture-only, parser-only, codec-only, shadow-only, or test-only leaf.

graph LR
  A["NIDX-01A<br/>baseline count"] --> B["NIDX-01B<br/>generations + deletions"]
  B --> C["NIDX-01C<br/>stored migration walks"]
  C --> D["NIDX-01D<br/>exact schema walks"]
  D --> E["NIDX-01E<br/>repair search-after"]
  E --> P["decompose NIDX-02<br/>from merged seams"]
  P --> S["decompose NIDX-03"] --> T["decompose NIDX-04"] --> R["decompose NIDX-05"]

Pre-filing all downstream leaves would be horizontal slicing: their contracts would describe interfaces that do not exist yet and would be stale when selected. File them only when the preceding workstream closes; their issue numbers will then naturally follow dependency order.

Current executable leaves: NIDX-01

1. #14008 — baseline single-segment ReadOnlyDocCount
2. #14009 — multi-segment generations and deletion masks, blocked by [NIDX-01A] Cut over baseline read-only document count #14008
3. #14010 — match-all stored walks for migration sources, blocked by [NIDX-01B] Count committed generations with deletions #14009
4. #14011 — exact-term/OR schema walks, blocked by [NIDX-01C] Walk stored documents for migration source reads #14010
5. #14012 — Property repair tuple search-after, blocked by [NIDX-01D] Cut over exact-term schema walks #14011

Each leaf owns checked-in compatibility-writer fixture bytes, literal expected values, one pre-agreed behavioral seam, RED and e2e commands, focused race/package suites, immutable-file proof, typed bounded corruption behavior, and explicit rollback.

Compatibility and durability contract

• Native readers open supported historical ICE v3/snapshot directories and return canonical logical results.
• During the retirement window, the pinned legacy oracle opens native output and can query, merge, restart, and externally receive it where applicable.
• No new disk version, opening-time migration, replay, or rewrite is allowed merely to remove dependencies.
• Required CRC32 fields remain layout bytes only; native code never calculates, validates, caches, or uses them.
• Publication is atomic; restart selects a complete preceding or new generation, never a partial snapshot.
• Malformed input returns bounded typed errors without panic, hang, leak, or unbounded allocation.
• Native index code uses BanyanDB database ownership for writer exclusion; it creates and requires no index-local PID or lock file.
• Read-only operations create no runtime file and do not change bytes, mtimes, or directory entries.

Required final product surface

• Bounded reads: committed generation, visible count, match-all stored walk, exact term/OR, deletion masks, and Property repair search-after.
• Property: upsert/delete, stored fields, required filters/sort, callbacks, snapshots, recovery, expiry merge, GC, repair, and backup.
• Series: field-aware mutation/projection, identity matching/dictionaries, filters/sort, index-mode Measure, snapshots, cache reset, and mixed-version receive.
• Stream elements: batch/NRT, required filters/analyzers, paired postings, doc-value sort, snapshot/restart/receive.
• Administration: verify, rebuild, union/dedup, reconstruction, migration output, dump, and final dependency removal.

BM25, boosts, phrase/fuzzy/geo, highlighting, facets, explanations, generic collectors, and plugin compatibility remain out of scope.

Lexical non-regression

Repository changes for every implementation leaf before the final removal may delete existing references but must add zero new case-insensitive bluge tokens and zero matching tracked paths. The gate includes imports and aliases, function/type/variable names, filenames and runtime names, strings, comments and messages, tests, fixture/provenance data, scripts, configuration, and generated assets. Compatibility evidence uses neutral legacy oracle or compatibility writer labels plus an immutable revision or content hash rather than adding a retired module name.

This lexical gate applies to repository changes, not to issue or archived-design prose that names the dependency in order to specify its removal.

Completion criteria

• Every leaf is merged and its named caller is native; no dormant native implementation lands.
• Product, historical-fixture, two-binary, crash, corruption, race, resource, and performance gates pass for the applicable workstream.
• Operational metrics, mismatch artifacts, recovery guidance, and rollback instructions are exercised.
• NIDX-01 through NIDX-04 add no new case-insensitive bluge token or matching tracked path in active repository surfaces.
• The last NIDX-05 removal leaf deletes every pre-existing case-insensitive bluge token and matching tracked path outside archived design/research evidence and Git history; no alias, filename, runtime name, fixture metadata, or generated license remains.
• Final dependency scans find no retired runtime or CLI edge; compatibility fixtures use neutral oracle identity and immutable hashes.

Top-level workstreams

1. NIDX-01 — bounded read-only access, with five ordered TDD leaves above
2. NIDX-02 — Property shard index, decompose after NIDX-01 closes
3. NIDX-03 — per-segment series sidx, decompose after NIDX-02 closes
4. NIDX-04 — Stream element idx, decompose after NIDX-03 closes
5. NIDX-05 — administration and dependency removal, decompose after NIDX-01 through NIDX-04 close and retirement is approved