Skip to content

[NIDX-04 workstream] Replace the Stream element index #14004

Closed

Description

@hanahmily

[NIDX-04 workstream] Replace the Stream element index

Parent: #13990
Blocked by: #14003

Tracking parent — do not apply Backlog. Split this workstream only after NIDX-03 closes, using the actual native writer, reader, posting, and receive seams then present on main.

End-state boundary

banyand/stream.newElementIndex selects the native idx batch writer, filter executor, posting collector, sorter, snapshot path, restart path, and external receiver.

Required behavior

  • Committed batches become visible at the documented near-real-time boundary.
  • Exact, existence, range, AND/OR/NOT, HAVING, IN, prefix, wildcard, and MATCH return fixed document-ID sets.
  • Keyword, simple, standard, and URL analysis produce fixed terms; MATCH AND/OR overrides remain distinct.
  • Document-ID/timestamp postings and ascending/descending doc-value sort are deterministic.
  • Legacy/native snapshots and raw segments are cross-readable; interrupted receive never publishes partial data.
  • Scoring, phrase/fuzzy/geo, highlighting, facets, explanations, collectors, and plugins remain absent.

Just-in-time decomposition gate

After #14003 closes, file ordered leaves around live Stream callers and one observable feature each. Every leaf needs a pre-agreed seam, literal terms/document IDs, a RED command, an e2e command, resource/concurrency bounds, and same-file rollback. Do not merge analyzers, filters, postings, sorters, or receiver code without activating a production path. Apply Backlog to one unblocked leaf only.

Completion criteria

All future leaves merge; the complete newElementIndex role is native; fixture, NRT, analyzer, posting, sort, restart, receive, race, and compatibility suites pass; and this workstream closes before NIDX-05 decomposition.

Lexical non-regression

Repository changes for every implementation leaf before the final removal may delete existing references but must add zero new case-insensitive bluge tokens and zero matching tracked paths. The gate includes imports and aliases, function/type/variable names, filenames and runtime names, strings, comments and messages, tests, fixture/provenance data, scripts, configuration, and generated assets. Compatibility evidence uses neutral legacy oracle or compatibility writer labels plus an immutable revision or content hash rather than adding a retired module name.

This lexical gate applies to repository changes, not to issue or archived-design prose that names the dependency in order to specify its removal.

Design

BDB-NIDX-SPEC-001 revision 0.2 — NIDX-04 behavior

Activity

  1. added this to the BanyanDB - 0.12.0 milestone on Aug 25, 2026
  2. changed the title [-][NIDX-03] Replace the Stream element index with native postings and sort[/-] [+][NIDX-04] Replace the Stream element index with native postings and sort[/+] on Aug 25, 2026
  3. changed the title [-][NIDX-04] Replace the Stream element index with native postings and sort[/-] [+][NIDX-04 workstream] Replace the Stream element index[/+] on Aug 25, 2026
  4. hanahmily commented on Oct 7, 2026

    @hanahmily
    ContributorAuthor

    Completed by apache/skywalking-banyandb#1399 (element-index migration and verification tools) on top of #1390 (runtime element index): the Stream element index role is native and banyand/stream no longer imports the legacy engine.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions