Repository navigation
[Chore] Keep Stream filtering and projection on batches until output #14067
Description
Summary
Stop constructing protobuf Element objects merely to filter rows or remove internally projected tags. This follows the legacy-engine removal child #14066 (closed; implemented by BanyanDB PR apache/skywalking-banyandb#1326, merged) under #13998.
Scope and dependencies
Boundary: Stream batch filtering and client projection through banyand/query/processor.go, with eligible frame egress and required protobuf egress. It spans filter placement, projection, and response-path activation, implemented here as one task against the now-merged legacy-engine removal, not split into separate PRs.
Prerequisite already merged: BanyanDB PR apache/skywalking-banyandb#1331 provides pre-merge filtering for index-order queries.
Unblocked: #14066 is closed; its completion criterion, BanyanDB PR apache/skywalking-banyandb#1326, is merged.
Requirements
R1. Reuse apache/skywalking-banyandb#1331's pre-merge index-order filtering and remove the redundant element-level recheck. Preserve filter-first duplicate winners and the bounded merge.
R2. Apply timestamp-order filtering to batches at its existing position after the cap; do not change its existing under-fill behavior.
R3. Strip criteria-only and hidden ordering tags through batch projection. Retain internal ordering keys until ordering is finished, without exposing hidden tags in the client's projection.
R4. Enable frame output for eligible distributed queries. Preserve required protobuf responses, including traced responses and final client output.
Concrete oracle
Input:
ID=A, order=1, state=closed, service=old
ID=A, order=2, state=open, service=new
ID=B, order=3, state=open, service=other
With criteria state=open, ascending index order, and projection service, return A/new then B/other. Neither state nor the ordering tag may appear in client tag families. Offset 1, limit 1 returns B/other. Expected values are specified here, not derived from a legacy engine.
Completion criteria and RED-test direction
- Real processor tests prove eligible filtered queries emit frames. Today their filter gate selects protobuf output, so this is an observable failing path assertion, not only output parity.
- Exact selected values, duplicate winners, hidden-tag removal, null behavior, and offset/limit are tested across batches.
- Timestamp-order filtering remains behind the cap, and traced responses retain tracing information.
- Each implementation leaf activates a production caller and includes its own failing test; batch ownership is correct on success, cancellation, and errors.
go test ./pkg/query/vectorized/stream/... ./pkg/query/logical/stream ./banyand/query
go test ./test/integration/standalone/query/... ./test/integration/distributed/query/...Out of scope
Reimplementing apache/skywalking-banyandb#1331; changing timestamp under-fill semantics; local cross-group or distributed result merging (a separate workstream).
Parent: #13998.
Compatibility and review policy
- Preserve existing on-disk and wire formats and mixed-version compatibility. Any necessary protocol change requires a separately reviewed compatibility contract.
- Use explicit, hand-calculated expected results; do not depend on the removed row engine as the test oracle.
- Human review enforces the no-new-row-path policy. A row-call-site lint guard is not required.
Activity
SammyUrfen commentedon Sep 17, 2026
Hi @hanahmily, I would like to work on this. Can I get it assigned? It follows on from #14056 and apache/skywalking-banyandb#1326.
hanahmily commentedon Oct 4, 2026
Updated: removed the tracking-parent/split-into-tasks framing and reworked this as a single, directly executable task — #14066 (the blocker) is closed and merged (apache/skywalking-banyandb#1326), so R1-R4 can be implemented as one cohesive scope against that now-merged baseline instead of being split out.
SammyUrfen commentedon Oct 6, 2026
Thanks for reworking it. I will start on R1 to R4 as one scope.