Skip to content

fix(planner): scalar-seeded chain for correlated PK equalities - #1062

Open
adsharma wants to merge 2 commits into
mainfrom
fix/q6-scalar-seeded-chain
Open

adsharma wants to merge 2 commits into
mainfrom
fix/q6-scalar-seeded-chain

Conversation

@adsharma

Copy link
Copy Markdown
Contributor

Q6 regressed 722ms -> 1064ms after degree-stats fan-out costing (#1046): the last MATCH (UNNEST_CORRELATED leg over f, post, t, tag with t.ID = knownTagId pulled above the join) flipped from tag-seeded BWD chains to a full 1M-Post scan with FWD extends plus two tag-attach hash joins, because honest fan-out put both candidates in the same league and the full-plan comparison tipped the wrong way (same 2.76M rows, ~300ms more expensive route).

Changes (plan_subquery.cpp, planner.h)

New tryPlanScalarSeededChain, called from planRegularMatch after the collect-membership unnest (mutually exclusive triggers). Plans the leg from the seed value directly:

  • PK lookup of the seed node keyed by the outer scalar; tree-walk extends covering every rel, preferring outer-bound neighbors; outer-bound neighbors via fresh variable copies (extending into a bound variable would collide scopes); deferred residual filters split by evaluability (a pre-existing f._ID=f._ID tautology must wait for the joined scope); property-attach joins for unfetched leg-node properties.
  • Dedup (preferred): an independent key subplan (deep copy of the outer prefix, audited operator types only, schemas/cards recomputed, graceful bail) computes the chain once per DISTINCT key and hash-joins back. Fallback appends the chain onto the outer plan (correct, repeats per row).
  • Probe-to-build SIP prohibited on the new joins: the build recomputes the probe's own values, so masks prune nothing while materializing wide probe rows.

Strict bail-outs throughout (single graph, non-recursive rels, single convertible equality with outer-var key, no LIST_CONTAINS, acyclic full coverage, filter-placeable residuals) fall back to regular planning untouched.

Measurements (LDBC SNB SF1, lbug shell)

  • Q6: 1064ms -> ~140ms, results identical row-for-row (also beats the 747ms pre-regression baseline ~5x)
  • Q5/Q7/Q10/Q12/Q14 flat (Q12 still ~0.6s)

clang-format clean. Full e2e left to CI.

Q6 regressed 722ms -> 1064ms after degree-stats fan-out costing: the last
MATCH (UNNEST_CORRELATED leg over f, post, t, tag with t.ID = knownTagId
pulled above the join) flipped from tag-seeded BWD chains to a full 1M-Post
scan with FWD extends plus two tag-attach hash joins (2.76M rows at ~300ms
extra), because honest fan-out put both candidates in the same league and the
full-plan comparison tipped the wrong way.

Plan the leg from the seed value instead (tryPlanScalarSeededChain): PK
lookup of the seed node keyed by the outer scalar, tree-walk extends covering
every rel (outer-bound neighbors via fresh variable copies), deferred
residual filters, property-attach joins. Fires only for single-graph,
non-recursive legs with exactly one convertible PK equality whose key is
outer-evaluable with an outer dependent, full acyclic coverage, and
filter-placeable residuals; anything else falls back untouched.

Two execution hazards handled:
- An independent key subplan (deep copy of the outer prefix, audited operator
  types only) computes the chain once per DISTINCT key and hash-joins back,
  instead of repeating the walk per outer row (8263 x 863-post fan-out).
- Probe-to-build SIP prohibited on the new joins: the build recomputes the
  probe's own values, so masks prune nothing while materializing wide probe
  rows (140MB friends-list round-trip).

Measured on LDBC SNB SF1 (lbug shell): Q6 1064ms -> ~140ms with identical
results (also beats the 747ms pre-regression baseline); Q5/Q7/Q10/Q12/Q14
unchanged (Q7 blip verified as noise vs baseline binary).
…#935)

Classify every LogicalOperatorType enumerator explicitly (18 audited true,
44 conservative false) instead of default:false, so -Wswitch fails the build
on unclassified new operators. A trailing return false guards only
out-of-range values and does not silence the warning (verified: removing one
case produces 'enumeration value not handled in switch', fatal under
ENABLE_WERROR=ON). No behavior change.
@adsharma

Copy link
Copy Markdown
Contributor Author

Q6 plan change (before → after), LBUG_DUMP_LOGICAL style

Measured on LDBC SNB SF1 (lbug shell): 1097ms → ~140ms, results identical row-for-row (also beats the 747ms pre-regression baseline ~5x).

The outer friends leg (knownTag idx-scan → FoF knows*1..2 → COLLECT(DISTINCT friend) → UNWIND friends AS f, 8263 rows) is byte-identical in both plans. Only the last MATCH leg changed. (Parameters: personId=2199023262543, tagName='Carl_Gustaf_Emil_Mannerheim'.)

Before: post-seeded, filter applied last

FILTER [EQUALS(t.ID,knownTagId)]              <- applied AFTER everything
  FILTER [EQUALS(f._ID,f._ID)]
    HASH_JOIN [INNER keys=f._ID]               <- top join, build = 2,763,342 rows
      <outer: 8263 f rows>
      PROJECTION
        HASH_JOIN [INNER keys=post._ID]
          EXTEND postHasCreator bwd bound=f   <- from FULL Person scan (11k)
            SCAN Person f
          SEMI_MASKER[post]
            FILTER [NOT(t=tag)]
              HASH_JOIN [tag._ID]
                SCAN Tag tag (+ACCUMULATE+SEMI)
                HASH_JOIN [t._ID]
                  SCAN Tag t (+ACCUMULATE+SEMI)
                  EXTEND postHasTag fwd bound=post nbr=t
                    EXTEND postHasTag fwd bound=post nbr=tag
                      SCAN_NODE_TABLE [Post post]   <- FULL 1M-post scan

After: t-seeded chain, filter consumed as seed

HASH_JOIN [INNER keys=tag._ID]                <- property-attach only (tag.name)
  <joined rows: 2063, probe; SIP probe→build PROHIBITED>
  SCAN Tag tag                                <- masked to ~hundreds
FILTER [EQUALS(f._ID,f._ID)]                  <- pre-existing tautology, harmless
  HASH_JOIN [INNER keys=f._ID+knownTagId]     <- pair join, SIP probe→build PROHIBITED
    <outer: 8263 f rows>
    PROJECTION [chain cols]                   <- prunes copied prefix cols
      FILTER [NOT(t=tag)]
        EXTEND postHasTag fwd bound=post nbr=tag
          EXTEND postHasCreator fwd bound=post nbr=_0__22_f  <- FRESH f-copy
            EXTEND postHasTag bwd bound=t nbr=post
              QUERY_PRIMARY_KEY_LOOKUP [table=6(Tag) key=knownTagId]
                DISTINCT [knownTagId]         <- 1 row
                  <deep copy of outer prefix>

What changed, operator by operator

Before After Why faster
SCAN Post (1M) → 2× FWD extend → 2 tag hash joins PK_LOOKUP Tag BY knownTagId → 1 tag Seed instead of scan; the t.ID=knownTagId filter disappears (consumed as the seed)
Tag side blows up to 3.4M (post,tag) rows 863 posts → 2992 (post,tag) pairs → 181 creators, computed once DISTINCT-key subplan + pair join removes the 8263× per-row duplication
f via full Person scan inside the 2.76M-row build post→_0__22_f (fresh copy; the binder rebinds f per scope, so correlation is matched at _22_f._ID level) joined back on (f,fresh)+(key,key) Correlation without re-scan; extending into the bound f would collide scopes since physical extend overwrites
3× ACCUMULATE barriers (probe-side materialization of wide rows incl. 8263-int friends lists, ~800ms) none on the new joins (PROHIBIT_PROBE_TO_BUILD; the build recomputes probe's own values so masks prune nothing) Single biggest time drop
tag.name via 2 hash joins over millions of rows one attach join, probe = 2063 joined rows, build = masked Tag scan Attach happens after the fan-in, not during

Net: 2,763,342-row build → ~5k-row chain + 2063-row join output (same 2063 → 1520 groups → top-10).

Repro query (Q6, SF1 params inline)

MATCH (knownTag:Tag {name: 'Carl_Gustaf_Emil_Mannerheim'})
WITH knownTag.ID AS knownTagId
MATCH (person:Person {ID: 2199023262543})-[:knows*1..2]-(friend:Person)
WHERE NOT person = friend
WITH knownTagId, COLLECT(DISTINCT friend) AS friends
UNWIND friends AS f
MATCH (f)<-[:postHasCreator]-(post:Post),
      (post)-[:postHasTag]->(t:Tag {ID: knownTagId}),
      (post)-[:postHasTag]->(tag:Tag)
WHERE NOT t = tag
WITH tag.name AS tagName, COUNT(post) AS postCount
RETURN tagName, postCount
ORDER BY postCount DESC, tagName ASC
LIMIT 10;

Dump via LBUG_DUMP_LOGICAL=1 lbug ldbc_snb_sf1.lbdb -r (pre-PR binary shows the "before" shape above).

@adsharma adsharma changed the title fix(planner): scalar-seeded chain for correlated PK equalities (Q6 regression) fix(planner): scalar-seeded chain for correlated PK equalities Sep 29, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant