Skip to content

perf(parquet): decode nullable plain values into valid runs - #1314

Merged
zeroshade merged 1 commit into
apache:mainfrom
fallintoplace:perf/parquet-direct-nullable-plain
Sep 15, 2026
Merged

zeroshade merged 1 commit into
apache:mainfrom
fallintoplace:perf/parquet-direct-nullable-plain

Conversation

@fallintoplace

Copy link
Copy Markdown
Contributor

What does this change?

  • PlainDecoder.DecodeSpaced used to decode all non-null values into a dense prefix first, then move them into their bitmap positions.
  • For simple validity patterns, decode each valid run directly into its final output range.
  • Keep the existing dense plus spaced expansion path for highly fragmented validity.
  • Use the reverse bitmap reader so trailing-null pages stop after finding the dense prefix.
  • Add edge-case tests and a focused benchmark.

Benchmark

Apple M1 Pro, Go 1.26.3, 65,536 int32 values, 8 runs, 300ms per sample:

Validity pattern Before After
leading null 11.6 us/op 7.1 us/op
clustered nulls 8.6 us/op 7.2 us/op
trailing null 5.6 us/op 5.6 us/op

Random 10% nulls and alternating validity keep the existing fallback path. All cases stayed at 0 B/op and 0 allocs/op.

Tests

  • PARQUET_TEST_DATA=parquet-testing/data ARROW_TEST_DATA=arrow-testing/data go test -p 2 ./... -count=1
  • go test -race -p 2 ./parquet/internal/encoding ./parquet/file
  • go vet -p 2 ./parquet/internal/encoding ./parquet/file

@zeroshade zeroshade left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No valid-input regression found in the valid-run decoder. Focused spaced-decoding tests passed (go test -p 2 -count=1 ./parquet/internal/encoding -run TestPlainDecoderDecodeSpaced).

@zeroshade
zeroshade merged commit 7c210e3 into apache:main Sep 15, 2026
54 of 55 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants