Skip to content

Reduce allocations in aligned page reads and Tablet serialization - #1101

Open
Caideyipi wants to merge 1 commit into
apache:developfrom
Caideyipi:perf/pipe-tablet-read-serialize
Open

Caideyipi wants to merge 1 commit into
apache:developfrom
Caideyipi:perf/pipe-tablet-read-serialize

Conversation

@Caideyipi

@Caideyipi Caideyipi commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Aligned BatchData reads without record filters box primitive values only to check whether a row is entirely null. Tablet stream serialization also creates temporary byte arrays for each numeric value. This change reduces those allocations while preserving filtering, deletion handling, and the Tablet wire format.

  • Check TsPrimitiveType presence directly when no record filters are installed. For example, tree-model rows [42, null] and [null, null] still produce only the first row; table-model reads still retain both rows.
  • Add ReadWriteIOUtils.write overloads for DataOutputStream that write primitive values and Binary lengths directly.
  • Make Tablet.serialize() write to a precisely sized ByteBuffer and add serialize(ByteBuffer) for writing at the caller's current position. For example, a caller can reserve an 8-byte prefix and append a Tablet without an intermediate serialized buffer:
ByteBuffer buffer = ByteBuffer.allocate(8 + tablet.serializedSize());
buffer.position(8);
tablet.serialize(buffer);

The new overload writes the same big-endian bytes as serialize(DataOutputStream) and preserves the caller's byte order. Float/double array buffer serialization canonicalizes NaNs to match the stream format.

Performance measurements from the earlier snapshot-base experiment (baseline 8f665d113dbb5257b901c528c31c8bc24da2a018, before rebasing onto develop):

Operation Baseline ops/s Optimized ops/s Throughput ratio Baseline bytes/op Optimized bytes/op Allocation reduction
Tablet.serialize(DataOutputStream) 3,569.81 12,526.74 3.51x 162,128 112,976 30.3%
Tablet.serialize() 3,524.83 38,140.42 10.82x 162,456 113,200 30.3%
Unfiltered aligned page to BatchData 7,118.77 8,148.29 1.145x 645,248 481,352 25.4%
  • Linux x86_64, Java 17.0.9, Maven 3.9.12. Each JVM used -Xms512m -Xmx512m -XX:+UseParallelGC -XX:ActiveProcessorCount=2.
  • A standalone microbenchmark used 1,024-row Tablets with 10 columns covering BOOLEAN, INT32, INT64, FLOAT, DOUBLE, TEXT, STRING, BLOB, DATE, and TIMESTAMP. The page-read case used 1,024 rows and 10 INT32 columns, without filters or nulls.
  • Two independent JVM runs per variant, in baseline/optimized/optimized/baseline order; each operation had 2 seconds of warmup and 3 seconds of measurement. The table shows arithmetic means; allocation was measured with ThreadMXBean.getThreadAllocatedBytes.
  • All four runs produced the same serialized payload SHA-256: 8f7c77d267d547928aad66c1357432098c682cd32af413e058beb74d2f98f2fb.

These earlier microbenchmarks measure the listed operations. Performance at the final rebased PR head and end-to-end Pipe throughput still require separate measurements.

Validation on Linux x86_64 with Java 17 and Maven 3.9.12: clean compilation and all 110 selected tests passed (zero failures, errors, or skipped tests).

mvn -o clean test -P with-java -pl java/tsfile -am \
  -Dtest=TabletTest,ReadWriteIOUtilsTest,AlignedPageReaderPushDownTest,TypeTest,PageReaderTest \
  -Dsurefire.failIfNoSpecifiedTests=false

Verification uses an isolated offline dependency repository with build scan extensions disabled in the verification source copy. Tests cover stream/buffer byte equivalence, supported data types, nulls, Unicode schemas and properties, NaNs, heap/direct/sliced buffers, nonzero positions, byte-order restoration, filters, deletions, and tree/table all-null row behavior.

@Caideyipi

Copy link
Copy Markdown
Contributor Author

Initial independent self-review completed (one round).

  • Reviewed the full eight-file diff against base 2c22af783d416467841f945d005ffdd778638ba1, including the related page-reader/filter/deletion paths, type and schema serializers, bitmap sizing, and test fixtures.
  • No confirmed actionable source-behavior or test-logic issues were found. No fixes, new commits, or push were needed.
  • Reviewed SHA, final local SHA, confirmed fork branch SHA, and confirmed PR head are all eaaad099d6a29bff60cc374ad0ff30fad6b40e3d.
  • Validation evidence: the previously reported 110 selected tests passed remotely. This round did not rerun those checks or run local tests; all seven newly added tests remain unrun locally. No additional test pass is claimed.
  • CI was still running when checked and was not awaited. The PR remains Draft as requested.

@Caideyipi
Caideyipi marked this pull request as ready for review November 13, 2026 04:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant