Skip to content

Fix Binary.compareTo to order bytes as unsigned in UDF and Pipe APIs - #19271

Open
amalsaidov wants to merge 1 commit into
apache:masterfrom
amalsaidov:fix-binary-unsigned-compareto
Open

amalsaidov wants to merge 1 commit into
apache:masterfrom
amalsaidov:fix-binary-unsigned-compareto

Conversation

@amalsaidov

Copy link
Copy Markdown

Description

Binary in udf-api and pipe-api compared its bytes with a signed subtraction:

return this.values[i] - other.values[i];

A Java byte is signed, so any byte with the high bit set (>= 0x80) is negative. That makes compareTo sort such bytes before plain ASCII bytes, which disagrees with:

  • TsFile's own org.apache.tsfile.utils.Binary.compareTo, which compares bytes unsigned (val[index] & 0xff), and
  • java.lang.String ordering.

Any UDF/UDAF or pipe plugin that sorts or compares Binary values containing non-ASCII UTF-8 text or binary data with high bytes therefore got the wrong order.

Scenario and before/after

"A" is the byte 0x41 (65). "é" is UTF-8 0xC3 0xA9, so its first byte is 0xC3 (195 unsigned, but -61 signed).

Comparison Correct (unsigned, String/TsFile) Before this fix (signed)
new Binary("A").compareTo(new Binary("é")) < 0 (A sorts first) > 0 (wrong)
new Binary(new byte[]{0x41}).compareTo(new Binary(new byte[]{(byte)0x80})) < 0 > 0 (wrong)

Fix

Compare each byte as an unsigned value, matching the ordering used everywhere else:

return (this.values[i] & 0xFF) - (other.values[i] & 0xFF);

ASCII-only comparisons are unchanged (all ASCII bytes are < 0x80), and equals, which only checks for compareTo == 0, is unaffected.

Tests

Added BinaryTest to both modules covering unsigned ordering, agreement with String ordering for non-ASCII text, ASCII ordering, and prefix/length handling. The unsigned-ordering and String-agreement cases fail on the old code and pass with the fix.

Tests run: 4, Failures: 0, Errors: 0, Skipped: 0  (udf-api BinaryTest)
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0  (pipe-api BinaryTest)

This PR has:

  • been self-reviewed.
  • added unit tests to cover the fixed code path.

AI-assisted; directed, reviewed and tested by the author (amalsaidov).

The Binary type in udf-api and pipe-api compared bytes with a signed
subtraction (this.values[i] - other.values[i]). Because a Java byte is
signed, any byte with the high bit set (>= 0x80) was treated as negative
and sorted before ASCII bytes. That ordering disagrees with TsFile's own
Binary.compareTo and with java.lang.String ordering, both of which treat
bytes as unsigned.

So a value containing non-ASCII UTF-8 (for example "é", encoded as
0xC3 0xA9) or binary data with high bytes sorted incorrectly relative to
plain ASCII values when a UDF or pipe plugin compared Binary values.

Compare each byte as an unsigned value (& 0xFF) so the result matches the
lexicographic ordering used everywhere else. Added unit tests in both
modules covering unsigned ordering, agreement with String ordering,
ASCII ordering, and prefix/length handling.
@HTHou
HTHou requested a review from JackieTien97 November 14, 2026 00:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant