GH-3735: Use unsigned UTF-8 byte order for Variant object field keys

[peterxcli]

Rationale for this change

The Variant specification requires an object's field ids to be sorted by the unsigned byte order of the field names' UTF-8 encoding, so that a reader can binary search them. VariantBuilder sorted those fields with String.compareTo, which orders UTF-16 code units, and Variant.getFieldByKey searched them with the same comparison.

The two orders agree on every name in the Basic Multilingual Plane, so they only diverge for a name holding a character above U+FFFF. String.compareTo places a leading high surrogate (0xD800 to 0xDBFF) below code points in U+E000 to U+FFFF, while UTF-8 byte order places it above. An object that parquet-java writes with such a name therefore carries its field ids in an order the specification forbids, and a reader that follows the specification can miss fields when it binary searches. The gap runs the other way too: parquet-java could fail to find a name above U+FFFF in an object written by another implementation.

This work started from #3736 by @rayokota. It also carries over the fallback that lets a reader still find names in objects written in the old order, which came from the equivalent Spark fix in apache/spark#58239 and the discussion there.

What changes are included in this PR?

VariantUtil.compareKeys(String, String) orders two field names the way their UTF-8 encodings compare as unsigned bytes, without encoding either one. UTF-8 byte order is exactly code point order, so the method walks the UTF-16 code units and adjusts for surrogates: a surrogate always encodes a code point above U+FFFF, so U+D800 to U+DFFF rank above U+E000 to U+FFFF, and the latter shift down to fill the gap they leave. VariantBuilder.FieldEntry.compareTo sorts an object's fields through this method, so the sort allocates nothing.

Variant.getFieldByKey first scans the lookup key for a code unit at or above U+D800. A key without one compares the same way under both orders, so the binary search runs once and uses String.compareTo, as it did before. A key with one runs the search twice, comparing through compareKeys on the first pass and String.compareTo on the second, which finds the name in objects that older versions wrote in UTF-16 order. Both passes share the one loop.

Are these changes tested?

Four tests in TestVariantObjectBuilder cover the fix.

• testObjectKeysSortedByUtf8ByteOrder builds an object whose names are U+FFFF (EF BF BF) and U+10000 (F0 90 80 80), appends them in reverse, and asserts the encoded order is U+FFFF then U+10000. The old comparison put them the other way around.
• testLargeObjectBinarySearchWithSupplementaryKey builds an object of 42 fields, above BINARY_SEARCH_THRESHOLD, mixing ASCII names with U+FFFF and U+10000, and asserts getFieldByKey finds both through the binary search.
• testLegacyUtf16OrderedObjectLookup rewrites a canonical object's id and offset lists into the old UTF-16 order, then asserts getFieldByKey still finds the ASCII, U+FFFF, and U+10000 names, and still returns null for a name the object does not hold.
• testCompareKeysMatchesUtf8ByteOrder checks compareKeys against Arrays.compareUnsigned on the two names' UTF-8 bytes, over names covering every UTF-8 length, both sides of the surrogate range, prefixes, and 200 random names. It leaves out unpaired surrogates, which have no UTF-8 encoding because Java's encoder substitutes ? for them, as the method documents.

All 184 parquet-variant tests pass, along with the parquet-avro variant read and write suites.

I also measured lookups on an object of 256 fields, alternating JVM runs on Zulu 21 and Apple silicon and taking the lowest of five runs, in ns/op.

Lookup	master	this PR
ASCII name, present	43	45
ASCII name, absent	21	23
Name above U+FFFF	35	42

The ASCII rows run the same binary search as before, and the small difference is the scan of the lookup key for a code unit at or above U+D800. The last row compares through compareKeys and, against an object in the old order, searches a second time.

Are there any user-facing changes?

An object written from now on orders a field name above U+FFFF as the specification requires. getFieldByKey still finds such names in objects already written in the UTF-16 order.

Closes #3735

[divjotarora]

One comment about potential perf issues, non-blocking as obviously correctness is more important. Also, great test coverage!

[divjotarora]

IIUC VariantUtil.encodeKey(midKey) means we now do an allocation on every iteration, correct? Can we measure the impact of this? Would it be better to cache the encoded keys, similar to getMetadataKeyCached?

[wgtmac]

Or do something like this?

boolean needsUtf8 = containsCodeUnitAtLeast(key, Character.MIN_SURROGATE);
byte[] keyBytes = needsUtf8 ? VariantUtil.encodeKey(key) : null;

// binary-search loop:
int cmp = needsUtf8
    ? VariantUtil.compareKeys(VariantUtil.encodeKey(midKey), keyBytes)
    : midKey.compareTo(key);

[divjotarora]

Small suggestion for code cleanup, but I will defer to @wgtmac and others on that. Overall looks good for correctness.

[divjotarora]

Do we need two separate search loops? There's some non-obvious stuff like using (low + high) >>> 1 that could easily diverge. One idea similar to what @wgtmac suggested in a previous comment:

boolean needsUtf8 = containsCodeUnitAtLeast(key, Character.MIN_SURROGATE);
int maxAttempts = needsUtf8 ? 2 : 1;
for (int i = 0; i < maxAttempts; i++) {
  String midKey = getMetadataKeyCached(midId)
  int cmp = (needsUtf8 && attempt == 0) ? VariantUtil.compareKeys(midKey, key) : midKey.compareTo(key);
}