Skip to content

Fix CAST(DOUBLE AS INT32) precision loss from narrowing to FLOAT - #19275

Open
Karman580 wants to merge 1 commit into
apache:masterfrom
Karman580:fix-double-to-int32-precision
Open

Karman580 wants to merge 1 commit into
apache:masterfrom
Karman580:fix-double-to-int32-precision

Conversation

@Karman580

Copy link
Copy Markdown
Contributor

Description

Closes #19274

Root cause

CastFunctionUtils.castDoubleToInt rounded (float) value. A FLOAT keeps only 24 significand bits, so in-range doubles above 2^24 can change before they are rounded, and doubles just below a .5 tie can be moved onto the tie. For example, 123456789.0 came out as 123456792 and 2.4999999999 as 3. Every DOUBLE to INT32 cast goes through this method: the per-value and batch column casts in the tree and table models, and table-model constant folding.

Change

- return Math.round((float) value);
+ return (int) Math.round(value);

This matches castDoubleToLong and the documented rule for DOUBLE to INT32 ("Math.round()"). The range check above it is unchanged. It limits the value to [Integer.MIN_VALUE, Integer.MAX_VALUE], so Math.round(value) stays within int range and the cast from long cannot overflow. Half values keep their existing behaviour (2.5 -> 3, -2.5 -> -2), and out-of-range values still throw.

Tests

New cases in CastFunctionUtilsTest:

  • castDoubleToIntKeepsDoublePrecision: 16777217, ±123456789, 2147483000, 2147483646, -2147483647, Integer.MAX_VALUE, Integer.MIN_VALUE, 2.4999999999 and 100000001.5
  • castDoubleToIntRoundsHalfUp: 2.5, -1.5 and -2.5
  • castDoubleToIntJustAboveMax / castDoubleToIntJustBelowMin: 2147483648.0 and -2147483649.0 throw SemanticException

Results with ./mvnw -pl iotdb-core/calc-commons test -Dtest=CastFunctionUtilsTest (Temurin JDK 25.0.2, macOS arm64):

Before the fix:

[ERROR] Tests run: 23, Failures: 1, Errors: 0, Skipped: 0
[ERROR]   CastFunctionUtilsTest.castDoubleToIntKeepsDoublePrecision:79 expected:<16777217> but was:<16777216>

After the fix:

[INFO] Tests run: 23, Failures: 0, Errors: 0, Skipped: 0
[INFO] BUILD SUCCESS

./mvnw -pl iotdb-core/calc-commons test (whole module) also passes: Tests run: 91, Failures: 0, Errors: 0, Skipped: 0. Spotless check passes.

I also ran the SQL from #19274 on a standalone 2.0.11-SNAPSHOT server with the patched class. Tree, table and constant-folded casts now return the same values as CAST(... AS INT64):

|  1.6777217E7|  16777217|  16777217|
| 1.23456789E8| 123456789| 123456789|
|-1.23456789E8|-123456789|-123456789|
|   2.147483E9|2147483000|2147483000|
| 2.4999999999|         2|         2|
|1.000000015E8| 100000002| 100000002|

SELECT CAST(2147483648.0 AS INT32) still fails with 701: Double value 2147483648.000000 is out of range of integer value., and TRY_CAST(-2147483649.0 AS INT32) returns null.

Not run locally: the integration-test suites (IoTDBCastFunctionIT, IoTDBCastFunctionTableIT) and the datanode module tests. I checked that no existing IT asserts the old FLOAT-rounded values.


This PR has:

  • been self-reviewed.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold
    for code coverage.

Key changed/added classes (or packages if there are too many classes) in this PR
  • CastFunctionUtils (calc-commons)
  • CastFunctionUtilsTest

castDoubleToInt rounded (float) value, so in-range doubles above 2^24, or
just below a .5 tie, were changed before rounding (123456789.0 became
123456792 and 2.4999999999 became 3). Round the double directly; the
existing range check keeps the result within int, so the narrowing from
long cannot overflow.

Closes apache#19274
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] CAST(DOUBLE AS INT32) returns wrong integers because values are narrowed to FLOAT before rounding

1 participant