Skip to content

feat(rust/sedona-raster-functions): add RS_SummaryStats - #1352

Merged
james-willis merged 2 commits into
apache:mainfrom
james-willis:james/rs-summarystats
Sep 25, 2026
Merged

james-willis merged 2 commits into
apache:mainfrom
james-willis:james/rs-summarystats

Conversation

@james-willis

@james-willis james-willis commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Adds RS_SummaryStats(raster, statType[, band[, excludeNoDataValue]]), Sedona Spark's single-statistic band summary. statType is one of count, sum, mean, stddev, min or max and is case-insensitive. stddev is the population standard deviation. The band defaults to 1 and excludeNoDataValue defaults to true, as in Spark. Over no pixels, count and sum are 0 and the other statistics are NaN, which matches Spark (checked against 1.9.1).

Bit-for-bit with Spark

Spark computes these with Apache Commons Math (DescriptiveStatistics plus a population StandardDeviation). Commons Math's mean adds a second-pass correction term, and its variance subtracts an accum2²/n term. Both change the low bits compared with a naive formula. The kernel repeats that arithmetic step for step, including the left-to-right sum and the NaN-skipping min/max, so results match exactly with no tolerance.

The anchored fixture is six values near 1e8 with fractional parts. On it, every shortcut from Commons Math gives a different answer from the anchors: a different summation order, a mean without the correction pass, a variance without the accum2 term, and a variance around the naive mean. A unit test (fixture_separates_the_shortcuts) checks that each one does, and the anchors are Spark 1.9.1's own output.

Each statistic re-scans the band rather than copying its values out: count, sum, min and max take one pass, mean two and stddev three. The band bytes are already in memory, and a Vec<f64> copy would be 8 bytes per pixel outside the query's memory accounting. count decodes nothing, so it also works on 64-bit bands whose values f64 can't hold.

The semantics follow Sedona's main branch, not the 1.9.1 release. apache/sedona#3366 made a NaN nodata value exclude NaN pixels after 1.9.1 shipped, so that spark-parity case is an xfail until the suite's Sedona pin moves past it. Two other divergences are recorded as xfails: Spark 1.9.1 reads UInt32 pixels as signed, so values at or above 2^31 wrap negative; and SedonaDB rounds a fractional file nodata (0.5) into an integer band's type, which drops real 0 pixels (the known issue test_rs_bandnodatavalue.py already records). A 64-bit integer pixel beyond 2^53 is an error rather than a silently rounded value, as in RS_Value, and the error names the pixel.

Shared pixel scan

The kernel reads the band through pixel_scan.rs, the crate-private whole-band scan and NodataMatcher that landed with RS_MinConvexHull (#1351). Integer nodata compares by bytes and float nodata numerically.

Tests

  • Rust unit tests: bit-exact anchors from Spark and the guard that the fixture separates the shortcuts, case-insensitivity, invalid statType, including or excluding nodata, NaN nodata, empty input, a single value, NaN data, band errors, 64-bit precision and counting, and every arity with NULL propagation (including a NULL band).
  • python/sedonadb/tests/functions/test_rs_summarystats.py: GeoTIFF fixtures read through RS_FromPath, checked against numpy. The values are chosen so every statistic is exact in binary, so the comparisons use == even though numpy sums pairwise.
  • integration/spark-parity/test_rs_summarystats.py: anchored parity against Sedona Spark 1.9.1 over float64, float32, int16 and uint8 bands (34 pass, 3 xfail as described above).

@jiayuasu jiayuasu left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Compute each statistic in re-scans of the band instead of copying it into a
Vec<f64>, count without decoding, and name the pixel in the 2^53 error.
Replace the anchored fixture with one that every Commons Math shortcut rounds
differently (guarded by a unit test), and add float32, int16, uint32 and
fractional-nodata parity cases. The shared pixel scan rejects negative shapes
and lists its integer types explicitly.
@james-willis
james-willis marked this pull request as ready for review September 25, 2026 19:06
@james-willis
james-willis merged commit bdfdbc6 into apache:main Sep 25, 2026
17 checks passed
@james-willis
james-willis deleted the james/rs-summarystats branch October 1, 2026 20:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants