Repository navigation
feat(rust/sedona-raster-functions): add RS_SummaryStats - #1352
Merged
Merged
Conversation
This was referenced Sep 25, 2026
Compute each statistic in re-scans of the band instead of copying it into a Vec<f64>, count without decoding, and name the pixel in the 2^53 error. Replace the anchored fixture with one that every Commons Math shortcut rounds differently (guarded by a unit test), and add float32, int16, uint32 and fractional-nodata parity cases. The shared pixel scan rejects negative shapes and lists its integer types explicitly.
james-willis
force-pushed
the
james/rs-summarystats
branch
from
September 25, 2026 18:46
826a67e to
a94ddd4
Compare
james-willis
marked this pull request as ready for review
September 25, 2026 19:06
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
RS_SummaryStats(raster, statType[, band[, excludeNoDataValue]]), Sedona Spark's single-statistic band summary.statTypeis one ofcount,sum,mean,stddev,minormaxand is case-insensitive.stddevis the population standard deviation. The band defaults to 1 andexcludeNoDataValuedefaults to true, as in Spark. Over no pixels,countandsumare 0 and the other statistics are NaN, which matches Spark (checked against 1.9.1).Bit-for-bit with Spark
Spark computes these with Apache Commons Math (
DescriptiveStatisticsplus a populationStandardDeviation). Commons Math's mean adds a second-pass correction term, and its variance subtracts anaccum2²/nterm. Both change the low bits compared with a naive formula. The kernel repeats that arithmetic step for step, including the left-to-right sum and the NaN-skippingmin/max, so results match exactly with no tolerance.The anchored fixture is six values near 1e8 with fractional parts. On it, every shortcut from Commons Math gives a different answer from the anchors: a different summation order, a mean without the correction pass, a variance without the
accum2term, and a variance around the naive mean. A unit test (fixture_separates_the_shortcuts) checks that each one does, and the anchors are Spark 1.9.1's own output.Each statistic re-scans the band rather than copying its values out: count, sum, min and max take one pass, mean two and stddev three. The band bytes are already in memory, and a
Vec<f64>copy would be 8 bytes per pixel outside the query's memory accounting.countdecodes nothing, so it also works on 64-bit bands whose valuesf64can't hold.The semantics follow Sedona's main branch, not the 1.9.1 release. apache/sedona#3366 made a NaN nodata value exclude NaN pixels after 1.9.1 shipped, so that spark-parity case is an
xfailuntil the suite's Sedona pin moves past it. Two other divergences are recorded asxfails: Spark 1.9.1 reads UInt32 pixels as signed, so values at or above 2^31 wrap negative; and SedonaDB rounds a fractional file nodata (0.5) into an integer band's type, which drops real 0 pixels (the known issuetest_rs_bandnodatavalue.pyalready records). A 64-bit integer pixel beyond 2^53 is an error rather than a silently rounded value, as inRS_Value, and the error names the pixel.Shared pixel scan
The kernel reads the band through
pixel_scan.rs, the crate-private whole-band scan andNodataMatcherthat landed withRS_MinConvexHull(#1351). Integer nodata compares by bytes and float nodata numerically.Tests
statType, including or excluding nodata, NaN nodata, empty input, a single value, NaN data, band errors, 64-bit precision and counting, and every arity with NULL propagation (including a NULL band).python/sedonadb/tests/functions/test_rs_summarystats.py: GeoTIFF fixtures read throughRS_FromPath, checked against numpy. The values are chosen so every statistic is exact in binary, so the comparisons use==even though numpy sums pairwise.integration/spark-parity/test_rs_summarystats.py: anchored parity against Sedona Spark 1.9.1 over float64, float32, int16 and uint8 bands (34 pass, 3 xfail as described above).