Repository navigation
fix(rust/sedona-raster-gdal): read a file nodata the band type cannot hold as none - #1357
Merged
james-willis merged 1 commit intoSep 30, 2026
Conversation
… hold as none GDAL reports a band's nodata as an f64 (for GeoTIFF, the ASCII TIFFTAG_GDAL_NODATA tag), so a file can declare -9999 or NaN on a UInt8 band, or 0.5 on an Int32 band. The loader packed it into the band type with saturating `as` casts: -9999 and NaN became 0, 256 became 255, 0.5 became 0, and every real pixel equal to the result read as nodata (RS_Value returned NULL for real 0 pixels). No pixel can equal such a value, so the band now reads as having no nodata. The GDAL-side conversion is replaced by the exact check in sedona_raster::traits::nodata_f64_to_bytes, with its error mapped to "no nodata" rather than failing the read: the file is valid, and GDAL, rasterio and Sedona Spark all open it. Sedona Spark matches no pixel either, and GDAL's own mask band ignores an out-of-range nodata the same way. write_geotiff now sets nodata through rasterio's band setter, which unlike the open() argument accepts values outside the dtype's range (files are byte-identical otherwise), so tests can write these files.
james-willis
marked this pull request as ready for review
September 25, 2026 20:08
james-willis
added a commit
to james-willis/sedona-db
that referenced
this pull request
Sep 30, 2026
…fixed Loading a raster whose file nodata is not representable in the band dtype now drops the nodata rather than packing it (apache#1357), so SedonaDB no longer turns 0.5 into 0 and no longer excludes real 0 pixels. Both engines agree, and strict xfail surfaced the marker as stale.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The GDAL loader packed a file's nodata into the band's data type with saturating
ascasts. GDAL reports nodata as anf64(for GeoTIFF, the ASCIITIFFTAG_GDAL_NODATAtag), so a file can declare a value its band type cannot hold, and the cast turned that value into a real pixel value. Every pixel equal to the result then read as nodata. Before this change, on a 2x2 UInt8 GeoTIFF with pixels 0, 255, 1, 7:RS_BandNoDataValueRS_BandNoDataValueBoth loaders (out-db
RS_FromPath, in-dbRS_FromGDALRaster) go throughband_nodata_to_bytes, so every consumer of band nodata was affected:RS_Value/RS_Values,RS_BandNoDataValue,RS_MinConvexHull, and the pendingRS_BandIsNoData(#1350) andRS_SummaryStats(#1352).Semantics
A band now has no nodata when the file's value is not exactly representable in the band type. No pixel can equal such a value, and that is how the other readers treat it:
all_validfor an out-of-range or NaN nodata (checked with GDAL 3.12.4), and rasterio'snodatavalsreportsNonefor one.Raising an error instead would refuse files that GDAL, rasterio and Spark all open, and a user cannot fix a file's metadata from SQL. That is the difference from
sedona_raster::traits::nodata_f64_to_bytes, which errors because its input is a query argument (RS_SetBandNoDataValue,RS_Clip,RS_Tile). The loader now uses that same exact check and treats its error as "no nodata". The saturating converter ingdal_common.rs, and its re-export, are removed.Two known differences remain:
RS_BandNoDataValue. It returns NULL where Spark reports the declared -9999, 256 or 0.5, because SedonaDB stores a band's nodata in the band type. For NaN on an integer band, both engines now return NULL.Float bands are unchanged: Float32 rounds to the nearest f32, as GDAL does, and NaN stays NaN. UInt64/Int64 go through GDAL's exact 64-bit getters and are unchanged too.
Tests
test_band_nodata_to_bytes_drops_unrepresentable_nodataruns over MEM bands. Values below or above the range, NaN, and fractions are dropped. Range bounds and a Float32 value are kept.python/sedonadb/tests/functions/test_rs_value.py:test_unrepresentable_file_nodata_is_droppedruns through both loaders.RS_BandNoDataValueis NULL, and every pixel samples verbatim, including one planted at the value the old cast produced. It is anchored to the data, which rasterio's comparator also returns. The file's fixtures switch from a raw geotransform to the equivalent bbox.write_geotiffnow sets nodata through rasterio's band setter. Unlikerasterio.open(nodata=), the setter accepts a value outside the dtype's range, and it writes byte-identical files for every other value.test_raster_testing.pychecks that the value lands in the file by reading it back through a GDAL VRT copy, because rasterio hides an out-of-range nodata.test_rs_value.py: a pixel-level case over -9999, 256, NaN and 0.5 now passes on both engines (the planted pixel reads verbatim).test_rs_bandnodatavalue.py: NaN on uint8 passes, anchored to NULL. The fractional xfail becomes one parametrized xfail over -9999, 256 and 0.5, with the reason updated to the NULL-vs-verbatim divergence.Once this lands, #1352's
test_rs_summarystats_fractional_nodata_on_int_bandxfail will XPASS and can be flipped.