[GH-3365] Support NaN as a raster band nodata value - #3366
Merged
Merged
Conversation
Sedona used NaN as its own marker for "no nodata value", so a GeoTIFF whose GDAL_NODATA tag is nan (what GDAL writes for floating point rasters) was indistinguishable from one without nodata: the value read back as null, every exclude-nodata path kept the NaN pixels, and RS_SetBandNoDataValue(raster, NaN) was a silent no-op. RasterUtils.hasNoDataValue tells a NaN nodata value from a missing one and RasterUtils.isNoData compares pixels NaN-safely; the nodata checks in counts, statistics, zonal statistics, bandIsNoData, set/replace/remove nodata, normalizeAll, setPixelType, the pixel editors, min convex hull, resampling and tiling go through them. The resampling nodata mask is a 0/1 flag restored from the band's nodata value (the NaN-marker mask also zeroed every valid pixel of integer rasters), holes are filled as deep as the interpolation kernel reads, replace keeps untouched bands, the serializer reads the with_bands override flags before deciding what a NaN on the wire means, and tile padding takes a nullable value. The stock GeoTools reader already keeps a NaN nodata category and the stock ImageIO-Ext tag parser already accepts inf, so no reader change is needed. Writing NaN nodata back with RS_AsGeoTiff is a follow-up.
This was referenced Sep 23, 2026
james-willis
added a commit
to james-willis/sedona-db
that referenced
this pull request
Sep 30, 2026
An xfail that starts passing is a result, not a non-event: it means the behaviour it describes has changed, usually because an upstream fix landed, and the marker is now lying about the state of the world. pytest's default reports XPASS and exits 0, so that goes unnoticed and stale reasons accumulate — the suite keeps asserting divergences that no longer exist, and anyone reading one is sent to re-investigate a closed problem. Turn xfail_strict on so the run fails and names the test instead. The setting lives in a new root pytest.ini because CI runs each suite from its own directory, so pytest walks up from there and one file covers every suite, including ones added later. All 82 xfail markers are currently in integration/spark-parity; the other Python suites have none, so this is mostly a guard on future ones. Four reasons were already stale, and are rewritten to name the release they wait on. Sedona master supports NaN as a raster band nodata value (apache/sedona#3366, with #3368 writing it to GeoTIFF), so both engines now agree on the NaN cases in test_rs_bandnodatavalue and test_rs_setbandnodatavalue. That is not in a release — 1.9.1 predates it, and a release is what CI installs — so the tests keep their xfail, now saying so rather than describing a divergence that is already fixed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Did you read the Contributor Guide?
Is this PR related to a ticket?
[GH-XXX] my subject. Closes Support NaN as a raster band nodata value #3365What changes were proposed in this PR?
Sedona used
NaNas its internal marker for "this band has no nodata value", so a GeoTIFF whoseGDAL_NODATAtag isnan(what GDAL writes for floating point rasters such as DTM/DSM tiles) was indistinguishable from one with no nodata at all:RS_BandNoDataValuereturned null, every exclude-nodata path kept the NaN pixels, andRS_SetBandNoDataValue(raster, NaN)was a silent no-op.RasterUtils.hasNoDataValuetells a NaN nodata value from a missing one;RasterUtils.isNoDatacompares pixels NaN-safely (and treats-0.0as0.0).RS_BandNoDataValuereturnsNaNfor such bands,RS_SetBandNoDataValuecan set, replace and remove it (NaN on an integral band throws).Double.isNaN(getNoDataValue(...))or compared with==/!=goes through the helpers:RS_Count,RS_SummaryStats*,RS_ZonalStats*,RS_BandIsNoData,RS_NormalizeAll,RS_SetPixelType(refuses NaN nodata on integral targets),RS_SetValues,RS_MinConvexHull(returns null when every pixel is nodata),RS_ReplaceNoDataValues/RS_NoDataValueMask.RS_Resampleis a 0/1 flag restored from the band's nodata value. The old NaN-marker encoding could not restore a NaN nodata value and zeroed every valid pixel of integer rasters. Holes are filled as deep as the interpolation kernel reads before resampling.RS_SetBandNoDataValue(..., replace=true)compares through the NaN-safe helper; the band copy itself comes from [GH-3330] Preserve other raster bands when replacing NoData #3347.with_bandsoverride flags first: an explicitnodata=float("nan")clears any NODATA category; an omitted nodata with a dtype change keeps an inherited NaN nodata for float/double outputs and rejects it for integral ones.doubleentry points stay for compatibility with NaN meaning "inherit".RS_BandNoDataValue,RS_SetBandNoDataValue,RS_FromGeoTiff,RS_MinConvexHull,RS_Tile,RS_TileExplode.The stock GeoTools 33.1 reader already keeps a NaN nodata category and the stock ImageIO-Ext 1.4.15 tag parser already accepts
inf, so no reader change is needed. Writing a NaN nodata value back withRS_AsGeoTiffstill needs a writer change and is a follow-up.How was this patch tested?
spark/common/src/test/resources/raster_geotiff_nodata/(4x4 Float32,GDAL_NODATA=nan,inf,-inf, with and without scale/offset; built withgdal_translate/tiffset).RasterConstructorsTest,RasterBandAccessorsTest,RasterBandEditorsTest,RasterEditorsTest,GeometryFunctionsTest,SerdeTest,RasterUtilsTest, andrasteralgebraTest(read, counts, statistics, replace,RS_Value,RS_SetBandNoDataValuewithdouble('NaN')).commonsuite andrasteralgebraTestpass locally.Did this PR include necessary documentation updates?