[core] Rebuild the default value row when the write type changes - #10208
Open
LuciferYang wants to merge 1 commit into
Open
LuciferYang wants to merge 1 commit into
LuciferYang wants to merge 1 commit into
Conversation
TableWriteImpl sized its DefaultValueRow once, from the row type of the constructor. withWriteType only refreshed the write type and the not-null indexes, so after switching to a different type — the Spark copy-on-write rewrite writes with the row-tracking-extended type, a partial write with a narrower projection — the stale default value row no longer matched the rows it wraps: reading one of the appended positions that is null (rewritten rows carry null sequence numbers) fell through to the default row's underlying array of the old length and crashed the write with an ArrayIndexOutOfBoundsException. Rebuild the default value row from the new write type, whose fields carry the same column defaults at the new positions. Assisted-by: GLM-5.3
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
TableWriteImplbuilds itsDefaultValueRowonce in the constructor, sized to the constructor row type.withWriteTyperefreshed the write type and the not-null indexes but left theDefaultValueRowuntouched, so once the write type changed the stale default value row no longer matched the rows it wraps. The Spark copy-on-write rewrite writes with the row-tracking-extended type, and a partial write uses a narrowed projection, both of which shift arity and positions. Reading an appended position that is null, such as the null sequence number the rewrite carries, fell through to the default row's underlying array of the old length and crashed the write with anArrayIndexOutOfBoundsException. This rebuilds the default value row from the new write type so its arity and positions follow the type.Tests
Added
testWithWriteTypeRebuildsDefaultValueRow: after switching to the row-tracking-extended write type with a null sequence number, an explicit value on the defaulted column survives the type change.Added
testWithWriteTypeSubstitutesDefaultValue: same shape but the defaulted column is left null, so the rebuilt default value row fills the column default at its new position.API and Format
No.
Documentation
No.