Fix Delta write stats if data schema is missing columns relative to table schema [databricks] #8287
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…able schema Signed-off-by: Jason Lowe <[email protected]>
|
build |
tgravescs
approved these changes
May 15, 2023
|
I manually tested on Databricks 11.3 on query where I found the issue and this fixes it. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #8263.
Delta Lake 2.2+ and Delta Lake on Databricks 11.3+ supports overwriting a table while preserving the old table's column for statistics gathering. This requires mapping the data schema onto the stats collection schema, the latter of which may refer to columns that are missing in the data schema or be reordered relative to the stats collection schema.
This resolves the issue by creating an exploded schema map for the data schema. As we walk the stats collection schema as usual to generate the statistics, the exploded map can be referenced to know whether the column is present, and if it is, what ordinal it's at.