[SPARK-31019][SQL] make it clear that people can deduplicate map keys #27772
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
Author
|
Test build #119232 has finished for PR 27772 at commit
|
viirya
reviewed
Mar 3, 2020
| .booleanConf | ||
| .createWithDefault(false) | ||
|
|
||
| val LEGACY_ALLOW_DUPLICATED_MAP_KEY = |
Member
There was a problem hiding this comment.
We need to update sql-migration-guide doc too. We already documented spark.sql.legacy.allowDuplicatedMapKeys there.
Member
There was a problem hiding this comment.
+1,
Also find another mistake I made in the migration guide, fixed in #27782, please take a look.
|
Test build #119283 has finished for PR 27772 at commit
|
Member
|
retest this please |
|
Test build #119291 has finished for PR 27772 at commit
|
Contributor
Author
|
retest this please |
|
Test build #119312 has finished for PR 27772 at commit
|
Contributor
Author
|
retest this please |
|
Test build #119320 has finished for PR 27772 at commit
|
Contributor
Author
|
retest this please |
|
Test build #119318 has finished for PR 27772 at commit
|
|
Test build #119327 has finished for PR 27772 at commit
|
HyukjinKwon
approved these changes
Mar 5, 2020
Member
|
Merged to master, and branch-3.0. |
HyukjinKwon
pushed a commit
that referenced
this pull request
Mar 5, 2020
rename the config and make it non-internal. Now we fail the query if duplicated map keys are detected, and provide a legacy config to deduplicate it. However, we must provide a way to get users out of this situation, instead of just rejecting to run the query. This exit strategy should always be there, while legacy config indicates that it may be removed someday. no, just rename a config which was added in 3.0 add more tests for the fail behavior. Closes #27772 from cloud-fan/map. Authored-by: Wenchen Fan <[email protected]> Signed-off-by: HyukjinKwon <[email protected]>
sjincho
pushed a commit
to sjincho/spark
that referenced
this pull request
Apr 15, 2020
### What changes were proposed in this pull request? rename the config and make it non-internal. ### Why are the changes needed? Now we fail the query if duplicated map keys are detected, and provide a legacy config to deduplicate it. However, we must provide a way to get users out of this situation, instead of just rejecting to run the query. This exit strategy should always be there, while legacy config indicates that it may be removed someday. ### Does this PR introduce any user-facing change? no, just rename a config which was added in 3.0 ### How was this patch tested? add more tests for the fail behavior. Closes apache#27772 from cloud-fan/map. Authored-by: Wenchen Fan <[email protected]> Signed-off-by: HyukjinKwon <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
rename the config and make it non-internal.
Why are the changes needed?
Now we fail the query if duplicated map keys are detected, and provide a legacy config to deduplicate it. However, we must provide a way to get users out of this situation, instead of just rejecting to run the query. This exit strategy should always be there, while legacy config indicates that it may be removed someday.
Does this PR introduce any user-facing change?
no, just rename a config which was added in 3.0
How was this patch tested?
add more tests for the fail behavior.