-
Notifications
You must be signed in to change notification settings - Fork 29k
[SPARK-33617][SQL][FOLLOWUP] refine the default parallelism SQL config #30736
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
Test build #132662 has finished for PR 30736 at commit
|
dongjoon-hyun
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could you fix the build failure?
|
Test build #132810 has finished for PR 30736 at commit
|
|
Test build #132816 has finished for PR 30736 at commit
|
|
Kubernetes integration test starting |
|
Kubernetes integration test status failure |
|
Test build #132817 has finished for PR 30736 at commit
|
|
thanks for the review, merging to master! |
What changes were proposed in this pull request?
This is a followup of #30559 . The default parallelism config in Spark core is not good, as it's unclear where it applies. To not inherit this problem in Spark SQL, this PR refines the default parallelism SQL config, to make it clear that it only applies to leaf nodes.
Why are the changes needed?
Make the config clearer.
Does this PR introduce any user-facing change?
It changes an unreleased config.
How was this patch tested?
existing tests