[SPARK-26023][SQL] Dumping truncated plans and generated code to a file #23018
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Member
Author
|
The is #22429 without the |
|
Test build #98742 has finished for PR 23018 at commit
|
wangyum
reviewed
Nov 13, 2018
| verbose: Boolean, | ||
| addSuffix: Boolean, | ||
| maxFields: Option[Int]): Unit = { | ||
| generateTreeString(0, Nil, writer, verbose, "", addSuffix) |
Member
There was a problem hiding this comment.
How about add another function only save nodeName? We can use it here: #22879
Member
There was a problem hiding this comment.
If #22879 is merged first, we should add that function here. If this one is merged first, that PR better should have the function there.
Member
Author
There was a problem hiding this comment.
I would prefer to avoid overcomplicating the PR again, frankly speaking.
Member
|
Looks fine to me. adding @cloud-fan and @hvanhovell |
|
Test build #98763 has finished for PR 23018 at commit
|
Contributor
|
retest this please |
|
Test build #98771 has finished for PR 23018 at commit
|
Contributor
|
Merging to master. Thanks! |
Member
Author
|
@hvanhovell @HyukjinKwon Thank you for the review. |
This was referenced Nov 14, 2018
jackylee-ch
pushed a commit
to jackylee-ch/spark
that referenced
this pull request
Feb 18, 2019
## What changes were proposed in this pull request? In the PR, I propose new method for debugging queries by dumping info about their execution to a file. It saves logical, optimized and physical plan similar to the `explain()` method + generated code. One of the advantages of the method over `explain` is it does not materializes full output as one string in memory which can cause OOMs. ## How was this patch tested? Added a few tests to `QueryExecutionSuite` to check positive and negative scenarios. Closes apache#23018 from MaxGekk/truncated-plan-to-file. Authored-by: Maxim Gekk <[email protected]> Signed-off-by: Herman van Hovell <[email protected]>
dvallejo
pushed a commit
to Telefonica/spark
that referenced
this pull request
Aug 31, 2021
The PR puts in a limit on the size of a debug string generated for a tree node. Helps to fix out of memory errors when large plans have huge debug strings. In addition to SPARK-26103, this should also address SPARK-23904 and SPARK-25380. AN alternative solution was proposed in apache#23076, but that solution doesn't address all the cases that can cause a large query. This limit is only on calls treeString that don't pass a Writer, which makes it play nicely with apache#22429, apache#23018 and apache#23039. Full plans can be written to files, but truncated plans will be used when strings are held in memory, such as for the UI. - A new configuration parameter called spark.sql.debug.maxPlanLength was added to control the length of the plans. - When plans are truncated, "..." is printed to indicate that it isn't a full plan - A warning is printed out the first time a truncated plan is displayed. The warning explains what happened and how to adjust the limit. Unit tests were created for the new SizeLimitedWriter. Also a unit test for TreeNode was created that checks that a long plan is correctly truncated. Closes apache#23169 from DaveDeCaprio/text-plan-size. Lead-authored-by: Dave DeCaprio <[email protected]> Co-authored-by: David DeCaprio <[email protected]> Signed-off-by: Marcelo Vanzin <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
In the PR, I propose new method for debugging queries by dumping info about their execution to a file. It saves logical, optimized and physical plan similar to the
explain()method + generated code. One of the advantages of the method overexplainis it does not materializes full output as one string in memory which can cause OOMs.How was this patch tested?
Added a few tests to
QueryExecutionSuiteto check positive and negative scenarios.