[SPARK-21490][core] Make sure SparkLauncher redirects needed streams.#18696
Closed
vanzin wants to merge 3 commits intoapache:masterfrom
Closed
[SPARK-21490][core] Make sure SparkLauncher redirects needed streams.#18696vanzin wants to merge 3 commits intoapache:masterfrom
vanzin wants to merge 3 commits intoapache:masterfrom
Conversation
The code was failing to account for some cases when setting up log redirection. For example, if a user redirected only stdout to a file, the launcher code would leave stderr without redirection, which could lead to child processes getting stuck because stderr wasn't being read. So detect cases where only one of the streams is redirected, and redirect the other stream to the log as appropriate. For the old "launch()" API, redirection of the unconfigured stream only happens if the user has explicitly requested for log redirection. Log redirection is on by default with "startApplication()". Most of the change is actually adding new unit tests to make sure the different cases work as expected. As part of that, I moved some test that were in the core/ module to the launcher/ module instead, since they don't depend on spark-submit.
|
Test build #79808 has started for PR 18696 at commit |
Contributor
Author
|
retest this please |
|
Test build #79848 has finished for PR 18696 at commit
|
|
Test build #79872 has finished for PR 18696 at commit
|
|
Test build #79913 has finished for PR 18696 at commit
|
Contributor
Author
|
@srowen want to take a look? |
Contributor
Author
|
retest this please |
|
Test build #80084 has finished for PR 18696 at commit
|
Contributor
Author
|
I'll wait until tomorrow for a review, otherwise I'll push this to unblock further work on this code. |
Contributor
Author
|
Merging to master. |
asfgit
pushed a commit
that referenced
this pull request
Aug 2, 2017
The code was failing to account for some cases when setting up log redirection. For example, if a user redirected only stdout to a file, the launcher code would leave stderr without redirection, which could lead to child processes getting stuck because stderr wasn't being read. So detect cases where only one of the streams is redirected, and redirect the other stream to the log as appropriate. For the old "launch()" API, redirection of the unconfigured stream only happens if the user has explicitly requested for log redirection. Log redirection is on by default with "startApplication()". Most of the change is actually adding new unit tests to make sure the different cases work as expected. As part of that, I moved some tests that were in the core/ module to the launcher/ module instead, since they don't depend on spark-submit. Author: Marcelo Vanzin <vanzin@cloudera.com> Closes #18696 from vanzin/SPARK-21490.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The code was failing to account for some cases when setting up log
redirection. For example, if a user redirected only stdout to a file,
the launcher code would leave stderr without redirection, which could
lead to child processes getting stuck because stderr wasn't being
read.
So detect cases where only one of the streams is redirected, and
redirect the other stream to the log as appropriate.
For the old "launch()" API, redirection of the unconfigured stream
only happens if the user has explicitly requested for log redirection.
Log redirection is on by default with "startApplication()".
Most of the change is actually adding new unit tests to make sure the
different cases work as expected. As part of that, I moved some tests
that were in the core/ module to the launcher/ module instead, since
they don't depend on spark-submit.