[SPARK-26737][YARN] Executor/Task STDERR & STDOUT log urls are not correct in Yarn deployment mode #23659
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
correct in Yarn deployment mode
|
Test build #101707 has finished for PR 23659 at commit
|
HeartSaVioR
reviewed
Jan 26, 2019
| // Add log urls | ||
| container.foreach { c => | ||
| sys.env.filterKeys(_.endsWith("USER")).foreach { user => | ||
| sys.env.get("SPARK_USER").orElse(sys.env.get("USER")).foreach { user => |
Contributor
Contributor
There was a problem hiding this comment.
Maybe explicitly set the SPARK_USER env variable to the value of Utils.getCurrentUserName? Instead of trying to figure out what's the correct value from the env. (USER could be wrong if you have kerberos, for example.)
Contributor
|
Please add |
devaraj-kavali
changed the title
Contributor
|
Ah sorry I thought you didn't file a new issue and submit a PR to SPARK-22404. Could you please remove `[FOLLOWUP]? Sorry to bother you. |
HyukjinKwon
changed the title
Contributor
|
#23260 will actually fix this too, so might as well just stick to that one patch. I'll close this one and put a note on the bug when that patch is merged. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
As part of the SPARK-22404 PR, base log url formation has been changed for unmanaged am which causes to include key is also part of the base url. This PR corrects the addition of key into base url.
How was this patch tested?
Verified the url manually and see that this has been corrected with the PR change.
Before this PR the base log url is like,
Unmanaged AM enabled:
http://ip:8042/node/containerlogs/container_1544212645385_0252_01_000002/(USER, devaraj)Unmanaged AM disabled:
http://ip:8042/node/containerlogs/container_1544212645385_0254_01_000002/(SPARK_USER, devaraj)With the PR change, base log url is like this:
http://ip:8042/node/containerlogs/container_1544212645385_0253_01_000002/devaraj