[SPARK-1870] Made deployment with --jars work in yarn-standalone mode. #1013
Conversation
|
Merged build triggered. |
|
Merged build started. |
|
Merged build finished. All automated tests passed. |
|
All automated tests passed. |
|
@dbtsai Thanks for fixing it in branch-0.9! Did you verify the patch on a YARN cluster? |
|
Work in my local VM. Should work in real yarn cluster. Will test it tomorrow in the office. |
|
@dbtsai could you update the title of this PR to make it more clear what issue this relates to? |
|
Merged build triggered. |
|
Merged build started. |
|
Tested in PivotalHD 1.1 Yarn 4 node cluster. With --addjars file:///somePath/to/jar, launching spark application works. |
dbtsai
changed the title
|
Merged build finished. All automated tests passed. |
|
All automated tests passed. |
There was a problem hiding this comment.
I think this can be a val
There was a problem hiding this comment.
This actually returns a one element Array("") if there are no secondary jars. We should .filter(_.nonEmpty) after splitting.
There was a problem hiding this comment.
Thanks. You are right. It will add empty string to array, and then add the folder without file into classpath. Will fix in master as well.
|
Merged build triggered. |
There was a problem hiding this comment.
minor: Not your code, but could you bump .toBoolean up one line? I'm pretty sure it's under 100 characters.
|
Merged build triggered. |
|
Merged build started. |
1 similar comment
|
Merged build started. |
|
Merged build finished. All automated tests passed. |
|
All automated tests passed. |
|
Merged build finished. All automated tests passed. |
|
All automated tests passed. |
|
Merged build triggered. |
|
Merged build started. |
|
Merged build finished. |
|
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15600/ |
|
Jenkins, retest this please. |
|
Merged build triggered. |
|
Merged build started. |
|
Merged build finished. All automated tests passed. |
|
All automated tests passed. |
|
LGTM. Thanks! |
Ported from 1.0 branch to 0.9 branch. Sent secondary jars to distributed cache of all containers and add the cached jars to classpath before executors start. Author: DB Tsai <[email protected]> Closes #1013 from dbtsai/branch-0.9 and squashes the following commits: c5696f4 [DB Tsai] fix line too long b085f10 [DB Tsai] Make sure that empty string is filtered out when we get secondary jars 3cc1085 [DB Tsai] changed from var to val ab94aa1 [DB Tsai] Code formatting. 0956af9 [DB Tsai] Ported SPARK-1870 from 1.0 branch to 0.9 branch
5 participants
Ported from 1.0 branch to 0.9 branch. Sent secondary jars to distributed cache of all containers and add the cached jars to classpath before executors start.