Skip to content

[SPARK-30845] Do not upload local pyspark archives for spark-submit on Yarn #27598

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
shanyu wants to merge 4 commits into from
Closed

[SPARK-30845] Do not upload local pyspark archives for spark-submit on Yarn #27598

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
624206e
Merge pull request #1 from apache/master
shanyu Sep 6, 2019
df86c1e
Merge pull request #3 from apache/master
shanyu Feb 16, 2020
f212848
[SPARK-30845] Do not upload local pyspark archives for spark-submit o…
shanyu Feb 16, 2020
20a7a9c
Fix typo
shanyu Jun 5, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -635,7 +635,12 @@ private[spark] class Client(
distribute(args.primaryPyFile, appMasterOnly = true)
}

pySparkArchives.foreach { f => distribute(f) }
pySparkArchives.foreach { f =>
Copy link
Member

@HyukjinKwon HyukjinKwon Jun 9, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does it work when Spark is not installed in other nodes? IIRC, we can run the application in the cluster where Spark is not installed because the jars are shipped together in Yarn cluster.

Likewise, PySpark was able to run. From my very cursory look, it's going to break this case because it will not distribute the local pyspark archive anymore. Can you confirm this @shanyu and @tgravescs?

Copy link
Contributor

@tgravescs tgravescs Jun 9, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is the case someone explicitly put local: on the url so its expected to be on every machine. YARN distributes everything that is file: or downloads it if its hdfs:

Copy link
Member

@HyukjinKwon HyukjinKwon Jun 10, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, thanks for clarification. LGTM

val uri = Utils.resolveURI(f)
if (uri.getScheme != Utils.LOCAL_SCHEME) {
distribute(f)
}
}

// The python files list needs to be treated especially. All files that are not an
// archive need to be placed in a subdirectory that will be added to PYTHONPATH.
Expand Down