Skip to content

Commit 0a30269

Browse files
zjffduMarcelo Vanzin
authored andcommitted
[SPARK-14845][SPARK_SUBMIT][YARN] spark.files in properties file is n…
## What changes were proposed in this pull request? initialize SparkSubmitArgument#files first from spark-submit arguments then from properties file, so that sys property spark.yarn.dist.files will be set correctly. ``` OptionAssigner(args.files, YARN, ALL_DEPLOY_MODES, sysProp = "spark.yarn.dist.files"), ``` ## How was this patch tested? manul test. file defined in properties file is also distributed to driver in yarn-cluster mode. Author: Jeff Zhang <[email protected]> Closes #12656 from zjffdu/SPARK-14845.
1 parent 0513c3a commit 0a30269

File tree

1 file changed

+1
-0
lines changed

1 file changed

+1
-0
lines changed

core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -173,6 +173,7 @@ private[deploy] class SparkSubmitArguments(args: Seq[String], env: Map[String, S
173173
.orNull
174174
name = Option(name).orElse(sparkProperties.get("spark.app.name")).orNull
175175
jars = Option(jars).orElse(sparkProperties.get("spark.jars")).orNull
176+
files = Option(files).orElse(sparkProperties.get("spark.files")).orNull
176177
ivyRepoPath = sparkProperties.get("spark.jars.ivy").orNull
177178
packages = Option(packages).orElse(sparkProperties.get("spark.jars.packages")).orNull
178179
packagesExclusions = Option(packagesExclusions)

0 commit comments

Comments
 (0)