Skip to content

SPARK-1652: Remove incorrect deprecation warning in spark-submit #578

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
pwendell wants to merge 3 commits into from
Closed

SPARK-1652: Remove incorrect deprecation warning in spark-submit #578

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
4ac9cac
SPARK-1652: spark-submit for yarn prints warnings even though calling…
pwendell Apr 28, 2014
b5be173
Review feedback
pwendell Apr 28, 2014
96027c7
Test fixes
pwendell Apr 28, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -124,6 +124,8 @@ object SparkSubmit {
}
} else if (clusterManager == YARN) {
childMainClass = "org.apache.spark.deploy.yarn.Client"
// Special flag to avoid deprecation warnings at the client
sysProps("SPARK_SUBMIT_YARN") = "true"

@sryza sryza Apr 28, 2014

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: is YARN relevant? Could call it SPARK_SUBMIT and use it for standalone with cluster deploy mode as well.

@pwendell pwendell Apr 28, 2014

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One problem we have with existing configs and envs is that someone creates it for one purpose and then it gets used for a zillion other purposes and it becomes really hard to understand what it's for.

So in general I'm biased at this point towards creating fairly narrowly scoped internal options whenever possible and generalizing them (lazily) when it's necessary. In this case I'd actually hope we can just remove this thing if/when we decide to remove the YARN clients.

@pwendell pwendell Apr 28, 2014

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sryza if you feel really strongly I could make it more general. I'm a bit scarred from some cases where options like this have taken on lives of their own :)

@sryza sryza Apr 28, 2014

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Even if it's a hack / workaround, I think it's probably better to work with the assumption that it'll end up sticking around for a while. With that in mind, a system property that signifies we started something from spark-submit makes more sense to me than a system property that signifying we started from spark-submit AND happen to be running YARN. Especially if we'll need a similar one for standalone mode. Do you think SPARK_SUBMIT_YARN would be less likely to be abused than SPARK_SUBMIT?

That said, it's a tiny issue, and I don't feel super strongly, so if you feel differently, go ahead with the current patch.

childArgs += ("--jar", appArgs.primaryResource)
childArgs += ("--class", appArgs.mainClass)
}
Expand Down
6 changes: 4 additions & 2 deletions yarn/alpha/src/main/scala/org/apache/spark/deploy/yarn/Client.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -169,8 +169,10 @@ class Client(clientArgs: ClientArguments, hadoopConf: Configuration, spConf: Spa
object Client {

def main(argStrings: Array[String]) {
println("WARNING: This client is deprecated and will be removed in a future version of Spark.")
println("Use ./bin/spark-submit with \"--master yarn\"")
if (!sys.props.contains("SPARK_SUBMIT_YARN")) {
println("WARNING: This client is deprecated and will be removed in a " +
"future version of Spark. Use ./bin/spark-submit with \"--master yarn\"")
}

// Set an env variable indicating we are running in YARN mode.
// Note that anything with SPARK prefix gets propagated to all (remote) processes
Expand Down
6 changes: 4 additions & 2 deletions yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -171,8 +171,10 @@ class Client(clientArgs: ClientArguments, hadoopConf: Configuration, spConf: Spa
object Client {

def main(argStrings: Array[String]) {
println("WARNING: This client is deprecated and will be removed in a future version of Spark.")
println("Use ./bin/spark-submit with \"--master yarn\"")
if (!sys.props.contains("SPARK_SUBMIT_YARN")) {
println("WARNING: This client is deprecated and will be removed in a " +
"future version of Spark. Use ./bin/spark-submit with \"--master yarn\"")
}

// Set an env variable indicating we are running in YARN mode.
// Note: anything env variable with SPARK_ prefix gets propagated to all (remote) processes -
Expand Down