Skip to content

[SPARK-7504] [YARN] NullPointerException when initializing SparkContext in YARN-cluster mode #6083

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
zzvara wants to merge 6 commits into from
Closed

[SPARK-7504] [YARN] NullPointerException when initializing SparkContext in YARN-cluster mode #6083

Changes from 5 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
9f287c5
[SPARK-7504] [YARN] NullPointerException when initializing SparkConte…
zzvara May 12, 2015
39e4fa3
SPARK-7504 [YARN] NullPointerException when initializing SparkContext…
zzvara May 12, 2015
4924e01
[SPARK-7504] [YARN] NullPointerException when initializing SparkConte…
zzvara May 12, 2015
ea2a5fe
[SPARK-7504] [YARN] NullPointerException when initializing SparkConte…
zzvara May 12, 2015
7c89b6e
Remove false line.
zzvara May 12, 2015
926bd96
Moved check to SparkContext.
zzvara May 12, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions core/src/main/scala/org/apache/spark/SparkConf.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -448,6 +448,15 @@ class SparkConf(loadDefaults: Boolean) extends Cloneable with Logging {
}
}
}

Copy link
Member

@srowen srowen May 13, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can probably remove this on merge, but there's a stray blank here.

Copy link
Contributor Author

@zzvara zzvara May 13, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As I saw, usually there is a blank line before ending of long methods and classes. You might remove it if you will.

// System property spark.yarn.app.id must be set if user code ran by AM on a YARN cluster
// yarn-standalone is deprecated, but still supported
if ((get("spark.master") == "yarn-cluster" || get("spark.master") == "yarn-standalone") &&
Copy link
Contributor

@vanzin vanzin May 12, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if ((master == "yarn-cluster" || master == "yarn-standalone") && !_conf.contains("spark.yarn.app.id""))

I don't see any method named get so I'm not sure your code would even compile.

Copy link
Contributor Author

@zzvara zzvara May 12, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are two gets. Tested on local and remote YARN clusters.

Copy link
Contributor

@vanzin vanzin May 12, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh my bad. I thought this was still in SparkContext. In fact, I think the check should be there. When I mentioned "check for spark.yarn.app.id in SparkConf" I didn't mean to change SparkConf, I meant to do as I suggested above (!_conf.contains("spark.yarn.app.id"")).

Copy link
Contributor Author

@zzvara zzvara May 12, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What we do here is actually an act of validating the configuration. It seems to be a good place inside validateSettings. There is no information required from outside to do this check.

Copy link
Contributor

@vanzin vanzin May 12, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SparkConf is not just for SparkContext.

get("spark.yarn.app.id", null) == null) {
throw new SparkException("Detected yarn-cluster mode, but isn't running on a cluster. " +
"Deployment to YARN is not supported directly by SparkContext. Please use spark-submit.")
}

}

/**
Expand Down