Skip to content

[SPARK-11227][CORE] UnknownHostException can be thrown when NameNode HA is enabled. #13738

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
sarutak wants to merge 3 commits into from
Closed

[SPARK-11227][CORE] UnknownHostException can be thrown when NameNode HA is enabled. #13738

Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
d41bfe0
Force to load HdfsConfiguration
sarutak Jun 17, 2016
2d1ca93
Merge branch 'master' of git://git.apache.org/spark into SPARK-11227
sarutak Jul 25, 2016
fca2b4a
Modified SparkContext to enforce loading hdfs-site.xml
sarutak Jul 25, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -421,6 +421,13 @@ object SparkHadoopUtil {

val SPARK_YARN_CREDS_COUNTER_DELIM = "-"

// Just load HdfsConfiguration into the class loader to add
// hdfs-site.xml as a default configuration file otherwise
// some HDFS related configurations doesn't ship to Executors and
// it can cause UnknownHostException when NameNode HA is enabled.
// See SPARK-11227 for more details.
Utils.classForName("org.apache.hadoop.hdfs.HdfsConfiguration")
Copy link
Member

@srowen srowen Jun 21, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this need to be loaded by reflection -- is it not in Hadoop 2.2?

Copy link
Member Author

@sarutak sarutak Jun 21, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As you mentioned, Hadoop 2.2 have HdfsConfiguration but are there any way to load HdfsConfiguration explicitly without reflection?

Copy link
Member

@srowen srowen Jun 21, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just reference it in any way, but, I guess we should ask, what does classloading do that we need, and is there any way to do that directly? this is fairly indirect. Is it that Configuration.addDefaultResource("hdfs-site.xml"); must be called?

Copy link
Member Author

@sarutak sarutak Jun 21, 2016 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, calling Configuration.addDefaultResource("hdfs-site.xml") directly may be better.

Copy link
Contributor

@tgravescs tgravescs Jun 21, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

actually I'd prefer not to reference hdfs-site.xml directly. HdfsConfiguration should know what it needs to load. For instance it also loads the defaults. HdfsConfiguration is also marked as @InterfaceAudience.Private so ideally we shouldn't be using it directly. Based on my other comments I would like to understand better why this isn't loaded on driver already.


/**
* Number of records to update input metrics when reading from HadoopRDDs.
*
Expand Down