Skip to content

[SPARK-10181][SQL] Do kerberos login for credentials during hive client initialization #9272

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
yolandagao wants to merge 17 commits into from
Closed

[SPARK-10181][SQL] Do kerberos login for credentials during hive client initialization #9272

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
59253b3
SPARK-10181 UserGroupInformation class needs to be shared for login c…
yolandagao Oct 21, 2015
3e79e94
Merge remote branch 'upstream/master'
yolandagao Oct 22, 2015
9efa819
Revert "SPARK-10181 UserGroupInformation class needs to be shared for…
yolandagao Oct 24, 2015
7934585
Merge remote branch 'upstream/master'
yolandagao Oct 25, 2015
30a7d05
SPARK-10181 Do kerberos login for credentials in hive client initiali…
yolandagao Oct 25, 2015
836bb8b
Merge remote branch 'upstream/master'
yolandagao Oct 25, 2015
7ec246f
Merge remote branch 'upstream/master'
yolandagao Nov 3, 2015
1fe7593
SPARK-10181 Do kerberos login for credentials during hive client init…
yolandagao Nov 3, 2015
3f0fee9
Merge remote branch 'upstream/master'
yolandagao Nov 4, 2015
06ee550
Merge remote branch 'upstream/master'
yolandagao Nov 5, 2015
906b3cb
SPARK-10181 Do kerberos login for credentials during hive client init…
yolandagao Nov 5, 2015
ef927e7
Merge remote branch 'upstream/master'
yolandagao Nov 5, 2015
7d09f5d
SPARK-10181 Do kerberos login for credentials during hive client init…
yolandagao Nov 5, 2015
36d5ef8
Merge remote-tracking branch 'upstream/master'
Nov 14, 2015
cdb3aa5
Merge remote-tracking branch 'upstream/master'
Nov 14, 2015
1fbc372
SPARK-10181 Add comments explaining the keytab principal confs propag…
yolandagao Nov 14, 2015
caf51a7
SPARK-10181 Add more comments for clarification
yolandagao Nov 14, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 14 additions & 3 deletions core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ import org.apache.ivy.plugins.matcher.GlobPatternMatcher
import org.apache.ivy.plugins.repository.file.FileRepository
import org.apache.ivy.plugins.resolver.{FileSystemResolver, ChainResolver, IBiblioResolver}

import org.apache.spark.{SparkUserAppException, SPARK_VERSION}
import org.apache.spark.{SparkException, SparkUserAppException, SPARK_VERSION}
import org.apache.spark.api.r.RUtils
import org.apache.spark.deploy.rest._
import org.apache.spark.util.{ChildFirstURLClassLoader, MutableURLClassLoader, Utils}
Expand Down Expand Up @@ -521,8 +521,19 @@ object SparkSubmit {
sysProps.put("spark.yarn.isPython", "true")
}
if (args.principal != null) {
require(args.keytab != null, "Keytab must be specified when the keytab is specified")
UserGroupInformation.loginUserFromKeytab(args.principal, args.keytab)
require(args.keytab != null, "Keytab must be specified when principal is specified")
if (!new File(args.keytab).exists()) {
throw new SparkException(s"Keytab file: ${args.keytab} does not exist")
} else {
// Add keytab and principal configurations in sysProps to make them available
// for later use; e.g. in spark sql, the isolated class loader used to talk
// to HiveMetastore will use these settings. They will be set as Java system
// properties and then loaded by SparkConf
sysProps.put("spark.yarn.keytab", args.keytab)
sysProps.put("spark.yarn.principal", args.principal)

UserGroupInformation.loginUserFromKeytab(args.principal, args.keytab)
}
}
}

Expand Down
24 changes: 23 additions & 1 deletion sql/hive/src/main/scala/org/apache/spark/sql/hive/client/ClientWrapper.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,9 +32,10 @@ import org.apache.hadoop.hive.ql.processors._
import org.apache.hadoop.hive.ql.session.SessionState
import org.apache.hadoop.hive.ql.{Driver, metadata}
import org.apache.hadoop.hive.shims.{HadoopShims, ShimLoader}
import org.apache.hadoop.security.UserGroupInformation
import org.apache.hadoop.util.VersionInfo

import org.apache.spark.Logging
import org.apache.spark.{SparkConf, SparkException, Logging}
import org.apache.spark.sql.catalyst.expressions.Expression
import org.apache.spark.sql.execution.QueryExecutionException
import org.apache.spark.util.{CircularBuffer, Utils}
Expand Down Expand Up @@ -149,6 +150,27 @@ private[hive] class ClientWrapper(
val original = Thread.currentThread().getContextClassLoader
// Switch to the initClassLoader.
Thread.currentThread().setContextClassLoader(initClassLoader)

// Set up kerberos credentials for UserGroupInformation.loginUser within
// current class loader
// Instead of using the spark conf of the current spark context, a new
// instance of SparkConf is needed for the original value of spark.yarn.keytab
// and spark.yarn.principal set in SparkSubmit, as yarn.Client resets the
// keytab configuration for the link name in distributed cache
val sparkConf = new SparkConf
Copy link
Contributor

@yhuai yhuai Nov 9, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of creating a new Spark Conf, can we use SparkEnv.get.conf to get the spark conf associated with the current spark context?

Copy link
Contributor Author

@yolandagao yolandagao Nov 13, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@yhuai Sorry for the late response. I did the testing after changing to SparkEnv.get.conf but it didn't work. The reason is that Yarn Client.scala resets property spark.yarn.keytab by appending some random strings to keytab file name during setupCredentials, which will be used as the link name in distributed cache. I think the one for the link name should be actually separated from the original keytab setting, e.g. using different property names.

Copy link
Contributor

@yhuai yhuai Nov 13, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@yolandagao Thanks for the explanation. Can you add comments to your code (including why we need to put those confs to sysProps and why we need to create a new SparkConf at here)? Basically, we need to document the flow of how these confs get propagated. Otherwise, it is not obvious why we need to do this change. Thanks!

Copy link
Contributor Author

@yolandagao yolandagao Nov 14, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@yhuai Sure. I should have done this earlier, to make everything clearer:) Added some comments there, and please help review. Thank you!

if (sparkConf.contains("spark.yarn.principal") && sparkConf.contains("spark.yarn.keytab")) {
Copy link
Contributor

@yhuai yhuai Nov 14, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's make it clear that we set these two settings in SparkSubmit.

val principalName = sparkConf.get("spark.yarn.principal")
val keytabFileName = sparkConf.get("spark.yarn.keytab")
if (!new File(keytabFileName).exists()) {
throw new SparkException(s"Keytab file: ${keytabFileName}" +
" specified in spark.yarn.keytab does not exist")
} else {
logInfo("Attempting to login to Kerberos" +
s" using principal: ${principalName} and keytab: ${keytabFileName}")
UserGroupInformation.loginUserFromKeytab(principalName, keytabFileName)
}
}

val ret = try {
val initialConf = new HiveConf(classOf[SessionState])
// HiveConf is a Hadoop Configuration, which has a field of classLoader and
Expand Down