Skip to content

[SPARK-11821] Propagate Kerberos keytab for all environments #9859

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
woj-i wants to merge 5 commits into from
Closed

[SPARK-11821] Propagate Kerberos keytab for all environments #9859

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
0b55df5
[SPARK-11821] location of Kerberos keytab in a property for all envir…
woj-i Nov 18, 2015
b3b0d59
[SPARK-11821] bugfix of renewing Kerberos credentials in context of t…
woj-i Nov 20, 2015
7802c0c
[SPARK-11821] remove unnecesary config instance, add comment, provide…
woj-i Nov 24, 2015
9365a7f
[SPARK-11821] Documentation update, move a code comment to the docume…
woj-i Nov 25, 2015
449cbbb
documentation nits
woj-i Nov 26, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -545,6 +545,10 @@ object SparkSubmit {
if (args.isPython) {
sysProps.put("spark.yarn.isPython", "true")
}
}

// assure a keytab is available from any place in a JVM
if (clusterManager == YARN || clusterManager == LOCAL) {
if (args.principal != null) {
require(args.keytab != null, "Keytab must be specified when principal is specified")
if (!new File(args.keytab).exists()) {
Expand Down
6 changes: 5 additions & 1 deletion sql/hive/src/main/scala/org/apache/spark/sql/hive/client/ClientWrapper.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,10 +20,12 @@ package org.apache.spark.sql.hive.client
import java.io.{File, PrintStream}
import java.util.{Map => JMap}

import org.apache.hadoop.conf.Configuration

import scala.collection.JavaConverters._
import scala.language.reflectiveCalls

import org.apache.hadoop.fs.Path
import org.apache.hadoop.fs.{CommonConfigurationKeysPublic, Path}
import org.apache.hadoop.hive.conf.HiveConf
import org.apache.hadoop.hive.metastore.api.{Database, FieldSchema}
import org.apache.hadoop.hive.metastore.{TableType => HTableType}
Expand Down Expand Up @@ -167,6 +169,8 @@ private[hive] class ClientWrapper(
} else {
logInfo("Attempting to login to Kerberos" +
s" using principal: ${principalName} and keytab: ${keytabFileName}")
// make sure the core-site.xml is provided in your classpath
Copy link
Contributor

@vanzin vanzin Nov 24, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Better than this would be to update docs/sql-programming-guide.md. It currently only mentions hive-site.xml, when really you need core-site.xml and hdfs-site.xml too (otherwise kerberos, in your case, or things like HDFS HA won't work).

The other changes in this file can be reverted, too.

// (with hadoop.security.authentication=kerberos)
UserGroupInformation.loginUserFromKeytab(principalName, keytabFileName)
}
}
Expand Down