Skip to content

[SPARK-11821] Propagate Kerberos keytab for all environments #9859

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
woj-i wants to merge 5 commits into from
Closed

[SPARK-11821] Propagate Kerberos keytab for all environments #9859

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
0b55df5
[SPARK-11821] location of Kerberos keytab in a property for all envir…
woj-i Nov 18, 2015
b3b0d59
[SPARK-11821] bugfix of renewing Kerberos credentials in context of t…
woj-i Nov 20, 2015
7802c0c
[SPARK-11821] remove unnecesary config instance, add comment, provide…
woj-i Nov 24, 2015
9365a7f
[SPARK-11821] Documentation update, move a code comment to the docume…
woj-i Nov 25, 2015
449cbbb
documentation nits
woj-i Nov 26, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -545,6 +545,10 @@ object SparkSubmit {
if (args.isPython) {
sysProps.put("spark.yarn.isPython", "true")
}
}

// assure a keytab is available from any place in a JVM
if (clusterManager == YARN || clusterManager == LOCAL) {
if (args.principal != null) {
require(args.keytab != null, "Keytab must be specified when principal is specified")
if (!new File(args.keytab).exists()) {
Expand Down
4 changes: 2 additions & 2 deletions docs/running-on-yarn.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -349,14 +349,14 @@ If you need a reference to the proper location to put log files in the YARN so t
<td>
The full path to the file that contains the keytab for the principal specified above.
This keytab will be copied to the node running the YARN Application Master via the Secure Distributed Cache,
for renewing the login tickets and the delegation tokens periodically.
for renewing the login tickets and the delegation tokens periodically. (Works also with the "local" master)
</td>
</tr>
<tr>
<td><code>spark.yarn.principal</code></td>
<td>(none)</td>
<td>
Principal to be used to login to KDC, while running on secure HDFS.
Principal to be used to login to KDC, while running on secure HDFS. (Works also with the "local" master)
</td>
</tr>
<tr>
Expand Down
7 changes: 4 additions & 3 deletions docs/sql-programming-guide.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1614,7 +1614,8 @@ This command builds a new assembly jar that includes Hive. Note that this Hive a
on all of the worker nodes, as they will need access to the Hive serialization and deserialization libraries
(SerDes) in order to access data stored in Hive.

Configuration of Hive is done by placing your `hive-site.xml` file in `conf/`. Please note when running
Configuration of Hive is done by placing your `hive-site.xml`, `core-site.xml` (for security configuration),
`hdfs-site.xml` (for HDFS configuration) file in `conf/`. Please note when running
the query on a YARN cluster (`cluster` mode), the `datanucleus` jars under the `lib_managed/jars` directory
and `hive-site.xml` under `conf/` directory need to be available on the driver and all executors launched by the
YARN cluster. The convenient way to do this is adding them through the `--jars` option and `--file` option of the
Expand Down Expand Up @@ -2020,7 +2021,7 @@ Beeline will ask you for a username and password. In non-secure mode, simply ent
your machine and a blank password. For secure mode, please follow the instructions given in the
[beeline documentation](https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Clients).

Configuration of Hive is done by placing your `hive-site.xml` file in `conf/`.
Configuration of Hive is done by placing your `hive-site.xml`, `core-site.xml` and `hdfs-site.xml` files in `conf/`.

You may also use the beeline script that comes with Hive.

Expand All @@ -2045,7 +2046,7 @@ To start the Spark SQL CLI, run the following in the Spark directory:

./bin/spark-sql

Configuration of Hive is done by placing your `hive-site.xml` file in `conf/`.
Configuration of Hive is done by placing your `hive-site.xml`, `core-site.xml` and `hdfs-site.xml` files in `conf/`.
You may run `./bin/spark-sql --help` for a complete list of all available
options.

Expand Down