[SPARK-5342][YARN] Allow long running Spark apps to run on secure YARN/HDFS

[harishreedharan]

Current Spark apps running on Secure YARN/HDFS would not be able to write data
to HDFS after 7 days, since delegation tokens cannot be renewed beyond that. This
means Spark Streaming apps will not be able to run on Secure YARN.

This commit adds basic functionality to fix this issue. In this patch:

• new parameters are added - principal and keytab, which can be used to login to a KDC
• the client logs in, and then get tokens to start the AM
• the keytab is copied to the staging directory
• the AM waits for 60% of the time till expiry of the tokens and then logs in using the keytab
• each time after 60% of the time, new tokens are created and sent to the executors

Currently, to avoid complicating the architecture, we set the keytab and principal in the
SparkHadoopUtil singleton, and schedule a login. Once the login is completed, a callback is scheduled.

This is being posted for feedback, so I can gather feedback on the general implementation.

There are currently a bunch of things to do:

• logging
• testing - I plan to manually test this soon. If you have ideas of how to add unit tests, comment.
• add code to ensure that if these params are set in non-YARN cluster mode, we complain
• documentation
• Have the executors request for credentials from the AM, so that retries are possible.

[SparkQA]

Test build #27709 timed out for PR 4688 at commit 2b0d745 after a configured wait of 120m.

[SparkQA]

Test build #27730 has finished for PR 4688 at commit f8fe694.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class UpdateCredentials(newCredentials: SerializableBuffer)

[harishreedharan]

Jenkins, retest this

[harishreedharan]

Jenkins, test this please

[SparkQA]

Test build #27739 has finished for PR 4688 at commit f8fe694.

• This patch fails MiMa tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class UpdateCredentials(newCredentials: SerializableBuffer)

[vanzin]

Hey @harishreedharan ,

Is this avoidable? This is a @Private API and it doesn't seem to exist on hadoop 1.x (which is probably the cause of the build failures).

[harishreedharan]

It does not seem like this is why the build is failing. It failed MiMA checks for some reason. I will look at it in some time.

[vanzin]

It says that, but it fails because when building Spark for the checks, it does so with Hadoop 1.x. And compilation at that step is failing.

[harishreedharan]

I take that back. Looks like this failed due to a compile issue. Let me see what we can do.

[harishreedharan]

We can probably move it out to a Utils class in the YARN module, and we should be ok, since this code is used only by the AM.

[harishreedharan]

Looks like ClientSuite.scala has a bug -- it does not set YARN mode to true, so we end up getting an instance of SparkHadoopUtil, instead of YarnSparkHadoopUtil

[SparkQA]

Test build #27769 has finished for PR 4688 at commit bcfc374.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class UpdateCredentials(newCredentials: SerializableBuffer)
  • public class JavaPowerIterationClusteringExample

[SparkQA]

Test build #27770 has finished for PR 4688 at commit d282d7a.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class UpdateCredentials(newCredentials: SerializableBuffer)

[harishreedharan]

Looks like the build issue was fixed by the last commit.

[SparkQA]

Test build #27828 has finished for PR 4688 at commit 41efde0.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class UpdateCredentials(newCredentials: SerializableBuffer)

[SparkQA]

Test build #27872 has finished for PR 4688 at commit fb27f46.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class UpdateCredentials(newCredentials: SerializableBuffer)

[SparkQA]

Test build #27879 has finished for PR 4688 at commit 8c6928a.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class UpdateCredentials(newCredentials: SerializableBuffer)

[SparkQA]

Test build #27901 has finished for PR 4688 at commit d79b2b9.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class UpdateCredentials(newCredentials: SerializableBuffer)

[tgravescs]

Generally speaking its a not a good idea to ship keytabs around. If one is compromised then the user who gets it can do anything as that user forever. YARN/HDFS generally uses tokens because they have a limited lifetime so if one was compromised it would be for a limited amount of time. I know our internal paranoids would not allow this change to be rolled out internally.

This really needs to be fixed on the YARN side but unfortunately not much work there: https://issues.apache.org/jira/browse/YARN-896

That said I realize there aren't many options right now so as long as we make sure this doesn't happen automatically I don't have any better ideas (other then work on the YARN jira).

[tgravescs]

So actually one other option would be to build our own mechanism for pushing new tgt's and updating the UGI. Storm actually has this functionality, you could look there as reference.

[harishreedharan]

Thanks for taking a look, Tom!

To get new tgts, we'd still need the keytab, right? I am wondering how to get around having the keytab being shipped to the AM, since the client can go away in cluster mode and the app can keep running.

I will take a look at storm later today.

[tgravescs]

The user has to have a cron job (or similar) that does a kinit and then would push the credentials out. This would be harder on spark as it would have to do it for each application and then you would have to create that communication protocol to send it.

I did also just notice that apache slider is actually using the keytab approach similar to what you are proposing.

Some other cons though to using the keytab approach:

• some kdc's might consider it a replay attack if on lots of hosts are doing kinit all at once (ie you are using headless user type keytab)
• if not using headless user keytab then you need to generate per host keytab and figure out how to distribute properly. This is a headache for operations team to generate those and manage them.

[harishreedharan]

Not sure if you got a chance to look at the patch. We are actually not logging in on every executor, but only on the Client and the AM. The client logs in, sets the tokens for the AM and the AM starts using those. At 60% of the expiry of AM's tokens (so around 4.2 days), the AM logs in and then generates tokens which are passed around the executors. So the replay attack should not be a concern, since the login happens days apart.

The keytab is actually used only by the client once and the AM once. It is also passed around securely - we copy it to the application staging directory and the AM reads it from there. So I don't think the distribution of the keytab is actually much of an issue.

[tgravescs]

ah ok, sorry no I did not look at the code. Thanks for the explanation.

So yes as long as all the permissions are setup properly anything copied to staging and then localized should be secure. Its more of a risk aspect then. If you have secure storm cluster but not using ssl someone could sniff it. or if there is a bug with permissions then someone could get it and the keytab lasts forever, vs just getting a single token and it lasts a limited time.

[SparkQA]

Test build #31203 has finished for PR 4688 at commit 09fe224.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

[SparkQA]

Test build #31231 has finished for PR 4688 at commit 611923a.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

[harishreedharan]

Looks like a Jenkins or build issue. Other builds are failing too: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31219/

Will restart build later

[harishreedharan]

Jenkins, test this please

[SparkQA]

Test build #31244 has finished for PR 4688 at commit 611923a.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

[harishreedharan]

@tgravescs - Do you have any more comments? I'd like to get this into 1.4 if possible.

[harishreedharan]

@tgravescs - Jira for Hive and HBase token updates: https://issues.apache.org/jira/browse/SPARK-7252

[tgravescs]

@harishreedharan sorry I've been swamped today, I'll look at this later tonight.

[harishreedharan]

Thanks!

[tgravescs]

this is in the wrong order. you want to do this after the await right?

[SparkQA]

Test build #31395 has finished for PR 4688 at commit 36eb8a9.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch removes the following dependencies:
  • RoaringBitmap-0.4.5.jar
  • activation-1.1.jar
  • akka-actor_2.10-2.3.4-spark.jar
  • akka-remote_2.10-2.3.4-spark.jar
  • akka-slf4j_2.10-2.3.4-spark.jar
  • aopalliance-1.0.jar
  • arpack_combined_all-0.1.jar
  • avro-1.7.7.jar
  • breeze-macros_2.10-0.11.2.jar
  • breeze_2.10-0.11.2.jar
  • chill-java-0.5.0.jar
  • chill_2.10-0.5.0.jar
  • commons-beanutils-1.7.0.jar
  • commons-beanutils-core-1.8.0.jar
  • commons-cli-1.2.jar
  • commons-codec-1.10.jar
  • commons-collections-3.2.1.jar
  • commons-compress-1.4.1.jar
  • commons-configuration-1.6.jar
  • commons-digester-1.8.jar
  • commons-httpclient-3.1.jar
  • commons-io-2.1.jar
  • commons-lang-2.5.jar
  • commons-lang3-3.3.2.jar
  • commons-math-2.1.jar
  • commons-math3-3.4.1.jar
  • commons-net-2.2.jar
  • compress-lzf-1.0.0.jar
  • config-1.2.1.jar
  • core-1.1.2.jar
  • curator-client-2.4.0.jar
  • curator-framework-2.4.0.jar
  • curator-recipes-2.4.0.jar
  • gmbal-api-only-3.0.0-b023.jar
  • grizzly-framework-2.1.2.jar
  • grizzly-http-2.1.2.jar
  • grizzly-http-server-2.1.2.jar
  • grizzly-http-servlet-2.1.2.jar
  • grizzly-rcm-2.1.2.jar
  • groovy-all-2.3.7.jar
  • guava-14.0.1.jar
  • guice-3.0.jar
  • hadoop-annotations-2.2.0.jar
  • hadoop-auth-2.2.0.jar
  • hadoop-client-2.2.0.jar
  • hadoop-common-2.2.0.jar
  • hadoop-hdfs-2.2.0.jar
  • hadoop-mapreduce-client-app-2.2.0.jar
  • hadoop-mapreduce-client-common-2.2.0.jar
  • hadoop-mapreduce-client-core-2.2.0.jar
  • hadoop-mapreduce-client-jobclient-2.2.0.jar
  • hadoop-mapreduce-client-shuffle-2.2.0.jar
  • hadoop-yarn-api-2.2.0.jar
  • hadoop-yarn-client-2.2.0.jar
  • hadoop-yarn-common-2.2.0.jar
  • hadoop-yarn-server-common-2.2.0.jar
  • ivy-2.4.0.jar
  • jackson-annotations-2.4.0.jar
  • jackson-core-2.4.4.jar
  • jackson-core-asl-1.8.8.jar
  • jackson-databind-2.4.4.jar
  • jackson-jaxrs-1.8.8.jar
  • jackson-mapper-asl-1.8.8.jar
  • jackson-module-scala_2.10-2.4.4.jar
  • jackson-xc-1.8.8.jar
  • jansi-1.4.jar
  • javax.inject-1.jar
  • javax.servlet-3.0.0.v201112011016.jar
  • javax.servlet-3.1.jar
  • javax.servlet-api-3.0.1.jar
  • jaxb-api-2.2.2.jar
  • jaxb-impl-2.2.3-1.jar
  • jcl-over-slf4j-1.7.10.jar
  • jersey-client-1.9.jar
  • jersey-core-1.9.jar
  • jersey-grizzly2-1.9.jar
  • jersey-guice-1.9.jar
  • jersey-json-1.9.jar
  • jersey-server-1.9.jar
  • jersey-test-framework-core-1.9.jar
  • jersey-test-framework-grizzly2-1.9.jar
  • jets3t-0.7.1.jar
  • jettison-1.1.jar
  • jetty-util-6.1.26.jar
  • jline-0.9.94.jar
  • jline-2.10.4.jar
  • jodd-core-3.6.3.jar
  • json4s-ast_2.10-3.2.10.jar
  • json4s-core_2.10-3.2.10.jar
  • json4s-jackson_2.10-3.2.10.jar
  • jsr305-1.3.9.jar
  • jtransforms-2.4.0.jar
  • jul-to-slf4j-1.7.10.jar
  • kryo-2.21.jar
  • log4j-1.2.17.jar
  • lz4-1.2.0.jar
  • management-api-3.0.0-b012.jar
  • mesos-0.21.0-shaded-protobuf.jar
  • metrics-core-3.1.0.jar
  • metrics-graphite-3.1.0.jar
  • metrics-json-3.1.0.jar
  • metrics-jvm-3.1.0.jar
  • minlog-1.2.jar
  • netty-3.8.0.Final.jar
  • netty-all-4.0.23.Final.jar
  • objenesis-1.2.jar
  • opencsv-2.3.jar
  • oro-2.0.8.jar
  • paranamer-2.6.jar
  • parquet-column-1.6.0rc3.jar
  • parquet-common-1.6.0rc3.jar
  • parquet-encoding-1.6.0rc3.jar
  • parquet-format-2.2.0-rc1.jar
  • parquet-generator-1.6.0rc3.jar
  • parquet-hadoop-1.6.0rc3.jar
  • parquet-jackson-1.6.0rc3.jar
  • protobuf-java-2.4.1.jar
  • protobuf-java-2.5.0-spark.jar
  • py4j-0.8.2.1.jar
  • pyrolite-2.0.1.jar
  • quasiquotes_2.10-2.0.1.jar
  • reflectasm-1.07-shaded.jar
  • scala-compiler-2.10.4.jar
  • scala-library-2.10.4.jar
  • scala-reflect-2.10.4.jar
  • scalap-2.10.4.jar
  • scalatest_2.10-2.2.1.jar
  • slf4j-api-1.7.10.jar
  • slf4j-log4j12-1.7.10.jar
  • snappy-java-1.1.1.7.jar
  • spark-bagel_2.10-1.4.0-SNAPSHOT.jar
  • spark-catalyst_2.10-1.4.0-SNAPSHOT.jar
  • spark-core_2.10-1.4.0-SNAPSHOT.jar
  • spark-graphx_2.10-1.4.0-SNAPSHOT.jar
  • spark-launcher_2.10-1.4.0-SNAPSHOT.jar
  • spark-mllib_2.10-1.4.0-SNAPSHOT.jar
  • spark-network-common_2.10-1.4.0-SNAPSHOT.jar
  • spark-network-shuffle_2.10-1.4.0-SNAPSHOT.jar
  • spark-repl_2.10-1.4.0-SNAPSHOT.jar
  • spark-sql_2.10-1.4.0-SNAPSHOT.jar
  • spark-streaming_2.10-1.4.0-SNAPSHOT.jar
  • spark-unsafe_2.10-1.4.0-SNAPSHOT.jar
  • spire-macros_2.10-0.7.4.jar
  • spire_2.10-0.7.4.jar
  • stax-api-1.0.1.jar
  • stream-2.7.0.jar
  • tachyon-0.6.4.jar
  • tachyon-client-0.6.4.jar
  • uncommons-maths-1.2.2a.jar
  • unused-1.0.0.jar
  • xmlenc-0.52.jar
  • xz-1.0.jar
  • zookeeper-3.4.5.jar

[tgravescs]

jenkins, test this please

[SparkQA]

Test build #31419 has finished for PR 4688 at commit 36eb8a9.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

[harishreedharan]

Tested the latest update, still working fine

[tgravescs]

+1, looks good. Thanks for testing the latest, I was about to ask that. I ran a few things with the patch but didn't configure things to expire.

[harishreedharan]

This was reverted because it broke Hadoop-1. I am moving the ExecutorDelegationTokenRenewer to the yarn module and using reflection to use it the CoarseGrainedExecutorBackend class.