[SPARK-1753 / 1773 / 1814] Update outdated docs for spark-submit, YARN, standalone etc.

docs/hadoop-third-party-distributions.md

@@ -9,12 +9,11 @@ with these distributions:
 
 # Compile-time Hadoop Version
 
-When compiling Spark, you'll need to 
-[set the SPARK_HADOOP_VERSION flag](index.html#a-note-about-hadoop-versions):
+When compiling Spark, you'll need to specify the Hadoop version by [defining the hadoop.version property](building-with-maven.html):
 
-    SPARK_HADOOP_VERSION=1.0.4 sbt/sbt assembly
+    mvn -Dhadoop.version=1.0.4 -DskipTests clean package
 
-The table below lists the corresponding `SPARK_HADOOP_VERSION` code for each CDH/HDP release. Note that
+The table below lists the corresponding `hadoop.version` code for each CDH/HDP release. Note that
 some Hadoop releases are binary compatible across client versions. This means the pre-built Spark
 distribution may "just work" without you needing to compile. That said, we recommend compiling with 
 the _exact_ Hadoop version you are running to avoid any compatibility errors.
@@ -46,6 +45,10 @@ the _exact_ Hadoop version you are running to avoid any compatibility errors.
   </tr>
 </table>
 
+In SBT, the equivalent can be achieved by setting the SPARK_HADOOP_VERSION flag:
+
+    SPARK_HADOOP_VERSION=1.0.4 sbt/sbt assembly
+
 # Linking Applications to the Hadoop Version
 
 In addition to compiling Spark itself against the right version, you need to add a Maven dependency on that

docs/running-on-yarn.md

@@ -43,18 +43,19 @@ Unlike in Spark standalone and Mesos mode, in which the master's address is spec
 
 To launch a Spark application in yarn-cluster mode:
 
-    ./bin/spark-submit --class path.to.your.Class --master yarn-cluster [options] <app jar> [app options]
+    ./bin/spark-submit --class path.to.your.Class --master yarn-cluster --deploy-mode cluster [options] <app jar> [app options]
 
 For example:
 
     $ ./bin/spark-submit --class org.apache.spark.examples.SparkPi \
         --master yarn-cluster \
+        --deploy-mode cluster \
         --num-executors 3 \
         --driver-memory 4g \
         --executor-memory 2g \
         --executor-cores 1
         examples/target/scala-{{site.SCALA_BINARY_VERSION}}/spark-examples-assembly-{{site.SPARK_VERSION}}.jar \
-        yarn-cluster 5
+        10
 
 The above starts a YARN client program which starts the default Application Master. Then SparkPi will be run as a child thread of Application Master. The client will periodically poll the Application Master for status updates and display them in the console. The client will exit once your application has finished running.  Refer to the "Viewing Logs" section below for how to see driver and executor logs.
 
@@ -68,11 +69,12 @@ In yarn-cluster mode, the driver runs on a different machine than the client, so
 
     $ ./bin/spark-submit --class my.main.Class \
         --master yarn-cluster \
+        --deploy-mode cluster \
         --jars my-other-jar.jar,my-other-other-jar.jar
         my-main-jar.jar
-        yarn-cluster 5
+        [app arguments]
 
-# Viewing logs
+# Debugging your Application
 
 In YARN terminology, executors and application masters run inside "containers". YARN has two modes for handling container logs after an application has completed. If log aggregation is turned on (with the yarn.log-aggregation-enable config), container logs are copied to HDFS and deleted on the local machine. These logs can be viewed from anywhere on the cluster with the "yarn logs" command.
 
@@ -82,6 +84,12 @@ will print out the contents of all log files from all containers from the given
 
 When log aggregation isn't turned on, logs are retained locally on each machine under YARN_APP_LOGS_DIR, which is usually configured to /tmp/logs or $HADOOP_HOME/logs/userlogs depending on the Hadoop version and installation. Viewing logs for a container requires going to the host that contains them and looking in this directory.  Subdirectories organize log files by application ID and container ID.
 
+To review per container launch environment, increase yarn.nodemanager.delete.debug-delay-sec to a
+large value (e.g. 36000), and then access the application cache through yarn.nodemanager.local-dirs
+on the nodes on which containers are launched. This directory contains the launch script, jars, and
+all environment variables used for launching each container. This process is useful for debugging
+classpath problems in particular.
+
 # Important notes
 
 - Before Hadoop 2.2, YARN does not support cores in container resource requests. Thus, when running against an earlier version, the numbers of cores given via command line arguments cannot be passed to YARN.  Whether core requests are honored in scheduling decisions depends on which scheduler is in use and how it is configured.