[SPARK-9570][Docs][YARN]Consistent recommendation for submitting spark apps to YARN, -master yarn --deploy-mode x vs -master yarn-x' #8071
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -48,6 +48,44 @@ Some of the commonly used options are: | |
| * `application-jar`: Path to a bundled jar including your application and all dependencies. The URL must be globally visible inside of your cluster, for instance, an `hdfs://` path or a `file://` path that is present on all nodes. | ||
| * `application-arguments`: Arguments passed to the main method of your main class, if any | ||
|
|
||
| Alternatively, for submitting on yarn, | ||
|
|
||
| {% highlight bash %} | ||
| ./bin/spark-submit \ | ||
| --class <main-class> | ||
| --master <yarn-deploy-mode> | ||
| --conf <key>=<value> \ | ||
| ... # other options | ||
| <application-jar> \ | ||
| [application-arguments] | ||
| {% endhighlight %} | ||
|
|
||
| * `--master`: The --master parameter is either `yarn-client` or `yarn-cluster`. Defaults to `yarn-client` | ||
|
Contributor
There was a problem hiding this comment. actually I would rather see us use the --deploy-mode option to choose cluster vs client and --master yarn. Unless something has changed we kept yarn-cluster and yarn-client around just for backwards compatibility.
Author
There was a problem hiding this comment. @srowen and @tgravescs, users still have the confusion regarding the "recommended/preferred" method of submission.
Member
There was a problem hiding this comment. I don't think it makes sense to promote both ways. I also think both have to be retained for compatibility. I'm still not clear what people think the usual way to do it is; I've always seen and used
Contributor
There was a problem hiding this comment. I agree we should have a preferred method and update docs to match. I think the existing docs just never were fully updated. There wouldn't be a reason to have the --deploy-mode if we used yarn-client/yarn-cluster. There was discussion about it back then. cc @sryza
Author
There was a problem hiding this comment. Will grep the docs to see the more popular approach to submission amongst the two. |
||
|
|
||
| # Master URLs | ||
|
|
||
| The master URL passed to Spark can be in one of the following formats: | ||
|
|
||
| <table class="table"> | ||
| <tr><th>Master URL</th><th>Meaning</th></tr> | ||
| <tr><td> local </td><td> Run Spark locally with one worker thread (i.e. no parallelism at all). </td></tr> | ||
| <tr><td> local[K] </td><td> Run Spark locally with K worker threads (ideally, set this to the number of cores on your machine). </td></tr> | ||
| <tr><td> local[*] </td><td> Run Spark locally with as many worker threads as logical cores on your machine.</td></tr> | ||
| <tr><td> spark://HOST:PORT </td><td> Connect to the given <a href="spark-standalone.html">Spark standalone | ||
| cluster</a> master. The port must be whichever one your master is configured to use, which is 7077 by default. | ||
| </td></tr> | ||
| <tr><td> mesos://HOST:PORT </td><td> Connect to the given <a href="running-on-mesos.html">Mesos</a> cluster. | ||
| The port must be whichever one your is configured to use, which is 5050 by default. | ||
| Or, for a Mesos cluster using ZooKeeper, use <code>mesos://zk://...</code>. | ||
| </td></tr> | ||
| <tr><td> yarn-client </td><td> Connect to a <a href="running-on-yarn.html"> YARN </a> cluster in | ||
| client mode. The cluster location will be found based on the HADOOP_CONF_DIR or YARN_CONF_DIR variable. | ||
| </td></tr> | ||
| <tr><td> yarn-cluster </td><td> Connect to a <a href="running-on-yarn.html"> YARN </a> cluster in | ||
| cluster mode. The cluster location will be found based on the HADOOP_CONF_DIR or YARN_CONF_DIR variable. | ||
| </td></tr> | ||
| </table> | ||
|
|
||
| <b>†</b> A common deployment strategy is to submit your application from a gateway machine | ||
| that is | ||
| physically co-located with your worker machines (e.g. Master node in a standalone EC2 cluster). | ||
|
|
@@ -99,7 +137,7 @@ run it with `--help`. Here are a few examples of common options: | |
| /path/to/examples.jar \ | ||
| 1000 | ||
|
|
||
| # Run on a YARN cluster without --deploy mode | ||
| export HADOOP_CONF_DIR=XXX | ||
| ./bin/spark-submit \ | ||
| --class org.apache.spark.examples.SparkPi \ | ||
|
|
@@ -116,31 +154,6 @@ export HADOOP_CONF_DIR=XXX | |
| 1000 | ||
| {% endhighlight %} | ||
|
|
||
| # Loading Configuration from a File | ||
|
|
||
| The `spark-submit` script can load default [Spark configuration values](configuration.html) from a | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yarn -> YARN
I don't know if this helps as it's just presenting both ways to set this, and not clarifying which is preferred. It also duplicates the rest of the example.