Skip to content

[SPARK-6469] Improving documentation on YARN local directories usage #5165

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
preaudc wants to merge 12 commits into from
Closed

[SPARK-6469] Improving documentation on YARN local directories usage #5165

Changes from 10 commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
8ea871f
Ensure that files are fetched atomically
Oct 20, 2014
7456a33
Merge remote-tracking branch 'upstream/master'
Oct 23, 2014
c6a5590
Revert commit 8ea871f8130b2490f1bad7374a819bf56f0ccbbd
Nov 3, 2014
54419ae
Merge remote-tracking branch 'upstream/master'
Nov 3, 2014
9ba89ca
Ensure that files are fetched atomically
Nov 3, 2014
a49a2ce
Merge remote-tracking branch 'upstream/master'
Mar 23, 2015
608dbfa
Merge remote-tracking branch 'upstream/master'
Mar 24, 2015
876ae5e
Clarify the local directories usage in YARN
Mar 24, 2015
436fb7d
Revert "Clarify the local directories usage in YARN"
Mar 24, 2015
eaaf519
Clarify the local directories usage in YARN
Mar 24, 2015
4fa8ec2
Merge remote-tracking branch 'upstream/master' into yarn-doc-local-dirs
Mar 24, 2015
6912b90
Fix some formatting issues.
Mar 24, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/running-on-yarn.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -274,6 +274,6 @@ If you need a reference to the proper location to put log files in the YARN so t
# Important notes

- Whether core requests are honored in scheduling decisions depends on which scheduler is in use and how it is configured.
- The local directories used by Spark executors will be the local directories configured for YARN (Hadoop YARN config `yarn.nodemanager.local-dirs`). If the user specifies `spark.local.dir`, it will be ignored.
- In yarn-cluster mode, the local directories used by the Spark executors and the Spark driver will be the local directories configured for YARN (Hadoop YARN config `yarn.nodemanager.local-dirs`). If the user specifies `spark.local.dir`, it will be ignored. In yarn-client mode, the Spark executors will use the local directories configured for YARN while the Spark driver will use those defined in `spark.local.dir` (this is because the Spark driver does not run on the YARN cluster in yarn-client mode, only the Spark executors do).

@srowen srowen Mar 24, 2015

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nits: you can back-tick-quote code-like values like yarn-cluster. The last parenthetical can just be a sentence. Yeah it's normal to show all the commits you made to your branch. They will be squashed on merge. You can also squash and force-push your branch if desired, though it's not that important.

- The `--files` and `--archives` options support specifying file names with the # similar to Hadoop. For example you can specify: `--files localtest.txt#appSees.txt` and this will upload the file you have locally named localtest.txt into HDFS but this will be linked to by the name `appSees.txt`, and your application should use the name as `appSees.txt` to reference it when running on YARN.
- The `--jars` option allows the `SparkContext.addJar` function to work if you are using it with local files and running in `yarn-cluster` mode. It does not need to be used if you are using it with HDFS, HTTP, HTTPS, or FTP files.