Skip to content

Commit 9fb86b8

Browse files
zsxwingJoshRosen
authored andcommitted
[SPARK-4931][Yarn][Docs] Fix the format of running-on-yarn.md
Currently, the format about log4j in running-on-yarn.md is a bit messy. ![running-on-yarn](https://cloud.githubusercontent.com/assets/1000778/5535248/204c4b64-8ab4-11e4-83c3-b4722ea0ad9d.png) Author: zsxwing <[email protected]> Closes #3774 from zsxwing/SPARK-4931 and squashes the following commits: 4a5f853 [zsxwing] Fix the format of running-on-yarn.md (cherry picked from commit 2d215ae) Signed-off-by: Josh Rosen <[email protected]>
1 parent f86fe08 commit 9fb86b8

File tree

1 file changed

+5
-5
lines changed

1 file changed

+5
-5
lines changed

docs/running-on-yarn.md

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -201,18 +201,18 @@ settings and a restart of all node managers. Thus, this is not applicable to hos
201201

202202
To use a custom log4j configuration for the application master or executors, there are two options:
203203

204-
- upload a custom log4j.properties using spark-submit, by adding it to the "--files" list of files
204+
- upload a custom `log4j.properties` using `spark-submit`, by adding it to the `--files` list of files
205205
to be uploaded with the application.
206-
- add "-Dlog4j.configuration=<location of configuration file>" to "spark.driver.extraJavaOptions"
207-
(for the driver) or "spark.executor.extraJavaOptions" (for executors). Note that if using a file,
208-
the "file:" protocol should be explicitly provided, and the file needs to exist locally on all
206+
- add `-Dlog4j.configuration=<location of configuration file>` to `spark.driver.extraJavaOptions`
207+
(for the driver) or `spark.executor.extraJavaOptions` (for executors). Note that if using a file,
208+
the `file:` protocol should be explicitly provided, and the file needs to exist locally on all
209209
the nodes.
210210

211211
Note that for the first option, both executors and the application master will share the same
212212
log4j configuration, which may cause issues when they run on the same node (e.g. trying to write
213213
to the same log file).
214214

215-
If you need a reference to the proper location to put log files in the YARN so that YARN can properly display and aggregate them, use "${spark.yarn.app.container.log.dir}" in your log4j.properties. For example, log4j.appender.file_appender.File=${spark.yarn.app.container.log.dir}/spark.log. For streaming application, configuring RollingFileAppender and setting file location to YARN's log directory will avoid disk overflow caused by large log file, and logs can be accessed using YARN's log utility.
215+
If you need a reference to the proper location to put log files in the YARN so that YARN can properly display and aggregate them, use `spark.yarn.app.container.log.dir` in your log4j.properties. For example, `log4j.appender.file_appender.File=${spark.yarn.app.container.log.dir}/spark.log`. For streaming application, configuring `RollingFileAppender` and setting file location to YARN's log directory will avoid disk overflow caused by large log file, and logs can be accessed using YARN's log utility.
216216

217217
# Important notes
218218

0 commit comments

Comments
 (0)