Skip to content

Commit c871e2d

Browse files
moutaiAndrew Or
authored andcommitted
Add a note for context termination for History server on Yarn
The history server on Yarn only shows completed jobs. This adds a note concerning the needed explicit context termination at the end of a spark job which is a best practice anyway. Related to SPARK-2972 and SPARK-3458 Author: moussa taifi <[email protected]> Closes #4721 from moutai/add-history-server-note-for-closing-the-spark-context and squashes the following commits: 9f5b6c3 [moussa taifi] Fix upper case typo for YARN 3ad3db4 [moussa taifi] Add context termination for History server on Yarn
1 parent 3fb53c0 commit c871e2d

File tree

1 file changed

+2
-0
lines changed

1 file changed

+2
-0
lines changed

docs/monitoring.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -173,6 +173,8 @@ follows:
173173
Note that in all of these UIs, the tables are sortable by clicking their headers,
174174
making it easy to identify slow tasks, data skew, etc.
175175

176+
Note that the history server only displays completed Spark jobs. One way to signal the completion of a Spark job is to stop the Spark Context explicitly (`sc.stop()`), or in Python using the `with SparkContext() as sc:` to handle the Spark Context setup and tear down, and still show the job history on the UI.
177+
176178
# Metrics
177179

178180
Spark has a configurable metrics system based on the

0 commit comments

Comments
 (0)