Skip to content

Commit 4968c5c

Browse files
mgummeltmgummelt
committed
Merge pull request apache#35 from mesosphere/edits-to-job-scheduling-doc
Infinitesimal changes
2 parents 368db12 + 949421d commit 4968c5c

File tree

1 file changed

+106
-58
lines changed

1 file changed

+106
-58
lines changed

‎docs/job-scheduling.md‎

Lines changed: 106 additions & 58 deletions
Original file line numberDiff line numberDiff line change
@@ -1,78 +1,126 @@
1-
This page is a simple overview of material described in greater detail
2-
in the Apache Spark documentation [here][1] and [here][2].
1+
This document is a simple overview of material described in greater
2+
detail in the Apache Spark documentation [here][1] and [here][2].
33

44
# Modes
5-
5+
66
Spark on Mesos supports two "modes" of operation: coarse-grained mode
7-
and fine-grained mode. Coarse-grained mode provides lower latency,
8-
whereas fine-grained mode provides higher utilization. More info
7+
and fine-grained mode. Coarse-grained mode provides lower latency,
8+
whereas fine-grained mode provides higher utilization. More info
99
[here][2].
1010

11-
## Coarse-grained mode
12-
11+
# Coarse-grained mode
12+
1313
"Coarse-grained" mode is so-called because each Spark **executor** is
14-
represented by a single Mesos task. As a result, executors have a
14+
represented by a single Mesos task. As a result, executors have a
1515
constant size throughout their lifetime.
1616

17-
* **Executor memory**: `spark.executor.memory`
18-
* **Executor CPUs**: `spark.executor.cores`, or all the cores in the offer.
19-
* **Number of Executors**: `spark.cores.max` / `spark.executor.cores`.
20-
Executors are brought up until `spark.cores.max` is reached.
21-
Executors survive for duration of the job.
22-
* **Executors per agent**: Multiple
17+
* **Executor memory**: `spark.executor.memory`
18+
* **Executor CPUs**: `spark.executor.cores`, or all the cores in the
19+
offer.
20+
* **Number of Executors**: `spark.cores.max` /
21+
`spark.executor.cores`. Executors are brought up until
22+
`spark.cores.max` is reached. Executors survive for duration of the
23+
job.
24+
* **Executors per agent**: Multiple
2325

2426
Notes:
2527

26-
- We highly recommend you set `spark.cores.max`. If you do not,
27-
your Spark job may consume all available resources in your cluster,
28-
resulting in unhappy peers.
29-
30-
## Fine-grained mode
28+
* We highly recommend you set `spark.cores.max`. If you do not, your
29+
Spark job may consume all available resources in your cluster,
30+
resulting in unhappy peers.
3131

32+
# Fine-grained mode
33+
3234
In "fine-grained" mode, each Spark **task** is represented by a single
33-
Mesos task. When a Spark task finishes, the resources represented by
34-
a its Mesos task are relinquished. This enables finer-grained
35-
resource allocation, at the cost of task startup latency.
35+
Mesos task. When a Spark task finishes, the resources represented by
36+
its Mesos task are relinquished. Fine-grained mode enables
37+
finer-grained resource allocation at the cost of task startup latency.
3638

37-
* **Executor memory**: `spark.executor.memory`
38-
* **Executor CPUs**: Increases and decreases as tasks start and terminate.
39-
* **Number of Executors**: Increases and decreases as tasks start and terminate.
40-
* **Executors per agent**: At most 1
39+
* **Executor memory**: `spark.executor.memory`
40+
* **Executor CPUs**: Increases and decreases as tasks start and
41+
terminate.
42+
* **Number of Executors**: Increases and decreases as tasks start
43+
and terminate.
44+
* **Executors per agent**: At most 1
4145

4246
# Properties
4347

4448
The following is a description of the most common Spark on Mesos
45-
scheduling properties. For a full list, see the
46-
[Spark configuration page][1] and the
47-
[Spark on Mesos configuration page][2].
49+
scheduling properties. For a full list, see the [Spark configuration
50+
page][1] and the [Spark on Mesos configuration page][2].
4851

49-
<table>
50-
<tr>
51-
<th>property</th>
52-
<th>default</th>
53-
<th>description</th>
54-
<tr>
55-
<td>spark.mesos.coarse</td>
56-
<td>true</td>
57-
<td>Described above</td>
58-
</tr>
59-
<tr>
60-
<td>spark.executor.memory</td>
61-
<td>1g</td>
62-
<td>Executor memory allocation</td>
63-
</tr>
64-
<tr>
65-
<td>spark.executor.cores</td>
66-
<td>All available cores in the offer</td>
67-
<td>Coarse-grained mode only. DCOS Spark >= 1.6.1. Executor CPU allocation.</td>
68-
</tr>
69-
<tr>
70-
<td>spark.cores.max</td>
71-
<td>unlimited</td>
72-
<td>Maximum total number of cores to allocate.</td>
73-
<tr>
74-
</tr>
75-
</table>
52+
<table class="table">
53+
<tr>
54+
<th>
55+
property
56+
</th>
57+
58+
<th>
59+
default
60+
</th>
61+
62+
<th>
63+
description
64+
</th>
65+
66+
<tr>
67+
<td>
68+
spark.mesos.coarse
69+
</td>
70+
71+
<td>
72+
true
73+
</td>
74+
75+
<td>
76+
Described above.
77+
</td>
78+
</tr>
79+
80+
<tr>
81+
<td>
82+
spark.executor.memory
83+
</td>
84+
85+
<td>
86+
1g
87+
</td>
88+
89+
<td>
90+
Executor memory allocation.
91+
</td>
92+
</tr>
93+
94+
<tr>
95+
<td>
96+
spark.executor.cores
97+
</td>
98+
99+
<td>
100+
All available cores in the offer
101+
</td>
102+
103+
<td>
104+
Coarse-grained mode only. DCOS Spark >= 1.6.1. Executor CPU
105+
allocation.
106+
</td>
107+
</tr>
108+
109+
<tr>
110+
<td>
111+
spark.cores.max
112+
</td>
113+
114+
<td>
115+
unlimited
116+
</td>
117+
118+
<td>
119+
Maximum total number of cores to allocate.
120+
</td>
121+
122+
<tr>
123+
</tr></table>
76124

77-
[1]: http://spark.apache.org/docs/latest/configuration.html
78-
[2]: http://spark.apache.org/docs/latest/running-on-mesos.html
125+
[1]: http://spark.apache.org/docs/latest/configuration.html
126+
[2]: http://spark.apache.org/docs/latest/running-on-mesos.html

0 commit comments

Comments
 (0)