Skip to content

Commit 7911de9

Browse files
belieferHyukjinKwon
authored andcommitted
[SPARK-30887][CORE][DOC] Add version information to the configuration of Deploy
### What changes were proposed in this pull request? 1.Add version information to the configuration of `Deploy`. 2.Update the docs of `Deploy`. I sorted out some information show below. Item name | Since version | JIRA ID | Commit ID | Note -- | -- | -- | -- | -- spark.deploy.recoveryMode | 0.8.1 | None | d66c01f#diff-29dffdccd5a7f4c8b496c293e87c8668 | spark.deploy.recoveryMode.factory | 1.2.0 | SPARK-1830 | deefd9d#diff-29dffdccd5a7f4c8b496c293e87c8668 | This configuration appears in branch-1.3, but the version number in the pom.xml file corresponding to the commit is 1.2.0-SNAPSHOT spark.deploy.recoveryDirectory | 0.8.1 | None | d66c01f#diff-29dffdccd5a7f4c8b496c293e87c8668 | spark.deploy.zookeeper.url | 0.8.1 | None | d66c01f#diff-4457313ca662a1cd60197122d924585c | spark.deploy.zookeeper.dir | 0.8.1 | None | d66c01f#diff-a84228cb45c7d5bd93305a1f5bf720b6 | spark.deploy.retainedApplications | 0.8.0 | None | 46eecd1#diff-29dffdccd5a7f4c8b496c293e87c8668 | spark.deploy.retainedDrivers | 1.1.0 | None | 7446f5f#diff-29dffdccd5a7f4c8b496c293e87c8668 | spark.dead.worker.persistence | 0.8.0 | None | 46eecd1#diff-29dffdccd5a7f4c8b496c293e87c8668 | spark.deploy.maxExecutorRetries | 1.6.3 | SPARK-16956 | ace458f#diff-29dffdccd5a7f4c8b496c293e87c8668 | spark.deploy.spreadOut | 0.6.1 | None | bb2b9ff#diff-0e7ae91819fc8f7b47b0f97be7116325 | spark.deploy.defaultCores | 0.9.0 | None | d8bcc8e#diff-29dffdccd5a7f4c8b496c293e87c8668 | ### Why are the changes needed? Supplemental configuration version information. ### Does this PR introduce any user-facing change? No ### How was this patch tested? Exists UT Closes #27668 from beliefer/add-version-to-deploy-config. Authored-by: beliefer <[email protected]> Signed-off-by: HyukjinKwon <[email protected]>
1 parent cc8d356 commit 7911de9

File tree

2 files changed

+15
-1
lines changed

2 files changed

+15
-1
lines changed

core/src/main/scala/org/apache/spark/internal/config/Deploy.scala

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,48 +19,59 @@ package org.apache.spark.internal.config
1919

2020
private[spark] object Deploy {
2121
val RECOVERY_MODE = ConfigBuilder("spark.deploy.recoveryMode")
22+
.version("0.8.1")
2223
.stringConf
2324
.createWithDefault("NONE")
2425

2526
val RECOVERY_MODE_FACTORY = ConfigBuilder("spark.deploy.recoveryMode.factory")
27+
.version("1.2.0")
2628
.stringConf
2729
.createWithDefault("")
2830

2931
val RECOVERY_DIRECTORY = ConfigBuilder("spark.deploy.recoveryDirectory")
32+
.version("0.8.1")
3033
.stringConf
3134
.createWithDefault("")
3235

3336
val ZOOKEEPER_URL = ConfigBuilder("spark.deploy.zookeeper.url")
3437
.doc(s"When `${RECOVERY_MODE.key}` is set to ZOOKEEPER, this " +
3538
"configuration is used to set the zookeeper URL to connect to.")
39+
.version("0.8.1")
3640
.stringConf
3741
.createOptional
3842

3943
val ZOOKEEPER_DIRECTORY = ConfigBuilder("spark.deploy.zookeeper.dir")
44+
.version("0.8.1")
4045
.stringConf
4146
.createOptional
4247

4348
val RETAINED_APPLICATIONS = ConfigBuilder("spark.deploy.retainedApplications")
49+
.version("0.8.0")
4450
.intConf
4551
.createWithDefault(200)
4652

4753
val RETAINED_DRIVERS = ConfigBuilder("spark.deploy.retainedDrivers")
54+
.version("1.1.0")
4855
.intConf
4956
.createWithDefault(200)
5057

5158
val REAPER_ITERATIONS = ConfigBuilder("spark.dead.worker.persistence")
59+
.version("0.8.0")
5260
.intConf
5361
.createWithDefault(15)
5462

5563
val MAX_EXECUTOR_RETRIES = ConfigBuilder("spark.deploy.maxExecutorRetries")
64+
.version("1.6.3")
5665
.intConf
5766
.createWithDefault(10)
5867

5968
val SPREAD_OUT_APPS = ConfigBuilder("spark.deploy.spreadOut")
69+
.version("0.6.1")
6070
.booleanConf
6171
.createWithDefault(true)
6272

6373
val DEFAULT_CORES = ConfigBuilder("spark.deploy.defaultCores")
74+
.version("0.9.0")
6475
.intConf
6576
.createWithDefault(Int.MaxValue)
6677

docs/configuration.md

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2597,22 +2597,25 @@ Spark subsystems.
25972597
### Deploy
25982598

25992599
<table class="table">
2600-
<tr><th>Property Name</th><th>Default</th><th>Meaning</th></tr>
2600+
<tr><th>Property Name</th><th>Default</th><th>Meaning</th><th>Since Version</th></tr>
26012601
<tr>
26022602
<td><code>spark.deploy.recoveryMode</code></td>
26032603
<td>NONE</td>
26042604
<td>The recovery mode setting to recover submitted Spark jobs with cluster mode when it failed and relaunches.
26052605
This is only applicable for cluster mode when running with Standalone or Mesos.</td>
2606+
<td>0.8.1</td>
26062607
</tr>
26072608
<tr>
26082609
<td><code>spark.deploy.zookeeper.url</code></td>
26092610
<td>None</td>
26102611
<td>When `spark.deploy.recoveryMode` is set to ZOOKEEPER, this configuration is used to set the zookeeper URL to connect to.</td>
2612+
<td>0.8.1</td>
26112613
</tr>
26122614
<tr>
26132615
<td><code>spark.deploy.zookeeper.dir</code></td>
26142616
<td>None</td>
26152617
<td>When `spark.deploy.recoveryMode` is set to ZOOKEEPER, this configuration is used to set the zookeeper directory to store recovery state.</td>
2618+
<td>0.8.1</td>
26162619
</tr>
26172620
</table>
26182621

0 commit comments

Comments
 (0)