Skip to content

[SPARK-29902][Doc][Minor]Add listener event queue capacity configuration to documentation #26529

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
shahidki31 wants to merge 3 commits into from
Closed

[SPARK-29902][Doc][Minor]Add listener event queue capacity configuration to documentation #26529

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
0a1fb5e
config add
shahidki31 Nov 14, 2019
acead31
address comments
shahidki31 Nov 14, 2019
76180fa
address comments
shahidki31 Nov 14, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 45 additions & 0 deletions docs/configuration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1857,6 +1857,51 @@ Apart from these, the following properties are also available, and may be useful
driver using more memory.
</td>
</tr>
<tr>
<td><code>spark.scheduler.listenerbus.eventqueue.shared.capacity</code></td>
<td><code>spark.scheduler.listenerbus.eventqueue.capacity</code></td>
<td>
Capacity for shared event queue in Spark listener bus, which hold events for external listener(s)
that register to the listener bus. Consider increasing value, if the listener events corresponding
to shared queue are dropped. Increasing this value may result in the driver using more memory.
</td>
</tr>
<tr>
<td><code>spark.scheduler.listenerbus.eventqueue.appStatus.capacity</code></td>
<td><code>spark.scheduler.listenerbus.eventqueue.capacity</code></td>
<td>
Capacity for appStatus event queue, which hold events for internal application status listeners.
Consider increasing value, if the listener events corresponding to appStatus queue are dropped.
Increasing this value may result in the driver using more memory.
</td>
</tr>
<tr>
<td><code>spark.scheduler.listenerbus.eventqueue.executorManagement.capacity</code></td>
<td><code>spark.scheduler.listenerbus.eventqueue.capacity</code></td>
<td>
Capacity for executorManagement event queue in Spark listener bus, which hold events for internal
executor management listeners. Consider increasing value if the listener events corresponding to
executorManagement queue are dropped. Increasing this value may result in the driver using more memory.
</td>
</tr>
<tr>
<td><code>spark.scheduler.listenerbus.eventqueue.eventLog.capacity</code></td>
<td><code>spark.scheduler.listenerbus.eventqueue.capacity</code></td>
<td>
Capacity for eventLog queue in Spark listener bus, which hold events for Event logging listeners
that write events to eventLogs. Consider increasing value if the listener events corresponding to eventLog queue
are dropped. Increasing this value may result in the driver using more memory.
</td>
</tr>
<tr>
<td><code>spark.scheduler.listenerbus.eventqueue.streams.capacity</code></td>
<td><code>spark.scheduler.listenerbus.eventqueue.capacity</code></td>
<td>
Capacity for streams queue in Spark listener bus, which hold events for internal streaming listener.
Consider increasing value if the listener events corresponding to streams queue are dropped. Increasing
this value may result in the driver using more memory.
</td>
</tr>
<tr>
<td><code>spark.scheduler.blacklist.unschedulableTaskSetTimeout</code></td>
<td>120s</td>
Expand Down