From 0a1fb5ecc43567b6d77d5d938f9b38ea7ed3fdf0 Mon Sep 17 00:00:00 2001 From: shahid Date: Thu, 14 Nov 2019 23:57:33 +0530 Subject: [PATCH 1/3] config add --- docs/configuration.md | 55 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 55 insertions(+) diff --git a/docs/configuration.md b/docs/configuration.md index 7fdcf4a958d9f..e47c028bd6e83 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -1857,6 +1857,61 @@ Apart from these, the following properties are also available, and may be useful driver using more memory. + + spark.scheduler.listenerbus.eventqueue.shared.capacity + spark.scheduler.listenerbus.eventqueue.capacity + + Capacity for shared event queue in Spark listener bus, must be greater than 0. + Unless otherwise specified it uses spark.scheduler.listenerbus.eventqueue.capacity. + shared event queue hold events for external listener that register to listener bus. + Consider increasing value if listener events corresponding to shared queue are dropped. + Increasing this value may result in the driver using more memory. + + + + spark.scheduler.listenerbus.eventqueue.appStatus.capacity + spark.scheduler.listenerbus.eventqueue.capacity + + Capacity for appStatus event queue in Spark listener bus, must be greater than 0. + Unless otherwise specified it uses spark.scheduler.listenerbus.eventqueue.capacity. + appStatus event queue hold events for internal application status listeners. Consider increasing + value if listener events corresponding to appStatus queue are dropped. + Increasing this value may result in the driver using more memory. + + + + spark.scheduler.listenerbus.eventqueue.executorManagement.capacity + spark.scheduler.listenerbus.eventqueue.capacity + + Capacity for executorManagement event queue in Spark listener bus, must be greater than 0. + Unless otherwise specified it uses spark.scheduler.listenerbus.eventqueue.capacity. + This queue hold events for internal executor management listeners. + Consider increasing value if listener events corresponding to executorManagement queue are dropped. + Increasing this value may result in the driver using more memory. + + + + spark.scheduler.listenerbus.eventqueue.eventLog.capacity + spark.scheduler.listenerbus.eventqueue.capacity + + Capacity for eventLog queue in Spark listener bus, must be greater than 0. + Unless otherwise specified it uses spark.scheduler.listenerbus.eventqueue.capacity. + This queue hold events for Event logging listeners that write events to eventLogs. + Consider increasing value if listener events corresponding to eventLog queue are dropped. + Increasing this value may result in the driver using more memory. + + + + spark.scheduler.listenerbus.eventqueue.streams.capacity + spark.scheduler.listenerbus.eventqueue.capacity + + Capacity for streams queue in Spark listener bus, must be greater than 0. + Unless otherwise specified it uses spark.scheduler.listenerbus.eventqueue.capacity. + This queue hold events for internal streaming listener. Consider increasing value if listener + events corresponding to streams queue are dropped. Increasing this value may result in the + driver using more memory. + + spark.scheduler.blacklist.unschedulableTaskSetTimeout 120s From acead31b414524dd66d8bc1e9e44631922eab67d Mon Sep 17 00:00:00 2001 From: shahid Date: Fri, 15 Nov 2019 03:28:04 +0530 Subject: [PATCH 2/3] address comments --- docs/configuration.md | 38 ++++++++++++++------------------------ 1 file changed, 14 insertions(+), 24 deletions(-) diff --git a/docs/configuration.md b/docs/configuration.md index e47c028bd6e83..a6e1901e037ec 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -1861,21 +1861,17 @@ Apart from these, the following properties are also available, and may be useful spark.scheduler.listenerbus.eventqueue.shared.capacity spark.scheduler.listenerbus.eventqueue.capacity - Capacity for shared event queue in Spark listener bus, must be greater than 0. - Unless otherwise specified it uses spark.scheduler.listenerbus.eventqueue.capacity. - shared event queue hold events for external listener that register to listener bus. - Consider increasing value if listener events corresponding to shared queue are dropped. - Increasing this value may result in the driver using more memory. + Capacity for shared event queue in Spark listener bus, which hold events for external listener(s) + that register to the listener bus. Consider increasing value, if the listener events corresponding + to shared queue are dropped. Increasing this value may result in the driver using more memory. spark.scheduler.listenerbus.eventqueue.appStatus.capacity spark.scheduler.listenerbus.eventqueue.capacity - Capacity for appStatus event queue in Spark listener bus, must be greater than 0. - Unless otherwise specified it uses spark.scheduler.listenerbus.eventqueue.capacity. - appStatus event queue hold events for internal application status listeners. Consider increasing - value if listener events corresponding to appStatus queue are dropped. + Capacity for appStatus event queue, which hold events for internal application status listeners. + Consider increasing value, if the listener events corresponding to appStatus queue are dropped. Increasing this value may result in the driver using more memory. @@ -1883,33 +1879,27 @@ Apart from these, the following properties are also available, and may be useful spark.scheduler.listenerbus.eventqueue.executorManagement.capacity spark.scheduler.listenerbus.eventqueue.capacity - Capacity for executorManagement event queue in Spark listener bus, must be greater than 0. - Unless otherwise specified it uses spark.scheduler.listenerbus.eventqueue.capacity. - This queue hold events for internal executor management listeners. - Consider increasing value if listener events corresponding to executorManagement queue are dropped. - Increasing this value may result in the driver using more memory. + Capacity for executorManagement event queue in Spark listener bus, which hold events for internal + executor management listeners. Consider increasing value if the listener events corresponding to + executorManagement queue are dropped. Increasing this value may result in the driver using more memory. spark.scheduler.listenerbus.eventqueue.eventLog.capacity spark.scheduler.listenerbus.eventqueue.capacity - Capacity for eventLog queue in Spark listener bus, must be greater than 0. - Unless otherwise specified it uses spark.scheduler.listenerbus.eventqueue.capacity. - This queue hold events for Event logging listeners that write events to eventLogs. - Consider increasing value if listener events corresponding to eventLog queue are dropped. - Increasing this value may result in the driver using more memory. + Capacity for eventLog queue in Spark listener bus, which hold events for Event logging listeners + that write events to eventLogs. Consider increasing value if listener events corresponding to eventLog queue + are dropped. Increasing this value may result in the driver using more memory. spark.scheduler.listenerbus.eventqueue.streams.capacity spark.scheduler.listenerbus.eventqueue.capacity - Capacity for streams queue in Spark listener bus, must be greater than 0. - Unless otherwise specified it uses spark.scheduler.listenerbus.eventqueue.capacity. - This queue hold events for internal streaming listener. Consider increasing value if listener - events corresponding to streams queue are dropped. Increasing this value may result in the - driver using more memory. + Capacity for streams queue in Spark listener bus, which hold events for internal streaming listener. + Consider increasing value if listener events corresponding to streams queue are dropped. Increasing this + value may result in the driver using more memory. From 76180fac61fb8f4c4fd4e97dfa5382df2df60dad Mon Sep 17 00:00:00 2001 From: shahid Date: Fri, 15 Nov 2019 03:45:34 +0530 Subject: [PATCH 3/3] address comments --- docs/configuration.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/configuration.md b/docs/configuration.md index a6e1901e037ec..bb3ac3cb39996 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -1889,7 +1889,7 @@ Apart from these, the following properties are also available, and may be useful spark.scheduler.listenerbus.eventqueue.capacity Capacity for eventLog queue in Spark listener bus, which hold events for Event logging listeners - that write events to eventLogs. Consider increasing value if listener events corresponding to eventLog queue + that write events to eventLogs. Consider increasing value if the listener events corresponding to eventLog queue are dropped. Increasing this value may result in the driver using more memory. @@ -1898,8 +1898,8 @@ Apart from these, the following properties are also available, and may be useful spark.scheduler.listenerbus.eventqueue.capacity Capacity for streams queue in Spark listener bus, which hold events for internal streaming listener. - Consider increasing value if listener events corresponding to streams queue are dropped. Increasing this - value may result in the driver using more memory. + Consider increasing value if the listener events corresponding to streams queue are dropped. Increasing + this value may result in the driver using more memory.