[WIP][SPARK-31198][CORE] Use graceful decommissioning as part of dynamic scaling#28818

Closed

holdenk wants to merge 10 commits intoapache:masterfrom

holdenk:SPARK-31198-use-graceful-decommissioning-as-part-of-dynamic-scaling-on-top-of-SPARK-1197

Commits on Jul 23, 2020

Shutdown executor once we are done decommissioning
holdenk
committed
Make Spark's dynamic allocation use decommissioning
holdenk
committed
Track the decommissioning executors in the core dynamic scheduler so we don't scale down too low, update the streaming ExecutorAllocationManager to also delegate to decommission
holdenk
committed
Fix our exiting and cleanup thread for better debugging next time. Cleanup the locks we use in decommissioning and clarify some more bits.
holdenk
committed
Verify executors decommissioned, then killed by external external cluster manager are re-launched
holdenk
committed
Verify some additional calls are not occuring in the executor allocation manager suite.
holdenk
committed
Dont' close the watcher until the end of the test
holdenk
committed
Use decommissionExecutors and set adjustTargetNumExecutors to false so that we can match the pattern for killExecutor/killExecutors
holdenk
committed
bump numparts up to 6
holdenk
committed
Revert "bump numparts up to 6"
holdenk
committed