Performance tests for the scheduler

[squito]

This code probably doesn't belong in Spark (at least, it doesn't have a nice home there presently), so for now just opening this as a central place to discuss.

Some crude performance tests for the scheduler using apache#8559

In these tests, you setup a mock backend, and run as many simulated jobs as possible in 10 seconds. This lets you compare the performance of the scheduler with different simulated backends. Eg., what happens when there are 1000 nodes? What about when one executor is bad?

I've tried to design things so that the mocking doesn't get in the way of measuring profile, while still allowing enough flexibility. Would definitely appreciate a second pair of eyes, in any case.

So far I've just been running this manually in sbt, using the "-z" option to limit to some comparisons, eg.

build/sbt -Pyarn -Phadoop-2.4 -Dhadoop.version=2.6.0
> project core
> test-only org.apache.spark.scheduler.SchedulerPerformanceSuite -- -z "COMPARE D bad execs"
...
[info] SchedulerPerformanceSuite:
ran 57 iterations in 10.3 s (179 ms per itr)
[info] - COMPARE D bad execs with advanced blacklist (11 seconds, 201 milliseconds)
ran 102 iterations in 10.1 s (98 ms per itr)
[info] - COMPARE D bad execs with simple blacklist (10 seconds, 126 milliseconds)

Some observations so far:

• scheduling is sloooooooooow when there are lots of executors. EDIT: see note below, this is now explained, but leaving discussion here. (this is true even before any of the blacklist changes are in place). on my laptop anyway, it starts to drop off dramatically when there are around 200 nodes (800 executors in my setup). Almost all the time is spent in taskScheduler.resourceOffers. I am pretty certain the slowdown is not coming from the mock setup -- it doesn't care how many nodes there are. The only part of it which does scale is that tasks will get queued up for how many slots there are -- but scheduling is fast even if I have one "super executor" with 20K cores. The code shown here includes the new blacklist mechanism, but the numbers are basically the same either way.
  To be clear -- the super node has about 5x as many cores as the 550 node cluster, yet the scheduling is > 30x slower on the 550 node cluster.

> test-only org.apache.spark.scheduler.SchedulerPerformanceSuite -- -z "COMPARE A"
...
ran 3 iterations in 12.7 s (4.2 s per itr)
[info] - COMPARE A Scheduling speed -- large job on a super node (13 seconds, 707 milliseconds)
ran 3 iterations in 12.5 s (4.2 s per itr)
[info] - COMPARE A Scheduling speed -- large job on 50 node cluster (12 seconds, 550 milliseconds)
ran 2 iterations in 10.1 s (5.0 s per itr)
[info] - COMPARE A Scheduling speed -- large job on 100 node cluster (10 seconds, 96 milliseconds)
ran 1 iterations in 10.5 s (10.5 s per itr)
[info] - COMPARE A: Scheduling speed -- large job on 200 node cluster (10 seconds, 531 milliseconds)
ran 1 iterations in 22.4 s (22.4 s per itr)
[info] - COMPARE A: Scheduling speed -- large job on 300 node cluster (22 seconds, 460 milliseconds)
ran 1 iterations in 46.9 s (46.9 s per itr)
[info] - COMPARE A: Scheduling speed -- large job on 400 node cluster (46 seconds, 954 milliseconds)
ran 1 iterations in 1.1 m (1.1 m per itr)
[info] - COMPARE A: Scheduling speed -- large job on 450 node cluster (1 minute, 4 seconds)
ran 1 iterations in 1.6 m (1.6 m per itr)
[info] - COMPARE A: Scheduling speed -- large job on 500 node cluster (1 minute, 33 seconds)
ran 1 iterations in 2.3 m (2.3 m per itr)
[info] - COMPARE A: Scheduling speed -- large job on 550 node cluster (2 minutes, 20 seconds)

• EDIT: The performance issue were mostly just a wrinkle of how I was doing the tests, and that I was stressing the new version more. After making the tests consistent, its no slower than before. In fact, after discovering one bottleneck even in the old blacklist code, and fixing it, the new version is 20x faster when there is a bad node and you use the node-blacklist strategy. I don't fully understand why this is yet, but when there are failures, the advanced blacklist strategy is significantly slower than the "single task" strategy. (Note that before there are any failures, it is just as fast). Here its 2x slower on a small example, but with more tasks it can easily be > 5x slower

results running on my laptop using 456f578
In particular, note how much faster the runs are when (1) we use the advanced blacklist (2) all the bad executors are on one node and (3) there are no nodes with just one bad executor (the run named "COMPARE D bad host with advanced blacklist (12 seconds, 608 milliseconds)")

Iteration 0 finished in 1.1 m
ran 1 iterations in 1.1 m (1.1 m per itr)
[info] - COMPARE D bad exec with simple blacklist (1 minute, 4 seconds)
Iteration 0 finished in 2.4 m
ran 1 iterations in 2.4 m (2.4 m per itr)
[info] - COMPARE D two bad execs with simple blacklist (2 minutes, 21 seconds)
Iteration 0 finished in 1.3 m
ran 1 iterations in 1.3 m (1.3 m per itr)
[info] - COMPARE D bad exec with advanced blacklist (1 minute, 17 seconds)
Iteration 0 finished in 4.2 s
Iteration 1 finished in 4.2 s
Iteration 2 finished in 4.2 s
ran 3 iterations in 12.6 s (4.2 s per itr)
[info] - COMPARE D bad host with advanced blacklist (12 seconds, 608 milliseconds)
Iteration 0 finished in 2.2 m
ran 1 iterations in 2.2 m (2.2 m per itr)
[info] - COMPARE D bad exec and host with advanced blacklist (2 minutes, 10 seconds)

• When there are no failures, the blacklist doesn't seem to significantly slow things down. And on a large cluster, scheduling is already really slow in taskScheduler.resourceOffers, so it doesn't make any real difference.

[squito]

as of 10a83cc, on my laptop:

ran 111 iterations in 10.0 s (90 ms per itr)
[info] - Scheduling speed -- small job on a small cluster (11 seconds, 49 milliseconds)
ran 3 iterations in 12.7 s (4.2 s per itr)
[info] - COMPARE C Scheduling speed -- large job on a small cluster (12 seconds, 760 milliseconds)
ran 3 iterations in 13.0 s (4.3 s per itr)
[info] - COMPARE C Scheduling speed -- large job on a small cluster with advanced blacklist (13 seconds, 71 milliseconds)
ran 2 iterations in 10.6 s (5.3 s per itr)
[info] - COMPARE A Scheduling speed -- large job on a super node (10 seconds, 617 milliseconds)
ran 3 iterations in 13.2 s (4.4 s per itr)
[info] - COMPARE A Scheduling speed -- large job on 50 node cluster (13 seconds, 262 milliseconds)
ran 2 iterations in 11.1 s (5.6 s per itr)
[info] - COMPARE A Scheduling speed -- large job on 100 node cluster (11 seconds, 142 milliseconds)
ran 1 iterations in 10.7 s (10.7 s per itr)
[info] - COMPARE A: Scheduling speed -- large job on 200 node cluster (10 seconds, 676 milliseconds)
ran 1 iterations in 25.8 s (25.8 s per itr)
[info] - COMPARE A: Scheduling speed -- large job on 300 node cluster (25 seconds, 787 milliseconds)
ran 1 iterations in 34.2 s (34.2 s per itr)
[info] - COMPARE A: Scheduling speed -- large job on 400 node cluster (34 seconds, 218 milliseconds)
ran 1 iterations in 1.1 m (1.1 m per itr)
[info] - COMPARE A: Scheduling speed -- large job on 450 node cluster (1 minute, 3 seconds)
ran 1 iterations in 1.5 m (1.5 m per itr)
[info] - COMPARE A: Scheduling speed -- large job on 500 node cluster (1 minute, 28 seconds)
ran 1 iterations in 2.4 m (2.4 m per itr)
[info] - COMPARE A: Scheduling speed -- large job on 550 node cluster (2 minutes, 26 seconds)
ran 2 iterations in 10.6 s (5.3 s per itr)
[info] - COMPARE B: Lots of nodes (10 seconds, 637 milliseconds)
ran 2 iterations in 11.0 s (5.5 s per itr)
[info] - COMPARE B: Lots of executors, one node (11 seconds, 81 milliseconds)
ran 3 iterations in 13.0 s (4.3 s per itr)
[info] - COMPARE B: Super executor (13 seconds, 7 milliseconds)
ran 1 iterations in 58.4 s (58.4 s per itr)
[info] - COMPARE D bad exec with simple blacklist (58 seconds, 433 milliseconds)
ran 1 iterations in 2.2 m (2.2 m per itr)
[info] - COMPARE D two bad execs with simple blacklist (2 minutes, 14 seconds)
ran 1 iterations in 1.2 m (1.2 m per itr)
[info] - COMPARE D bad exec with advanced blacklist (1 minute, 11 seconds)
ran 3 iterations in 12.7 s (4.2 s per itr)
[info] - COMPARE D bad host with advanced blacklist (12 seconds, 756 milliseconds)
ran 1 iterations in 2.1 m (2.1 m per itr)
[info] - COMPARE D bad exec and host with advanced blacklist (2 minutes, 3 seconds)

[squito]

Some updates on the slow scheduling with lots of nodes:

1. this was largely an artifact of the testing framework -- when a task completed, it revived offers on all executors, not just the executor that completed. Fixed here: 8ce8596 and now there is no noticeable difference in performance

2. But there was another performance issue this ended up unearthing. TaskSchedulerImpl.resourceOfferSingleTaskSet indexes into shuffledOffers by index, but that can be really slow if you've got a linkedlist style structure (all that is guaranteed by Seq, and indeed the normal structure you'll get from eg. where its called in CoarseGrainedSchedulerBackend). This only matters when you first submit tasks to a taskset, though, or when you add a new executor. With a large cluster, adding and removing executors can happen quite frequently with dynamic allocation, so this can be a notable slowdown. (though perhaps that call should also use makeOffers(executorId) instead? not sure why it doesn't already.)