EC2Cluster support for multiple instance types

[pwerth]

Background

With large clusters of popular instance types, it's not uncommon to get InsufficientInstanceCapacity errors from AWS. This request is for a feature to help avoid/circumvent these errors.

Desired Feature

Instead of specifying one worker instance type, one could instead specify a (sorted) list of instance types [X, Y, Z, ...]. If 100 workers are requested, we attempt to create 100 instances of type X. If N1 instances are created then we receive a capacity error, we request (100-N1) instances of type Y. This continues until all instance types are exhausted. Of course, we could still be left with fewer than the desired number of workers, but this approach increases our chances of success.

Note: the burden is on the user to ensure that the instance types are appropriate for the use case. For example, users would probably want to ensure that instances types 2, 3, ... have at least as many vCPU and memory as type 1. This is definitely a "super user" type of feature.

Proposed Implementation

To maintain backwards compatibility, we can update worker_instance_type to accept either a string or a list of strings. If we are not worried about backwards compatibility, we can change the argument to worker_instance_types and require a list, but this seems like an unnecessary breaking change.

I'm open to other implementations if you think the idea is worth working on. I'm also happy to contribute a solution here.

[jacobtomlinson]

This sounds interesting, definitely a power user feature as you say.

When it comes to power-user features I like to err on the side of non-breaking so that regular users can just ignore it. I really like the idea of making worker_instance_type optionally take a list, seems like a very minimal impact. I'm not a big fan of renaming the argument.

If you have interest in raising a PR I think this would be a good direction to do in, so feel free to ping me for a review.