Use gp2 400g for m5 instances on build01

[hongkailiu]

Follow up https://coreos.slack.com/archives/CHY2E1BL4/p1588346798357400?thread_ts=1588269732.276800&cid=CHY2E1BL4

1. gp2 400g as root device for m5 instances
2. deletePolicy: oldest to remove the oldest machines when scaling down
3. m5d machinessets are still there (we will verify @cwilkers 's fix if needed) and if gp2 goes through, we will remove m5d and raid0 objects.

/cc @stevekuznetsov @smarterclayton @cgwalters @rphillips @michaelgugino

[openshift-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: hongkailiu, stevekuznetsov

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• clusters/OWNERS [hongkailiu,stevekuznetsov]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[cgwalters]

It clearly simplifies everything to just use a single big disk, but I think the core idea with the m5d class of instances that have instance-attached storage is that it's cheaper/higher performance than using EBS because ultimately it's less reliable - it's instance attached, not replicated like EBS is.

I think for CI and other workloads we should eventually have an OpenShift component that helps one make use of these instance-attached drives automatically for things like container images and emptyDir.

[smarterclayton]

CI has never needed this level of performance though - we just don't write enough to disk to need more than a large gp2. I would reserve m5d for INSANE io use cases, like production databases doing 200k tps

[cgwalters]

I'd believe you, although no one has tried o numbers here right? Even if you're right for our current CI workload, I think the point still stands that the platform should make it easy to instance-attached disks for the container images/OS logs etc. (I'd hope OCS/Ceph makes it easy to use them for PVs, but that's a separate thing)

Could imagine some sort of:

instanceAttachedStorageMode: default|var

field somewhere...where default means "don't touch" (current state) and var is a linear mapping (not raid) maybe in the machineset and the MCO picks up handling it or something.