Protect controller from becoming unscheduleable

[jahkeup]

Also, the Operator Controller should handle killing itself, especially so in a single noded cluster! It should not prevent itself from getting scheduled in a Cluster.

Originally posted by @jahkeup in bottlerocket-os/bottlerocket#239 (comment)

[jahkeup]

One way to protect the controller could be to have the controller save its hosting node for to be updated last. Once it updated through the other nodes, the controller would delete its Pod to be rescheduled and only once started elsewhere would it continue to update that last node.

The controller's deployment should then include bottlerocket.aws/update-available in its antiAffinity weighted selector (preferring update-available==false) so that it lands on updated hosts first.

This method wouldn't account for a single noded cluster or one where only a node was Ready and Schedulable. The controller will have to check that it considers itself to be reschedulable prior to stopping its Pod.

[cbgbt]

Some thoughts from conversation with @somnusfish:

• Consider evicting the controller first when doing a drain
• Add a timeout to drains, error out and trigger our new crash loop handling code (0.2.0: Handle update-reboot failures/ "crash loops" #123) if they get stuck. Drains should never roll forward on timeouts.
• Add PDBs to apiserver deployment to ensure we always have at least 2 running in the cluster.

[cbgbt]

We want to add the ability to allow brupop to update many nodes simultaneously, which makes this more important. Adding this to the 1.0.0 release milestone.