[EKS] [request]: 1-click upgrades support blue/green or reversibility

[cgetzen]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request

In this year's re:invent's EKS deep dive, the speaker mentioned that more and more customers were moving from 1-click upgrade to blue/green EKS deployments.
I'd like to see the option to blue/green the API server with the same "1-click" ease.
This will enable one of AWS' well architected framework best practices "Make frequent, small, reversible changes"

Which service(s) is this request for?
EKS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
Reliable, reversible kubernetes upgrades currently require spinning up a new cluster and migrating workloads to it. This requires managing migration automation or spending resources on migration toil.

Are you currently working around this issue?
self-managed blue/green EKS deployments.

[mikestef9]

EKS automating blue/green upgrades is challenging given the variety of ways customers use their clusters. Some pain points/requirements we are aware of when moving to a new cluster include:

• Updating trust policy of IAM roles for service accounts, see [EKS]: IAM Roles for Service Accounts enhancements for usage across multiple clusters #1408 for ideas to simplify
• Moving EBS backed persistent volumes
• Service discovery - moving a service to a new cluster when a service in the old cluster depends on it
• Horizontal pod autoscaling (HPA) metrics
• IPv4 address exhaustion if creating the new cluster in the same VPC
• Cost - for a short period of time, you are creating double the resources
• Cross cluster load balancing for externally exposed services
• Dependencies on metrics/alarms based on cluster names etc
• Redirecting CD systems to deploy applications to the new cluster

Ultimately, we see success for customers here with admin teams being able to blue/green cluster upgrades with little to no involvement from downstream service teams.

Would love to get some more feedback here on additional challenges you either experience today with blue/green upgrades, or foresee if you were to take this approach.

[cgetzen]

That makes a lot of sense!
I agree that AWS probably shouldn't try to manage blue/green updates to the data plane.
I'm envisioning blue/green upgrades of the control plane only. I'd like the ability to test the new control plane against an existing cluster before cutting over. If that's not possible, it'd be nice to have an undo button in case the 1-click upgrade causes an outage due to incompatible operators/etc or performance issues.