Managing private-only clusters

[errordeveloper]

We are adding support for private VPC endpoints in eksctl. It would be nice to allow private-only access.

The main challenge is that aside from AWS API calls we do need to make calls to Kubernetes API, e.g. to update kube-system:ConfigMap/aws-auth or install Flux.

It's certainly possible to design cluster creation process in a way that lets you create a cluster from any machine and disable public endpoint right at the very end, this would require any drastic changes, we would just need to rearrange the order of a few calls.
Once the cluster is created, if user wants to run any follow-up eksctl commands, e.g. upgrade or scale the cluster, they would have run it from inside the VPC somehow.

Alternatively, they can manage the VPC separately and import it during cluster creation. It would be nice if this wasn't required, shared VPC (#892) would be one option, perhaps the easiest one.

There are probably other potential solutions, but those are likely to be out of scope for eksctl, e.g using Lambda or an EC2 instance via SSM.

[martina-if]

Consider adding some validation of the subnets as exlained here

[kalbir]

@martina-if Do we need this and #1765? I'm looking at #1765 to be the main on for tracking private clusters.

[martina-if]

@kalbir no, let's consolidate all dups into one and close the rest.

[martina-if]

closed via #2384