Existing VPC and subnets failure

[cdenneen]

What happened?
A description of actual behavior (with error messages).
Not using subnet's specified in configuration in eksctl output and subnets are empty in the resulting CloudFormation template uploaded to AWS.

[ℹ]  using region us-east-1
[ℹ]  subnets for us-east-1a - public:10.200.0.0/19 private:10.200.96.0/19
[ℹ]  subnets for us-east-1b - public:10.200.32.0/19 private:10.200.128.0/19
[ℹ]  subnets for us-east-1c - public:10.200.64.0/19 private:10.200.160.0/19
[ℹ]  nodegroup "ng-1-workers" will use "ami-0c24db5df6badc35a" [AmazonLinux2/1.11]
[ℹ]  creating EKS cluster "dev-test" in "us-east-1" region
[ℹ]  will create 2 separate CloudFormation stacks for cluster itself and the initial nodegroup
[ℹ]  if you encounter any issues, check CloudFormation console or try 'eksctl utils describe-stacks --region=us-east-1 --name=dev-test'
[ℹ]  creating cluster stack "eksctl-dev-test-cluster"
[✖]  unexpected status "DELETE_IN_PROGRESS" while waiting for CloudFormation stack "eksctl-dev-test-cluster" to reach "CREATE_COMPLETE" status
[ℹ]  fetching stack events in attempt to troubleshoot the root cause of the failure
[!]  AWS::CloudFormation::Stack/eksctl-dev-test-cluster: DELETE_IN_PROGRESS – "User Initiated"
[ℹ]  AWS::CloudFormation::Stack/eksctl-dev-test-cluster: ROLLBACK_IN_PROGRESS – "The following resource(s) failed to create: [ControlPlane, PolicyNLB, PolicyCloudWatchMetrics]. . Rollback requested by user."
[✖]  AWS::IAM::Policy/PolicyCloudWatchMetrics: CREATE_FAILED – "Resource creation cancelled"
[✖]  AWS::IAM::Policy/PolicyNLB: CREATE_FAILED – "Resource creation cancelled"
[✖]  AWS::EKS::Cluster/ControlPlane: CREATE_FAILED – "The subnet ID '' does not exist (Service: AmazonEC2; Status Code: 400; Error Code: InvalidSubnetID.NotFound; Request ID: ) (Service: AmazonEKS; Status Code: 400; Error Code: InvalidParameterException; Request ID: )"
[ℹ]  AWS::IAM::Policy/PolicyCloudWatchMetrics: CREATE_IN_PROGRESS – "Resource creation Initiated"
[ℹ]  AWS::IAM::Policy/PolicyNLB: CREATE_IN_PROGRESS – "Resource creation Initiated"
[ℹ]  AWS::IAM::Policy/PolicyCloudWatchMetrics: CREATE_IN_PROGRESS
[ℹ]  AWS::IAM::Policy/PolicyNLB: CREATE_IN_PROGRESS
[ℹ]  AWS::EKS::Cluster/ControlPlane: CREATE_IN_PROGRESS
[ℹ]  AWS::IAM::Role/ServiceRole: CREATE_COMPLETE
[ℹ]  AWS::EC2::SecurityGroupIngress/IngressInterNodeGroupSG: CREATE_COMPLETE
[ℹ]  AWS::EC2::SecurityGroupIngress/IngressInterNodeGroupSG: CREATE_IN_PROGRESS – "Resource creation Initiated"
[ℹ]  AWS::EC2::SecurityGroupIngress/IngressInterNodeGroupSG: CREATE_IN_PROGRESS
[ℹ]  AWS::EC2::SecurityGroup/ClusterSharedNodeSecurityGroup: CREATE_COMPLETE
[ℹ]  AWS::EC2::SecurityGroup/ControlPlaneSecurityGroup: CREATE_COMPLETE
[ℹ]  AWS::EC2::SecurityGroup/ClusterSharedNodeSecurityGroup: CREATE_IN_PROGRESS – "Resource creation Initiated"
[ℹ]  AWS::EC2::SecurityGroup/ControlPlaneSecurityGroup: CREATE_IN_PROGRESS – "Resource creation Initiated"
[ℹ]  AWS::IAM::Role/ServiceRole: CREATE_IN_PROGRESS – "Resource creation Initiated"
[ℹ]  AWS::IAM::Role/ServiceRole: CREATE_IN_PROGRESS
[ℹ]  AWS::EC2::SecurityGroup/ClusterSharedNodeSecurityGroup: CREATE_IN_PROGRESS
[ℹ]  AWS::EC2::SecurityGroup/ControlPlaneSecurityGroup: CREATE_IN_PROGRESS
[ℹ]  AWS::CloudFormation::Stack/eksctl-dev-test-cluster: CREATE_IN_PROGRESS – "User Initiated"
[ℹ]  1 error(s) occurred and cluster hasn't been created properly, you may wish to check CloudFormation console
[ℹ]  to cleanup resources, run 'eksctl delete cluster --region=us-east-1 --name=dev-test'
[✖]  waiting for CloudFormation stack "eksctl-dev-test-cluster" to reach "CREATE_COMPLETE" status: ResourceNotReady: failed waiting for successful resource state
[✖]  failed to create cluster "dev-test"

eksctl from homebrew

Versions
Please paste in the output of these commands:

$ eksctl version
version.Info{BuiltAt:"", GitCommit:"", GitTag:"0.1.19"}
$ uname -a
Darwin macbook.local 18.2.0 Darwin Kernel Version 18.2.0: Mon Nov 12 20:24:46 PST 2018; root:xnu-4903.231.4~2/RELEASE_X86_64 x86_64
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.2", GitCommit:"cff46ab41ff0bb44d8584413b598ad8360ec1def", GitTreeState:"clean", BuildDate:"2019-01-13T23:15:13Z", GoVersion:"go1.11.4", Compiler:"gc", Platform:"darwin/amd64"}

devtest-github.txt

[cdenneen]

based on #447 and #436 I'm trying with modifying the capitalization of public -> Public and private -> Private.
Also have to remove the upper availabilityZones as it conflicts with vpc.subnets

[cdenneen]

So with those changes it appears to have created the cluster this time but not the nodegroup:

[ℹ]  creating cluster stack "eksctl-dev-test-cluster"
[ℹ]  creating nodegroup stack "eksctl-dev-test-nodegroup-ng-1-workers"
[ℹ]  1 error(s) occurred and cluster hasn't been created properly, you may wish to check CloudFormation console
[ℹ]  to cleanup resources, run 'eksctl delete cluster --region=us-east-1 --name=dev-test'
[✖]  VPC doesn't have subnets in us-east-1a (subnets=map[string]v1alpha3.Network{"us-east-1b":v1alpha3.Network{ID:"subnet-44444444", CIDR:(*ipnet.IPNet)(0xc0007c34d0)}, "us-east-1c":v1alpha3.Network{ID:"subnet-55555555", CIDR:(*ipnet.IPNet)(0xc0007c3530)}, "us-east-1d":v1alpha3.Network{ID:"subnet-66666666", CIDR:(*ipnet.IPNet)(0xc0007c3590)}} AZs=[]string{"us-east-1a", "us-east-1b", "us-east-1c"})
[✖]  failed to create cluster "dev-test"

which ended up being a typo on my part... but I can't "update" at this point... cluster already exists and CF for it exists so errors but the instancegroup doesn't exist.

I even tried eksctl create instancegroup -f devtest.yaml but -f isn't a parameter for nodegroups.
There is also no update function for something like eksctl update cluster -f devtest.yaml so required to destroy and recreate cluster.

I believe eksctl create/delete/etc instancegroup should allow for -f config.yaml
and if you want to update a cluster.yaml with possibly adding more instancegroups or whatever... eksctl update cluster -f cluster.yaml should work as well. (this allows to modify instance groups, add them, change ami, etc.)

[cdenneen]

eksctl delete cluster doesn't accept -f flag?
So eksctl delete cluster -n devtest will delete the cluster but not the CloudFormation.
Can we have that work so it runs a delete-stack?

[rustysys-dev]

I have been using eksctl for a little while now, and -f flag should work for create cluster.

Also unless there was a break earlier eksctl should also delete the cluster and node-group CloudFormation stack. The command does not from my understanding wait for the stack to be deleted. After running the command you will have to wait while the CloudFormation stack deletes itself before you can run another create command.

Can you share your config.yaml and any errors from AWS console to understand the issue?

[cdenneen]

so the Delete does seem to look to initiate a CloudFormation stack deletion... I'm having an issue with "Rate exceeded" so it's not actually deleting which is very strange, not sure how many ListStacks calls eksctl is sending but I got almost 300 logged to CloudTrail in less than 5 minutes from eksctl.

I still would like the ability to update the stack (if cluster exists... then update the node group)... or -f nodegroup and it will ignore the cluster part but create/update nodegroups specified in the config... as well as delete -f removing cluster and nodegroup stacks.. but appears -f is only acceptable flag for create.

[errordeveloper]

Looks like duplicate of #471, isn't it?

[errordeveloper]

eksctl delete cluster doesn't accept -f flag?

Not yet, see #19. That would be an easy one to fix, if you'd like to contribute.

So eksctl delete cluster -n devtest will delete the cluster but not the CloudFormation.
Can we have that work so it runs a delete-stack?

It does delete all the stacks that eksctl create cluster|nodegroup creates.

[cdenneen]

Looks like duplicate of #471, isn't it?

@errordeveloper Similar... it was part of the Private and Public in the config vs the lowercase versions and it was part of me having the correct vpc.subnets but availabilityZones parameter in the nodegroup I had 1a but didn't have 1a in the vpc.subnets so created cluster without the nodegroup.

I can close this as it seems the case sensitivity of the Public/public and Private/private is already known issue, #484 handles the missing nodegroup issue and #19 while not explicitly for -f flags to all operations (create/delete/update) I think the ClusterAPI is good place to start.