Create cluster via config file #381

errordeveloper · 2019-01-02T15:12:20Z

Description

This is initial step towards #19. Let's add this in to start with and aim to evolve it as an experimental feature.

This change adds some of the Kubernetes API tooling, and adds JSON tags to api.ClusterConfig.
From user's perspective, there is now a --config-file/-f flag in eksctl create cluster. The format of the object right now is based directly on api.ClusterConfig, we may eventually mould it into Cluster API ProvideSpec, but for now let's keep it simple.

Checklist

Code compiles correctly (i.e make build)
All tests passing (i.e. make test)

Closes #377.

errordeveloper · 2019-01-02T15:17:16Z

TODOs:

squash
re-read the diff
fix AMI resolution problem
make all operation on ng use cfg.NodeGroups instead
test each of example configs that are usable directly:
- examples/01-simple-cluster.yaml
- examples/02-custom-vpc-cidr-no-nodes.yaml
- examples/05-two-nodegroups.yaml
complete iniline TODOs or move to issues

errordeveloper · 2019-01-02T16:04:29Z

pkg/apis/eksctl.io/v1alpha3/types.go

+	// +optional
+	AvailabilityZones []string `json:"availabilityZones,omitempty"`
+
+	// TODO: refactor and move IAM addons to nodegroup


Will save for another PR (see #361 & #363).

pkg/apis/eksctl.io/v1alpha3/types.go

pkg/ctl/create/cluster.go

errordeveloper · 2019-01-03T14:39:25Z

This PR now has a fix for #385 (f98fcf8).

errordeveloper · 2019-01-03T15:45:52Z

pkg/apis/eksctl.io/v1alpha3/types.go

+	// +optional
+	InstanceRoleARN string `json:"instanceRoleARN,omitempty"` // TODO: read-only
+
+	// TODO move to separate struct


errordeveloper · 2019-01-03T19:16:43Z

I'm rather happy with this, but would like to re-test examples/05-two-nodegroups.yaml once again, and run the integration suite.

errordeveloper · 2019-01-03T21:43:13Z

Integration tests passed, ./eksctl create cluster -f ./examples/05-two-nodegroups.yaml passed and nodes have right name labels set correctly.

errordeveloper · 2019-01-04T10:15:15Z

pkg/eks/api.go

+	ng.Labels[api.NodeGroupNameLabel] = ng.Name
+
+	for l := range ng.Labels {
+		if len(strings.Split(l, "/")) > 2 {


We should actually call proper validation function here.

mumoshu

Just a nit. LGTM overall! Thanks for your efforts ☺️

pkg/apis/eksctl.io/v1alpha3/types.go

- Make it incompatible with all flags that modify state of config - Most default are implemented by constructors (where possible)

- fix bug where tags slice was shared accidentially - move tag constants to api package - automatically set node labels for cluster and nodegroup names - ensure labels have max 1 '/' separator, otherwise kubelet fails to start

- fix missing return on nodegroup deletion error causing false-positive - ensure describe stack is called with StackId whenever possible, to avoid error on stacks that happen to get deleted already

errordeveloper · 2019-01-06T13:20:14Z

@mumoshu thanks for looking over this PR! I've removed stale TODO note, and made DesiredCapacity field optional. Please take another look, whenever you have a moment.

marccarre · 2019-01-07T00:55:10Z

pkg/apis/eksctl.io/v1alpha3/types.go

+	PrivateNetworking bool `json:"privateNetworking"`
+
+	// +optional
+	DesiredCapacity int `json:"desiredCapacity"`


Could the "desired" capacity not be met?
Would it be clearer to have something named like, e.g., the below?

Count int `json:"count"`

Yes, we had it as NodeCount at the top level some time ago, but now it's called DesiredCapacity to match the AWS API. However, the flag is still --nodes, so there is a little inconsistency there too. Some refactoring is pending here also (#385).

Fair enough 😅
Matching the AWS API probably takes precedence over any other argument/preference 👍

pkg/cfn/builder/outputs.go

pkg/ctl/create/cluster.go

errordeveloper force-pushed the cluster-config branch from f25a070 to b734001 Compare January 2, 2019 15:15

errordeveloper force-pushed the cluster-config branch 2 times, most recently from b92286b to 20f68cf Compare January 2, 2019 15:57

errordeveloper commented Jan 2, 2019

View reviewed changes

pkg/apis/eksctl.io/v1alpha3/types.go Outdated Show resolved Hide resolved

errordeveloper commented Jan 2, 2019

View reviewed changes

pkg/apis/eksctl.io/v1alpha3/types.go Outdated Show resolved Hide resolved

errordeveloper commented Jan 2, 2019

View reviewed changes

pkg/ctl/create/cluster.go Outdated Show resolved Hide resolved

errordeveloper force-pushed the cluster-config branch 5 times, most recently from 4363ad3 to b2ba54f Compare January 3, 2019 09:43

errordeveloper mentioned this pull request Jan 3, 2019

Use an existing vpc, generate subnets (WIP) #379

Closed

5 tasks

errordeveloper force-pushed the cluster-config branch from 6ca413d to 734f097 Compare January 3, 2019 14:29

errordeveloper force-pushed the cluster-config branch 2 times, most recently from f98fcf8 to 66fd306 Compare January 3, 2019 14:45

errordeveloper commented Jan 3, 2019

View reviewed changes

errordeveloper mentioned this pull request Jan 3, 2019

automatic label for nodegroups #377

Closed

errordeveloper force-pushed the cluster-config branch from 4555e43 to 8811196 Compare January 3, 2019 18:06

errordeveloper changed the title ~~WIP: Create cluster via config file~~ Create cluster via config file Jan 3, 2019

errordeveloper added 2 commits January 3, 2019 18:14

Add example config files

0a13447

Vendor a customised ipnet package

4c85bb8

errordeveloper force-pushed the cluster-config branch 3 times, most recently from 5123632 to 8a8dcc4 Compare January 3, 2019 19:15

errordeveloper mentioned this pull request Jan 4, 2019

Don't overload '--ssh-public-key' and allow bootstrapping multiple public key files #391

Closed

errordeveloper force-pushed the cluster-config branch 2 times, most recently from 0d66a41 to 41ea017 Compare January 4, 2019 10:07

errordeveloper commented Jan 4, 2019

View reviewed changes

errordeveloper force-pushed the cluster-config branch 2 times, most recently from 0b82a2c to 29e1992 Compare January 5, 2019 03:15

mumoshu reviewed Jan 6, 2019

View reviewed changes

pkg/apis/eksctl.io/v1alpha3/types.go Show resolved Hide resolved

This was referenced Jan 6, 2019

Node group's desired capacity should be optional #395

Closed

feat: add taints to the node on creation #396

Closed

mumoshu reviewed Jan 6, 2019

View reviewed changes

pkg/apis/eksctl.io/v1alpha3/types.go Outdated Show resolved Hide resolved

errordeveloper added 8 commits January 6, 2019 13:15

Switch to a Kubernetes-style API

80b8fa0

Implement eksctl create cluster --config-file=<configFile>

9900824

- Make it incompatible with all flags that modify state of config - Most default are implemented by constructors (where possible)

Implement support for multiple nodegroups

135dc80

Make nodegroup auth configmap operations idempotent

37fc0e5

Improve log messages

d25af70

Improve labels and tags

3651823

- fix bug where tags slice was shared accidentially - move tag constants to api package - automatically set node labels for cluster and nodegroup names - ensure labels have max 1 '/' separator, otherwise kubelet fails to start

Improve nodegroup deletion error handling

a7553b3

- fix missing return on nodegroup deletion error causing false-positive - ensure describe stack is called with StackId whenever possible, to avoid error on stacks that happen to get deleted already

Re-enumerate example config files, improve comments

c3dd21c

mumoshu previously approved these changes Jan 6, 2019

View reviewed changes

errordeveloper dismissed mumoshu’s stale review via f568b25 January 6, 2019 13:53

errordeveloper force-pushed the cluster-config branch from 29e1992 to f568b25 Compare January 6, 2019 13:53

marccarre reviewed Jan 7, 2019

View reviewed changes

pkg/cfn/builder/outputs.go Outdated Show resolved Hide resolved

marccarre reviewed Jan 7, 2019

View reviewed changes

pkg/ctl/create/cluster.go Outdated Show resolved Hide resolved

marccarre previously approved these changes Jan 7, 2019

View reviewed changes

Reduce debug logs

7b50b82

errordeveloper dismissed marccarre’s stale review via 7b50b82 January 7, 2019 09:00

errordeveloper force-pushed the cluster-config branch from f568b25 to 7b50b82 Compare January 7, 2019 09:00

errordeveloper merged commit 4d872f7 into master Jan 7, 2019

errordeveloper deleted the cluster-config branch January 7, 2019 09:20

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create cluster via config file #381

Create cluster via config file #381

errordeveloper commented Jan 2, 2019 •

edited

Loading

errordeveloper commented Jan 2, 2019 •

edited

Loading

errordeveloper Jan 2, 2019 •

edited

Loading

errordeveloper commented Jan 3, 2019 •

edited

Loading

errordeveloper Jan 3, 2019

errordeveloper commented Jan 3, 2019 •

edited

Loading

errordeveloper commented Jan 3, 2019

errordeveloper Jan 4, 2019

errordeveloper Jan 4, 2019

mumoshu left a comment

errordeveloper commented Jan 6, 2019 •

edited

Loading

marccarre Jan 7, 2019

errordeveloper Jan 7, 2019

marccarre Jan 7, 2019

Create cluster via config file #381

Create cluster via config file #381

Conversation

errordeveloper commented Jan 2, 2019 • edited Loading

Description

Checklist

errordeveloper commented Jan 2, 2019 • edited Loading

errordeveloper Jan 2, 2019 • edited Loading

Choose a reason for hiding this comment

errordeveloper commented Jan 3, 2019 • edited Loading

errordeveloper Jan 3, 2019

Choose a reason for hiding this comment

errordeveloper commented Jan 3, 2019 • edited Loading

errordeveloper commented Jan 3, 2019

errordeveloper Jan 4, 2019

Choose a reason for hiding this comment

errordeveloper Jan 4, 2019

Choose a reason for hiding this comment

mumoshu left a comment

Choose a reason for hiding this comment

errordeveloper commented Jan 6, 2019 • edited Loading

marccarre Jan 7, 2019

Choose a reason for hiding this comment

errordeveloper Jan 7, 2019

Choose a reason for hiding this comment

marccarre Jan 7, 2019

Choose a reason for hiding this comment

errordeveloper commented Jan 2, 2019 •

edited

Loading

errordeveloper commented Jan 2, 2019 •

edited

Loading

errordeveloper Jan 2, 2019 •

edited

Loading

errordeveloper commented Jan 3, 2019 •

edited

Loading

errordeveloper commented Jan 3, 2019 •

edited

Loading

errordeveloper commented Jan 6, 2019 •

edited

Loading