Skip to content

Commit bf35ebe

Browse files
author
Grzegorz Lisowski
committed
- Worker locals/defaults moved to workers submodule
- Create separate defaults for node groups - Workers IAM management left outside of module as both node_group and worker_groups uses them
1 parent c4edc6f commit bf35ebe

33 files changed

+961
-1360
lines changed

‎README.md‎

Lines changed: 5 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -49,12 +49,12 @@ module "my-cluster" {
4949
subnets = ["subnet-abcde012", "subnet-bcde012a", "subnet-fghi345a"]
5050
vpc_id = "vpc-1234556abcdef"
5151
52-
worker_groups = [
53-
{
52+
worker_groups = {
53+
group = {
5454
instance_type = "m4.large"
5555
asg_max_size = 5
5656
}
57-
]
57+
}
5858
}
5959
```
6060
## Conditional creation
@@ -150,7 +150,6 @@ MIT Licensed. See [LICENSE](https://github.com/terraform-aws-modules/terraform-a
150150
| kubernetes | >= 1.11.1 |
151151
| local | >= 1.4 |
152152
| null | >= 2.1 |
153-
| random | >= 2.1 |
154153
| template | >= 2.1 |
155154

156155
## Inputs
@@ -205,8 +204,7 @@ MIT Licensed. See [LICENSE](https://github.com/terraform-aws-modules/terraform-a
205204
| worker\_create\_cluster\_primary\_security\_group\_rules | Whether to create security group rules to allow communication between pods on workers and pods using the primary cluster security group. | `bool` | `false` | no |
206205
| worker\_create\_initial\_lifecycle\_hooks | Whether to create initial lifecycle hooks provided in worker groups. | `bool` | `false` | no |
207206
| worker\_create\_security\_group | Whether to create a security group for the workers or attach the workers to `worker_security_group_id`. | `bool` | `true` | no |
208-
| worker\_groups | A list of maps defining worker group configurations to be defined using AWS Launch Configurations. See workers\_group\_defaults for valid keys. | `any` | `[]` | no |
209-
| worker\_groups\_launch\_template | A list of maps defining worker group configurations to be defined using AWS Launch Templates. See workers\_group\_defaults for valid keys. | `any` | `[]` | no |
207+
| worker\_groups | A map of maps defining worker group configurations to be defined using AWS Launch Templates. See workers\_group\_defaults for valid keys. | `any` | `{}` | no |
210208
| worker\_security\_group\_id | If provided, all workers will be attached to this security group. If not given, a security group will be created with necessary ingress/egress to work with the EKS cluster. | `string` | `""` | no |
211209
| worker\_sg\_ingress\_from\_port | Minimum port number from which pods will accept communication. Must be changed to a lower value if some pods in your cluster will expose a port lower than 1025 (e.g. 22, 80, or 443). | `number` | `1025` | no |
212210
| workers\_additional\_policies | Additional policies to be added to workers | `list(string)` | `[]` | no |
@@ -235,17 +233,7 @@ MIT Licensed. See [LICENSE](https://github.com/terraform-aws-modules/terraform-a
235233
| node\_groups | Outputs from EKS node groups. Map of maps, keyed by var.node\_groups keys |
236234
| oidc\_provider\_arn | The ARN of the OIDC Provider if `enable_irsa = true`. |
237235
| security\_group\_rule\_cluster\_https\_worker\_ingress | Security group rule responsible for allowing pods to communicate with the EKS cluster API. |
238-
| worker\_iam\_instance\_profile\_arns | default IAM instance profile ARN for EKS worker groups |
239-
| worker\_iam\_instance\_profile\_names | default IAM instance profile name for EKS worker groups |
240-
| worker\_iam\_role\_arn | default IAM role ARN for EKS worker groups |
241-
| worker\_iam\_role\_name | default IAM role name for EKS worker groups |
236+
| worker\_groups | Outputs from EKS worker groups. Map of maps, keyed by var.worker\_groups keys |
242237
| worker\_security\_group\_id | Security group ID attached to the EKS workers. |
243-
| workers\_asg\_arns | IDs of the autoscaling groups containing workers. |
244-
| workers\_asg\_names | Names of the autoscaling groups containing workers. |
245-
| workers\_default\_ami\_id | ID of the default worker group AMI |
246-
| workers\_launch\_template\_arns | ARNs of the worker launch templates. |
247-
| workers\_launch\_template\_ids | IDs of the worker launch templates. |
248-
| workers\_launch\_template\_latest\_versions | Latest versions of the worker launch templates. |
249-
| workers\_user\_data | User data of worker groups |
250238

251239
<!-- END OF PRE-COMMIT-TERRAFORM DOCS HOOK -->

‎aws_auth.tf‎

Lines changed: 2 additions & 40 deletions
Original file line numberDiff line numberDiff line change
@@ -1,48 +1,10 @@
1-
data "aws_caller_identity" "current" {
2-
}
1+
data "aws_caller_identity" "current" {}
32

43
locals {
5-
auth_launch_template_worker_roles = [
6-
for index in range(0, var.create_eks ? local.worker_group_launch_template_count : 0) : {
7-
worker_role_arn = "arn:${data.aws_partition.current.partition}:iam::${data.aws_caller_identity.current.account_id}:role/${element(
8-
coalescelist(
9-
aws_iam_instance_profile.workers_launch_template.*.role,
10-
data.aws_iam_instance_profile.custom_worker_group_launch_template_iam_instance_profile.*.role_name,
11-
[""]
12-
),
13-
index
14-
)}"
15-
platform = lookup(
16-
var.worker_groups_launch_template[index],
17-
"platform",
18-
local.workers_group_defaults["platform"]
19-
)
20-
}
21-
]
22-
23-
auth_worker_roles = [
24-
for index in range(0, var.create_eks ? local.worker_group_count : 0) : {
25-
worker_role_arn = "arn:${data.aws_partition.current.partition}:iam::${data.aws_caller_identity.current.account_id}:role/${element(
26-
coalescelist(
27-
aws_iam_instance_profile.workers.*.role,
28-
data.aws_iam_instance_profile.custom_worker_group_iam_instance_profile.*.role_name,
29-
[""]
30-
),
31-
index,
32-
)}"
33-
platform = lookup(
34-
var.worker_groups[index],
35-
"platform",
36-
local.workers_group_defaults["platform"]
37-
)
38-
}
39-
]
40-
414
# Convert to format needed by aws-auth ConfigMap
425
configmap_roles = [
436
for role in concat(
44-
local.auth_launch_template_worker_roles,
45-
local.auth_worker_roles,
7+
module.worker_groups.aws_auth_roles,
468
module.node_groups.aws_auth_roles,
479
) :
4810
{

‎cluster.tf‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -103,7 +103,7 @@ resource "aws_security_group_rule" "cluster_https_worker_ingress" {
103103
description = "Allow pods to communicate with the EKS cluster API."
104104
protocol = "tcp"
105105
security_group_id = local.cluster_security_group_id
106-
source_security_group_id = local.worker_security_group_id
106+
source_security_group_id = module.worker_groups.worker_security_group_id
107107
from_port = 443
108108
to_port = 443
109109
type = "ingress"

‎data.tf‎

Lines changed: 0 additions & 137 deletions
Original file line numberDiff line numberDiff line change
@@ -13,33 +13,6 @@ data "aws_iam_policy_document" "workers_assume_role_policy" {
1313
}
1414
}
1515

16-
data "aws_ami" "eks_worker" {
17-
filter {
18-
name = "name"
19-
values = [local.worker_ami_name_filter]
20-
}
21-
22-
most_recent = true
23-
24-
owners = [var.worker_ami_owner_id]
25-
}
26-
27-
data "aws_ami" "eks_worker_windows" {
28-
filter {
29-
name = "name"
30-
values = [local.worker_ami_name_filter_windows]
31-
}
32-
33-
filter {
34-
name = "platform"
35-
values = ["windows"]
36-
}
37-
38-
most_recent = true
39-
40-
owners = [var.worker_ami_owner_id_windows]
41-
}
42-
4316
data "aws_iam_policy_document" "cluster_assume_role_policy" {
4417
statement {
4518
sid = "EKSClusterAssumeRole"
@@ -55,119 +28,9 @@ data "aws_iam_policy_document" "cluster_assume_role_policy" {
5528
}
5629
}
5730

58-
data "template_file" "userdata" {
59-
count = var.create_eks ? local.worker_group_count : 0
60-
template = lookup(
61-
var.worker_groups[count.index],
62-
"userdata_template_file",
63-
file(
64-
lookup(var.worker_groups[count.index], "platform", local.workers_group_defaults["platform"]) == "windows"
65-
? "${path.module}/templates/userdata_windows.tpl"
66-
: "${path.module}/templates/userdata.sh.tpl"
67-
)
68-
)
69-
70-
vars = merge({
71-
platform = lookup(var.worker_groups[count.index], "platform", local.workers_group_defaults["platform"])
72-
cluster_name = coalescelist(aws_eks_cluster.this[*].name, [""])[0]
73-
endpoint = coalescelist(aws_eks_cluster.this[*].endpoint, [""])[0]
74-
cluster_auth_base64 = coalescelist(aws_eks_cluster.this[*].certificate_authority[0].data, [""])[0]
75-
pre_userdata = lookup(
76-
var.worker_groups[count.index],
77-
"pre_userdata",
78-
local.workers_group_defaults["pre_userdata"],
79-
)
80-
additional_userdata = lookup(
81-
var.worker_groups[count.index],
82-
"additional_userdata",
83-
local.workers_group_defaults["additional_userdata"],
84-
)
85-
bootstrap_extra_args = lookup(
86-
var.worker_groups[count.index],
87-
"bootstrap_extra_args",
88-
local.workers_group_defaults["bootstrap_extra_args"],
89-
)
90-
kubelet_extra_args = lookup(
91-
var.worker_groups[count.index],
92-
"kubelet_extra_args",
93-
local.workers_group_defaults["kubelet_extra_args"],
94-
)
95-
},
96-
lookup(
97-
var.worker_groups[count.index],
98-
"userdata_template_extra_args",
99-
local.workers_group_defaults["userdata_template_extra_args"]
100-
)
101-
)
102-
}
103-
104-
data "template_file" "launch_template_userdata" {
105-
count = var.create_eks ? local.worker_group_launch_template_count : 0
106-
template = lookup(
107-
var.worker_groups_launch_template[count.index],
108-
"userdata_template_file",
109-
file(
110-
lookup(var.worker_groups_launch_template[count.index], "platform", local.workers_group_defaults["platform"]) == "windows"
111-
? "${path.module}/templates/userdata_windows.tpl"
112-
: "${path.module}/templates/userdata.sh.tpl"
113-
)
114-
)
115-
116-
vars = merge({
117-
platform = lookup(var.worker_groups_launch_template[count.index], "platform", local.workers_group_defaults["platform"])
118-
cluster_name = coalescelist(aws_eks_cluster.this[*].name, [""])[0]
119-
endpoint = coalescelist(aws_eks_cluster.this[*].endpoint, [""])[0]
120-
cluster_auth_base64 = coalescelist(aws_eks_cluster.this[*].certificate_authority[0].data, [""])[0]
121-
pre_userdata = lookup(
122-
var.worker_groups_launch_template[count.index],
123-
"pre_userdata",
124-
local.workers_group_defaults["pre_userdata"],
125-
)
126-
additional_userdata = lookup(
127-
var.worker_groups_launch_template[count.index],
128-
"additional_userdata",
129-
local.workers_group_defaults["additional_userdata"],
130-
)
131-
bootstrap_extra_args = lookup(
132-
var.worker_groups_launch_template[count.index],
133-
"bootstrap_extra_args",
134-
local.workers_group_defaults["bootstrap_extra_args"],
135-
)
136-
kubelet_extra_args = lookup(
137-
var.worker_groups_launch_template[count.index],
138-
"kubelet_extra_args",
139-
local.workers_group_defaults["kubelet_extra_args"],
140-
)
141-
},
142-
lookup(
143-
var.worker_groups_launch_template[count.index],
144-
"userdata_template_extra_args",
145-
local.workers_group_defaults["userdata_template_extra_args"]
146-
)
147-
)
148-
}
149-
15031
data "aws_iam_role" "custom_cluster_iam_role" {
15132
count = var.manage_cluster_iam_resources ? 0 : 1
15233
name = var.cluster_iam_role_name
15334
}
15435

155-
data "aws_iam_instance_profile" "custom_worker_group_iam_instance_profile" {
156-
count = var.manage_worker_iam_resources ? 0 : local.worker_group_count
157-
name = lookup(
158-
var.worker_groups[count.index],
159-
"iam_instance_profile_name",
160-
local.workers_group_defaults["iam_instance_profile_name"],
161-
)
162-
}
163-
164-
data "aws_iam_instance_profile" "custom_worker_group_launch_template_iam_instance_profile" {
165-
count = var.manage_worker_iam_resources ? 0 : local.worker_group_launch_template_count
166-
name = lookup(
167-
var.worker_groups_launch_template[count.index],
168-
"iam_instance_profile_name",
169-
local.workers_group_defaults["iam_instance_profile_name"],
170-
)
171-
}
172-
17336
data "aws_partition" "current" {}

‎docs/faq.md‎

Lines changed: 3 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
## How do I customize X on the worker group's settings?
44

5-
All the options that can be customized for worker groups are listed in [local.tf](https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/local.tf) under `workers_group_defaults_defaults`.
5+
All the options that can be customized for worker groups are listed in [local.tf](https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/modules/worker_groups/local.tf) under `workers_group_defaults_defaults`.
66

77
Please open Issues or PRs if you think something is missing.
88

@@ -61,12 +61,6 @@ You need to add the tags to the VPC and subnets yourself. See the [basic example
6161

6262
An alternative is to use the aws provider's [`ignore_tags` variable](https://www.terraform.io/docs/providers/aws/#ignore\_tags-configuration-block). However this can also cause terraform to display a perpetual difference.
6363

64-
## How do I safely remove old worker groups?
65-
66-
You've added new worker groups. Deleting worker groups from earlier in the list causes Terraform to want to recreate all worker groups. This is a limitation with how Terraform works and the module using `count` to create the ASGs and other resources.
67-
68-
The safest and easiest option is to set `asg_min_size` and `asg_max_size` to 0 on the worker groups to "remove".
69-
7064
## Why does changing the worker group's desired count not do anything?
7165

7266
The module is configured to ignore this value. Unfortunately Terraform does not support variables within the `lifecycle` block.
@@ -77,9 +71,9 @@ You can change the desired count via the CLI or console if you're not using the
7771

7872
If you are not using autoscaling and really want to control the number of nodes via terraform then set the `asg_min_size` and `asg_max_size` instead. AWS will remove a random instance when you scale down. You will have to weigh the risks here.
7973

80-
## Why are nodes not recreated when the `launch_configuration`/`launch_template` is recreated?
74+
## Why are nodes not recreated when the `launch_configuration` is recreated?
8175

82-
By default the ASG is not configured to be recreated when the launch configuration or template changes. Terraform spins up new instances and then deletes all the old instances in one go as the AWS provider team have refused to implement rolling updates of autoscaling groups. This is not good for kubernetes stability.
76+
By default the ASG is not configured to be recreated when the launch configuration changes. Terraform spins up new instances and then deletes all the old instances in one go as the AWS provider team have refused to implement rolling updates of autoscaling groups. This is not good for kubernetes stability.
8377

8478
You need to use a process to drain and cycle the workers.
8579

‎docs/spot-instances.md‎

Lines changed: 5 additions & 43 deletions
Original file line numberDiff line numberDiff line change
@@ -22,65 +22,27 @@ Notes:
2222
- There is an AWS blog article about this [here](https://aws.amazon.com/blogs/compute/run-your-kubernetes-workloads-on-amazon-ec2-spot-instances-with-amazon-eks/).
2323
- Consider using [k8s-spot-rescheduler](https://github.com/pusher/k8s-spot-rescheduler) to move pods from on-demand to spot instances.
2424

25-
## Using Launch Configuration
26-
27-
Example worker group configuration that uses an ASG with launch configuration for each worker group:
28-
29-
```hcl
30-
worker_groups = [
31-
{
32-
name = "on-demand-1"
33-
instance_type = "m4.xlarge"
34-
asg_max_size = 1
35-
kubelet_extra_args = "--node-labels=node.kubernetes.io/lifecycle=normal"
36-
suspended_processes = ["AZRebalance"]
37-
},
38-
{
39-
name = "spot-1"
40-
spot_price = "0.199"
41-
instance_type = "c4.xlarge"
42-
asg_max_size = 20
43-
kubelet_extra_args = "--node-labels=node.kubernetes.io/lifecycle=spot"
44-
suspended_processes = ["AZRebalance"]
45-
},
46-
{
47-
name = "spot-2"
48-
spot_price = "0.20"
49-
instance_type = "m4.xlarge"
50-
asg_max_size = 20
51-
kubelet_extra_args = "--node-labels=node.kubernetes.io/lifecycle=spot"
52-
suspended_processes = ["AZRebalance"]
53-
}
54-
]
55-
```
56-
5725
## Using Launch Templates
5826

5927
Launch Template support is a recent addition to both AWS and this module. It might not be as tried and tested but it's more suitable for spot instances as it allowed multiple instance types in the same worker group:
6028

6129
```hcl
62-
worker_groups = [
63-
{
64-
name = "on-demand-1"
30+
worker_groups = {
31+
on-demand-1 = {
6532
instance_type = "m4.xlarge"
6633
asg_max_size = 10
6734
kubelet_extra_args = "--node-labels=spot=false"
6835
suspended_processes = ["AZRebalance"]
69-
}
70-
]
71-
72-
73-
worker_groups_launch_template = [
74-
{
75-
name = "spot-1"
36+
},
37+
spot-1 = {
7638
override_instance_types = ["m5.large", "m5a.large", "m5d.large", "m5ad.large"]
7739
spot_instance_pools = 4
7840
asg_max_size = 5
7941
asg_desired_capacity = 5
8042
kubelet_extra_args = "--node-labels=node.kubernetes.io/lifecycle=spot"
8143
public_ip = true
8244
},
83-
]
45+
}
8446
```
8547

8648
## Important Notes

‎examples/basic/main.tf‎

Lines changed: 4 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -135,22 +135,20 @@ module "eks" {
135135

136136
vpc_id = module.vpc.vpc_id
137137

138-
worker_groups = [
139-
{
140-
name = "worker-group-1"
138+
worker_groups = {
139+
worker-group-1 = {
141140
instance_type = "t2.small"
142141
additional_userdata = "echo foo bar"
143142
asg_desired_capacity = 2
144143
additional_security_group_ids = [aws_security_group.worker_group_mgmt_one.id]
145144
},
146-
{
147-
name = "worker-group-2"
145+
worker-group-2 = {
148146
instance_type = "t2.medium"
149147
additional_userdata = "echo foo bar"
150148
additional_security_group_ids = [aws_security_group.worker_group_mgmt_two.id]
151149
asg_desired_capacity = 1
152150
},
153-
]
151+
}
154152

155153
worker_additional_security_group_ids = [aws_security_group.all_worker_mgmt.id]
156154
map_roles = var.map_roles

0 commit comments

Comments
 (0)