Support target_group_arns for managed node groups

[DmitriyStoyanov]

Is your request related to a new offering from AWS?

This feature request opened in aws provider for node groups
aws/containers-roadmap#709

but separate part can be added in module to support this feature

Is your request related to a problem? Please describe.

For the moment, we used nginx ingress controller and worker groups attached to alb via property target_group_arns
(A list of Application LoadBalancer (ALB) target group ARNs to be associated to the autoscaling group)

When we added managed node groups we found that alb target group does not contain instances from managed node groups. So ingress cannot work on these instances.

Describe the solution you'd like.

option as a target_group_arns for node_groups

Describe alternatives you've considered.

Currently to mitigate this issue we created separate resource (thanks to aws/containers-roadmap#709 (comment) ) :

resource "aws_autoscaling_attachment" "autoscaling_attachment" {
  for_each = {for name, value in module.eks.node_groups : name => lookup(lookup(lookup(value, "resources")[0], "autoscaling_groups")[0], "name")}
  autoscaling_group_name = each.value
  alb_target_group_arn   = aws_lb_target_group.alb.arn
}

So this part can be added in node_group submodule

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[daroga0002]

why you cannot use normal service Type=Loadbalancer or something like https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.2/guide/service/nlb/ ?

[DmitriyStoyanov]

From my case, I had previously created LB for nginx ingress for whole cluster and it was attached to ALB group via target_group_arns. But when I started to add managed nodegroups, faced, that this balancer cannot be attached to these node groups because of unavailability of such parameter in node group submodule. So it is added as separate aws_autoscaling_attachment terraform resource. If it will be added to submodule, it can be used as the same parameter directly during module description.

[daroga0002]

I really dont understand why you cannot use cloud provider or mentioned https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.2/guide/service/nlb/ ?

[DmitriyStoyanov]

Because ALB already exist in separate terraform module, with management of Security groups, WAF, Certificates, etc... in separate module.
And by the way, one of our cluster in China, so you cannot just recreate your LB, if you already pinned it with static IP's and applied ICP License for that stuff.

[jjhidalgar]

@daroga0002 Many of us don't use service type "LoadBalancer" because we prefer to use terraform when creating infrastructure. That way we can manage security groups, dns, and so on in a much controlled way than when kubernetes creates it automatically.

That said I use this without trouble

resource "aws_autoscaling_attachment" "node_groups--ELB_NAME" {
for_each = module.eks.node_groups

autoscaling_group_name = lookup(lookup(lookup(each.value, "resources")[0], "autoscaling_groups")[0], "name")
elb = aws_elb.elb_ELB_NAME.id
}

[DmitriyStoyanov]

From my side I already solved this issue by creation of

resource "aws_autoscaling_attachment" "autoscaling_attachment" {
  for_each = {for name, value in module.eks.node_groups : name => lookup(lookup(lookup(value, "resources")[0], "autoscaling_groups")[0], "name")}
  autoscaling_group_name = each.value
  alb_target_group_arn   = aws_lb_target_group.alb.arn
}

And just created this issue, to possible add this ability in the module as we already have for asg groups.

Or possible if someone also will face with the same issue, the same approach can be used.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[kkapoor1987]

@max-rocket-internet @antonbabenko can we pass target_group_arns to LTs for NodeGroups. This would provide similar behavior as we have for ASGs where we expose target_group_arns as input param and that to LTs
Just like this
#219 (comment)

[daroga0002]

I am still sceptical for this approach as this is not best pattern to follow. If we want create such resources then we should rather follow this approach https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.2/guide/targetgroupbinding/targetgroupbinding/

[kkapoor1987]

@daroga0002 We have ALB created by different TF module and MNG creation creates ASGs which would not have target_group_arns as in self-managed ASGs. I am not sure why we have to treat MNG ASGs differently than self-managed ASG where module allows to pass target_group_arns and set it in LT