aws_eks_node_group: Idempotency broken for empty remote_access block

[eytanhanig]

Note: I am unsure whether this is expected behavior for configuration blocks, however the end result is that you can't conditionally use remote_access and maintain idempotency. If it is expected behavior for configuration blocks then this issue turns into a feature request to have it not be a configuration block.

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

$ terraform -v
Terraform v0.12.16
+ provider.aws v2.40.0

Affected Resource(s)

• aws_eks_node_group

Terraform Configuration Files

This was discovered while helping with the PR to add Support for EKS Managed Node Groups to the module terraform-aws-modules/terraform-aws-eks. Below is a simplified case:

resource "aws_eks_node_group" "workers" {
  node_group_name = "debug"

  cluster_name    = "test-eks-gC3cDC9O"
  node_role_arn   = "arn:aws:iam::REDACTED:role/test-eks-gC3cDC9O-managed-node-groups"
  subnet_ids      = ["subnet-REDACTED1", "subnet-REDACTED2", "subnet-REDACTED3"]

  scaling_config {
    desired_size = "1"
    max_size     = "1"
    min_size     = "1"
  }

  remote_access {}
}

Debug Output

This produces no errors. terraform plan predicts the following and terraform apply implements it without issue:

      + remote_access { # forces replacement}`

Expected Behavior

• After the initial apply terraform plan and terraform apply should be idempotent and predict no changes.

Actual Behavior

Idempotency is broken - and therefore subsequent applies attempt to destroy & recreate the node group - when remote_access is used in the following ways:

remote_access {}

remote_access {
  ec2_ssh_key = ""
}

remote_access {
  ec2_ssh_key = null
}

Please note that this issue does not occur when a key is specificied.

Steps to Reproduce

1. terraform apply --auto-approve
2. terraform plan

Important Factoids

Despite being present in the list of optional arguments remote_apply is a configuration block and behaves as such. The two arguments that it contains are also both listed as optional.

Regardless of whether this is the expected behavior of configuration blocks - and I suspect it isn't - we need a way to make use conditionals with this functionality. Is it possible to use the same key_name and security_groups arguments that we have for launch_configuration, or is using a configuration block necessitated by the AWS API?

[Grejeru]

Looks like removing this line https://github.com/terraform-providers/terraform-provider-aws/blob/baa8f8c81aadcad7a3ad66552f1597e835ae548c/aws/resource_aws_eks_node_group.go#L96 will help if You will have only empty remote_access {} block, but when You define ec2_ssh_key as empty or null has to be tested.

[bflad]

The EKS API Reference denotes the remote access configuration as a separate API structure/object, which maps best in Terraform as a configuration block. We typically do not want to introduce our own abstractions on top of the API as they can easily become invalid or outdated. The EKS API also does not support adding or removing the remote access configuration via in-place update, so the ForceNew: true on the attribute is expected so Terraform can correctly replace the resource to match the configuration.

The Terraform configuration given above:

resource "aws_eks_node_group" "example" {
  # ... other configuration ...

  remote_access {}
}

Would only be valid if the API accepted enabling remote access for EKS Node Groups with some sort of default or automatically generated EC2 Key Pair. If the API does support this type of functionality, we should implement this as a separate feature request.

Terraform 0.12 supports new dynamic blocks, which allows Terraform to optionally create or omit configuration blocks based on an input. In many Terraform providers, with Terraform 0.11 and earlier which did not support dynamic, we did occassionally implement these workarounds of allowing empty argument values (such as ""). The maintainers prefer to not implement that type of extra logic going forward since it introduces ambiguity and abstraction in the provider as compared to the given configuration (such as needing to implement additional difference suppression logic, which will likely not be valid in future versions of Terraform).

To throw together a quick example of this in practice, slighly modifying the repository examples/eks-getting-started with the following configuration:

variable "remote_access_key" {
  default = ""
  type    = string
}

resource "aws_key_pair" "remote_access_key" {
  count = length(var.remote_access_key) > 0 ? 1 : 0

  key_name   = "remote_access"
  public_key = var.remote_access_key
}

resource "aws_eks_node_group" "demo" {
  cluster_name    = aws_eks_cluster.demo.name
  node_group_name = "demo"
  node_role_arn   = aws_iam_role.demo-node.arn
  subnet_ids      = aws_subnet.demo[*].id

  dynamic "remote_access" {
    for_each = aws_key_pair.remote_access_key

    content {
      ec2_ssh_key = remote_access.value.key_name
    }
  }

  scaling_config {
    desired_size = 1
    max_size     = 1
    min_size     = 1
  }

  depends_on = [
    aws_iam_role_policy_attachment.demo-node-AmazonEKSWorkerNodePolicy,
    aws_iam_role_policy_attachment.demo-node-AmazonEKS_CNI_Policy,
    aws_iam_role_policy_attachment.demo-node-AmazonEC2ContainerRegistryReadOnly,
  ]
}

With variable set to default = "":

  # aws_eks_node_group.demo will be created
  + resource "aws_eks_node_group" "demo" {
      + ami_type        = (known after apply)
      + arn             = (known after apply)
      + cluster_name    = "terraform-eks-demo"
      + disk_size       = (known after apply)
      + id              = (known after apply)
      + instance_types  = (known after apply)
      + node_group_name = "demo"
      + node_role_arn   = (known after apply)
      + release_version = (known after apply)
      + resources       = (known after apply)
      + status          = (known after apply)
      + subnet_ids      = (known after apply)
      + version         = (known after apply)

      + scaling_config {
          + desired_size = 1
          + max_size     = 1
          + min_size     = 1
        }
    }

With variable set to default = "...key contents...":

  # aws_eks_node_group.demo will be created
  + resource "aws_eks_node_group" "demo" {
      + ami_type        = (known after apply)
      + arn             = (known after apply)
      + cluster_name    = "terraform-eks-demo"
      + disk_size       = (known after apply)
      + id              = (known after apply)
      + instance_types  = (known after apply)
      + node_group_name = "demo"
      + node_role_arn   = (known after apply)
      + release_version = (known after apply)
      + resources       = (known after apply)
      + status          = (known after apply)
      + subnet_ids      = (known after apply)
      + version         = (known after apply)

      + remote_access {
          + ec2_ssh_key = "remote_access"
        }

      + scaling_config {
          + desired_size = 1
          + max_size     = 1
          + min_size     = 1
        }
    }

Hopefully this helps get you started on how you can introduce this concept into your configurations. Its certainly possible to implement a more complex dynamic configuration that can also account for handling source security groups as well, but that is outside the scope of this issue. If you are interested in potential feature requests for improving/simplifying these types of configurations in Terraform, please submit an issue following the feature request template in the Terraform repository.

If you're looking for general assistance beyond the above, please note that we use GitHub issues in this repository for tracking bugs and enhancements with the Terraform AWS Provider codebase rather than for questions. While we may be able to help with certain simple problems here it's generally better to use the community forums where there are far more people ready to help, whereas the GitHub issues here are generally monitored only by a few maintainers and dedicated community members interested in code development of the Terraform AWS Provider itself.

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!