AWS Auth ConfigMap Changes not Applied Due To local_exec Race Condition

[knittingdev]

I have issues

With manage_aws_auth=true, changes were not propagated to the aws-auth ConfigMap

I'm submitting a...

• bug report
• feature request
• support request - read the FAQ first!
• kudos, thank you, warm fuzzy

What is the current behavior?

Terraform identifies a change to your aws-auth Configmap during a plan. Example below:

  # module.eks.module.eks.null_resource.update_config_map_aws_auth[0] must be replaced
-/+ resource "null_resource" "update_config_map_aws_auth" {
      ~ id       = "5095510880687263162" -> (known after apply)
      ~ triggers = { # forces replacement
          ~ "config_map_rendered"      = <<~EOT
                apiVersion: v1
                kind: ConfigMap
                metadata:
                  name: aws-auth
                  namespace: kube-system
                data:
                  mapRoles: |
                    - rolearn: [REDACTED]
                      username: system:node:{{EC2PrivateDNSName}}
                      groups:
                        - system:bootstrappers
                        - system:nodes
                
                  
                    - "groups":
                      - "system:masters"
                      "rolearn": "[REDACTED]"
                      "username": "software_dev"
                    - "groups":
                      - "system:masters"
                      "rolearn": "[REDACTED]"
                      "username": "staging-ok8s-20190411194656305200000004"
                    - "groups":
                      - "ok8s-dispatcher"
                      - "airflow-deployer"
              +       - "bulldozer-deployer"
                      "rolearn": "[REDACTED]"
                      "username": "ci"
                    
                  
                  
                  
            EOT
            "endpoint"                 = "[REDACTED]"
            "kube_config_map_rendered" = "apiVersion: v1\npreferences: {}\nkind: Config\n\nclusters:\n- cluster:\n    server: [REDACTED]\n    certificate-authority-data: [REDACTED]  name: eks_eks-staging\n\ncontexts:\n- context:\n    cluster: eks_eks-staging\n    user: eks_eks-staging\n  name: eks_eks-staging\n\ncurrent-context: eks_eks-staging\n\nusers:\n- name: eks_eks-staging\n  user:\n    exec:\n      apiVersion: client.authentication.k8s.io/v1alpha1\n      command: aws\n      args:\n        - eks\n        - get-token\n        - --cluster-name\n        - eks-staging\n\n\n"
        }
    }

But upon apply, the local_exec command experiences an issue but does not exit with an error. Example below:

module.eks.module.eks.null_resource.update_config_map_aws_auth[0] (local-exec): error: unable to recognize "aws_auth_configmap.yaml": Unauthorized
module.eks.module.eks.null_resource.update_config_map_aws_auth[0]: Still creating... [10s elapsed]
module.eks.module.eks.null_resource.update_config_map_aws_auth[0] (local-exec): error: unable to recognize "aws_auth_configmap.yaml": Unauthorized
module.eks.module.eks.null_resource.update_config_map_aws_auth[0]: Still creating... [20s elapsed]
module.eks.module.eks.null_resource.update_config_map_aws_auth[0] (local-exec): error: unable to recognize "aws_auth_configmap.yaml": Unauthorized
module.eks.module.eks.null_resource.update_config_map_aws_auth[0]: Still creating... [30s elapsed]
module.eks.module.eks.null_resource.update_config_map_aws_auth[0] (local-exec): error: unable to recognize "aws_auth_configmap.yaml": Unauthorized
module.eks.module.eks.null_resource.update_config_map_aws_auth[0]: Still creating... [40s elapsed]
module.eks.module.eks.null_resource.update_config_map_aws_auth[0] (local-exec): error: unable to recognize "aws_auth_configmap.yaml": Unauthorized
module.eks.module.eks.null_resource.update_config_map_aws_auth[0]: Still creating... [50s elapsed]
module.eks.module.eks.null_resource.update_config_map_aws_auth[0]: Still creating... [1m0s elapsed]
module.eks.module.eks.null_resource.update_config_map_aws_auth[0] (local-exec): error: unable to recognize "aws_auth_configmap.yaml": Unauthorized
module.eks.module.eks.null_resource.update_config_map_aws_auth[0]: Still creating... [1m10s elapsed]
module.eks.module.eks.null_resource.update_config_map_aws_auth[0] (local-exec): error: unable to recognize "aws_auth_configmap.yaml": Unauthorized
module.eks.module.eks.null_resource.update_config_map_aws_auth[0]: Still creating... [1m20s elapsed]
module.eks.module.eks.null_resource.update_config_map_aws_auth[0] (local-exec): error: unable to recognize "aws_auth_configmap.yaml": Unauthorized
module.eks.module.eks.null_resource.update_config_map_aws_auth[0]: Still creating... [1m30s elapsed]
module.eks.module.eks.null_resource.update_config_map_aws_auth[0] (local-exec): error: unable to recognize "aws_auth_configmap.yaml": Unauthorized
module.eks.module.eks.null_resource.update_config_map_aws_auth[0]: Still creating... [1m40s elapsed]
module.eks.module.eks.null_resource.update_config_map_aws_auth[0] (local-exec): error: unable to recognize "aws_auth_configmap.yaml": Unauthorized
module.eks.module.eks.null_resource.update_config_map_aws_auth[0]: Still creating... [1m50s elapsed]
module.eks.module.eks.null_resource.update_config_map_aws_auth[0]: Creation complete after 1m58s [id=9073420618639204542]

If this is a bug, how to reproduce? Please include a code sample if relevant.

Since it appears to be a race condition, it's possible that this might be complicated to reproduce consistently. That said, you should be able to reproduce by making a change to eks_map_roles and attempt a plan and apply.

Here's an partial example from my files

locals {
  eks_map_roles = [
    {
      rolearn  = aws_iam_role.external_software_dev.arn
      username = aws_iam_role.external_software_dev.name
      groups   = ["system:masters"]
    },
    {
      rolearn  = aws_iam_role.external_analyst.arn
      username = aws_iam_role.external_analyst.name
      groups   = ["devflow"]
    },
  ]
}

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "6.0.1"

  cluster_name    = local.eks_cluster_name
  cluster_version = local.eks_cluster_version
  env                     = var.env

  cluster_enabled_log_types = [
    "api",
    "audit",
    "authenticator",
    "controllerManager",
    "scheduler",
  ]

  cluster_log_retention_in_days = 0

  subnets                                   = local.eks_subnet_ids
  tags                                      = local.eks_tags
  vpc_id                                    = local.eks_vpc_id
  manage_aws_auth                           = true
  map_roles                                 = local.eks_map_roles
  map_users                                 = "${local.eks_map_users}"
  worker_groups                             = local.worker_groups
  worker_additional_security_group_ids      = var.worker_additional_security_group_ids
  write_aws_auth_config                     = false
  write_kubeconfig                          = false
  kubeconfig_aws_authenticator_command      = "aws"
  kubeconfig_aws_authenticator_command_args = ["eks", "get-token", "--cluster-name", local.eks_cluster_name]

  workers_group_defaults = {
    root_volume_size = 150
  }
}

What's the expected behavior?

Two things:

1. Errors throw an exit code so that the apply fails. This issue was mentioned already in another issue Missing kubectl or aws-iam-authenticator should abort apply #341
2. aws-auth ConfigMap is updated after a plan and apply

Are you able to fix this problem and submit a PR? Link here if you have already.

Yes, Please see #592

Environment details

• Affected module version: 6.0.1
• OS: Ubuntu 18.04.2 LTS
• Terraform version: v0.12.7

Any other relevant info

[max-rocket-internet]

Resolved in #592

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.