Error during EKS Creation: Post "http://localhost/api/v1/namespaces/kube-system/configmaps": dial tcp 127.0.0.1:80: connect: connection refused

[lbornov2]

Description

When creating an EKS Cluster using terraform, we get the following error:

Error: Post "http://localhost/api/v1/namespaces/kube-system/configmaps": dial tcp 127.0.0.1:80: connect: connection refused

  on .terraform/modules/deployment.eks/aws_auth.tf line 65, in resource "kubernetes_config_map" "aws_auth":
  65: resource "kubernetes_config_map" "aws_auth" {

To fix this, we have to manually run:

aws eks update-kubeconfig --name ${var.context.app_name} --region ${var.context.region}

and then:

terraform apply -auto-approve

Versions

• Terraform: 0.12.21
• Provider(s):

• aws - 3.22.0
• terraform-aws-modules/eks/aws - 14.0.0
• terraform-aws-modules/vpc/aws - 2.61.0

• AWS CLI: 2.0.30
• Helm: 3.3.4
• Kubectl: 1.19.0

Reproduction

1. Run the terraform code in the Code Snippet to Reproduce section
2. Run terraform init && terraform apply -auto-approve
3. You will get the error in the description.

To fix, manually run:

1. aws eks update-kubeconfig --name ${var.context.app_name} --region ${var.context.region}
2. terraform apply -auto-approve

Code Snippet to Reproduce

terraform {
  required_version = ">= 0.12.21"
}

provider "aws" {
  version = "~> 3.22.0"
  region  = "${var.context.region}"
}


data "aws_availability_zones" "available" {}

resource "random_string" "suffix" {
  length  = 8
  special = false
}

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "2.61.0"

  name                 = "${var.context.app_name}"
  cidr                 = "10.0.0.0/16"
  azs                  = data.aws_availability_zones.available.names
  private_subnets      = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]
  public_subnets       = ["10.0.4.0/24", "10.0.5.0/24", "10.0.6.0/24"]
  enable_nat_gateway   = true
  single_nat_gateway   = true
  enable_dns_hostnames = true

  tags = {
    "kubernetes.io/cluster/${var.context.app_name}" = "shared"
  }

  public_subnet_tags = {
    "kubernetes.io/cluster/${var.context.app_name}" = "shared"
    "kubernetes.io/role/elb"                      = "1"
  }

  private_subnet_tags = {
    "kubernetes.io/cluster/${var.context.app_name}" = "shared"
    "kubernetes.io/role/internal-elb"             = "1"
  }
}

resource "aws_security_group" "all_worker_mgmt" {
  name_prefix = "${var.context.app_name}-all_worker_management"
  vpc_id      = "${module.vpc.vpc_id}"

  ingress {
    from_port = 22
    to_port   = 22
    protocol  = "tcp"

    cidr_blocks = [
      "10.0.0.0/8",
      "172.16.0.0/12",
      "192.168.0.0/16",
    ]
  }
}

module "eks" {
  source                               = "terraform-aws-modules/eks/aws"
  version                              = "14.0.0"
  cluster_name                         = "${var.context.app_name}"
  cluster_version                      = "1.19"
  subnets                              = "${module.vpc.private_subnets}"
  vpc_id                               = "${module.vpc.vpc_id}"
  cluster_create_timeout               = "30m"
  worker_groups = [
    {
      instance_type = "${var.context.kubernetes.aws.machine_type}"
      asg_desired_capacity = "${var.context.replica_count}"
      asg_min_size = "${var.context.replica_count}"
      asg_max_size  = "${var.context.replica_count}"
      root_volume_type = "gp2"
    }
  ]
  worker_additional_security_group_ids = ["${aws_security_group.all_worker_mgmt.id}"]
  map_users = var.context.iam.aws.map_users
  map_roles = var.context.iam.aws.map_roles
}

Expected behavior

The cluster gets created successfully

Actual behavior

We get this output:

Error: Post "http://localhost/api/v1/namespaces/kube-system/configmaps": dial tcp 127.0.0.1:80: connect: connection refused

  on .terraform/modules/deployment.eks/aws_auth.tf line 65, in resource "kubernetes_config_map" "aws_auth":
  65: resource "kubernetes_config_map" "aws_auth" {

[dak1n1]

It looks like the Kubernetes provider isn't receiving a configuration. Here's how I configure mine, which is similar to the EKS module README except it's for the newer version of the Kubernetes provider. (My team recently released version 2.0 of the Kubernetes provider and it requires a slightly different config than shown in this module's README).

data "aws_eks_cluster" "default" {
  name = module.cluster.cluster_id
}

data "aws_eks_cluster_auth" "default" {
  name = module.cluster.cluster_id
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.default.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.default.certificate_authority[0].data)
  token                  = data.aws_eks_cluster_auth.default.token
}

[lbornov2]

@dak1n1 - The code example I provided does NOT instantiate the kubernetes provider (at least not directly). The error happens during creation of the EKS module - not before or after. So - the problem happens in the EKS module itself - not outside of the module.

[dak1n1]

Right, but the EKS module uses the Kubernetes provider under the hood, so it needs a provider configuration. Otherwise, it will assume a default/empty config. You can skip using the Kubernetes provider within the EKS module by specifying manage_aws_auth = false in your EKS module config. That will skip the section that relies on the Kubernetes provider.

module "cluster" {
  source  = "terraform-aws-modules/eks/aws"
  version = "14.0.0" 
...
  manage_aws_auth  = false
...
}

[ArchiFleKs]

I have the same issue on a newly created cluster everything is fine and then I'm changing the module tag and get this error, the weird thing is that if I put everything back just like in the state everything work as expected

[RobertFischer]

@dak1n1 -- How am I supposed to instantiate the Kubernetes provider using output from this module, but before I instantiate this module?

(Also, is everyone who specifies manage_aws_auth = true experiencing this issue? If not, what's different that allows them to avoid it? In the alternative, if we are all experiencing this issue, then isn't manage_aws_auth = true straight-up broke?)

[dak1n1]

@dak1n1 -- How am I supposed to instantiate the Kubernetes provider using output from this module, but before I instantiate this module?

(Also, is everyone who specifies manage_aws_auth = true experiencing this issue? If not, what's different that allows them to avoid it? In the alternative, if we are all experiencing this issue, then isn't manage_aws_auth = true straight-up broke?)

The issues vary depending on the user's configuration, but they're all related to a subject I'm currently researching, which is why I'm volunteering my time here. The EKS module, which is a community-driven effort (not managed by Hashicorp), specifically is using a pattern that is discouraged by the creators of Terraform. (Specifically, where it says a provider config should only reference values that are known before the configuration is applied, that's the issue we're hitting here).

The issues being described in this bug report all have the same root cause, which is that they are passing variables into a provider configuration which are not known at plan time. Terraform simply doesn't support that, and so they encourage instead separating out the AWS provider and Kubernetes provider resources, so you can use two applies when needed. However, there are some work-arounds that can still help to achieve this workflow. Since so many users want to use this in a single apply, this is my area of interest, to try and enable that pattern to succeed, despite the current limitations in Terraform.

TLDR: you can copy/paste the config I gave above, which will only read the EKS cluster after the variables are known. You can also use the config in this repo's README, which is another way to establish this dependency.

However, since this pattern is not actually supported in Terraform, there will be times when it will fail. Specifically, when the EKS cluster's credentials become unknown, such as when replacing the cluster, or during destroy on Terraform 0.14.x. To avoid these errors, there are work-arounds such as terraform refresh prior to destroy, and removing the kubernetes config map from state prior to replacing/modifying the EKS cluster (I believe that would look something like terraform state rm module.cluster.kubernetes_config_map.aws-auth, but I don't know what impact that would have on the EKS worker nodes).

This page can be helpful for learning about configuring providers that are used with modules. https://www.terraform.io/docs/language/modules/develop/providers.html

I also have a couple working configs here, if anyone wants to reference them.

On the Kubernetes provider side, I have some plans to smooth this out a bit and provide more meaningful error messages. I'm also planning to implement a version of this old request that I found when researching the topic. That will allow the Kubernetes provider to keep trying to contact the Kubernetes API, rather than failing immediately with the obscure localhost error we all see. So there are some changes hopefully coming in the next few months.

[RobertFischer]

Thanks for the thorough response, Stef. I appreciate it. Ultimately, since my `eks` module is called within another module, the provider solution just doesn’t work out due to the limitation against nesting providers within modules. I’m taking a look at cdktf and seeing if it gives me the flexibility that I need.

[jgournet]

it probably won't apply to many people, but in case that can help: I had this config:

provider "kubernetes" {
  host                   = data.aws_eks_cluster.this.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.this.certificate_authority[0].data)
  config_context         = data.aws_eks_cluster.this.arn
  token                  = data.aws_eks_cluster_auth.this.token
}

Somehow, removing the "config_context" line made this error disappear ...

[charneykaye]

I have never set manage_aws_auth yet I experienced this issue after I had changed the name of my eks module.

The resolution was

• use terraform state list in order to discover the resources that were stored under the legacy module name
• use terraform state rm to remove the resources stored under the legacy module name
• use terraform import to re-import the module's resources under the new name

[stevehipwell]

@dak1n1 this issue is being caused by hashicorp/terraform#24886 (which is also a pretty large pain point in implementing a Hashicorp native credential flow e.g. OIDC -> Vault STS -> AWS provider). Without this being solved what is the logic for a aws_eks_cluster_auth not to be created until something else has been configured? The current "best practice" pattern errors when you plan and don't apply for over 15 mins; it also errors when you've layered other code on top of this module and a control plane or managed worker change take over 15 mins to complete.

TL;DR - Is there a hack to make a aws_eks_cluster_auth resource wait for something else before being calculated so we can target these to be ready when they're not going to expire?

[ArchiFleKs]

@stevehipwell Do you think this might be related to the issue you mentioned above ?

[stevehipwell]

@ArchiFleKs I suspect that it's related and I meant to add a comment to that effect. We're having to delete the aws_auth config map from state when we destroy a cluster, see #1280 (comment).

[ArchiFleKs]

@stevehipwell Yes my workaround is:

• remove aws_auth from state
• apply with manage_aws_auth=false
• import configmap aws_auth to state
• apply with manage_aws_auth=true

[stevehipwell]

@ArchiFleKs have you tried reverting the Kubernetes provider version?