Kubernetes provider 2.6.0 breaks EKS creation

[jabbrwcky]

Description

I ran into the issue that Terraform fails to apply the calculated plan to set up EKS and import it into Rancher on a new project without existing state.

Plan fails with an error where the kubernetes provider (released on 2021-10-19) complains about an nonexisting context.

After some experimenting it turns out that the issue can be avoided by using kubernetes provider in Vaersion <2.6.0.

I am not sure whether it is an issue of the EKS module or in kubernetes provider.

Versions

• Terraform: 1.0.9
• Provider(s):

.
├── provider[registry.terraform.io/gavinbunney/kubectl] ~> 1.7
├── provider[registry.terraform.io/hashicorp/aws] ~> 3.15
├── provider[registry.terraform.io/rancher/rancher2] < 1.20.0
├── provider[registry.terraform.io/gitlabhq/gitlab] ~> 3.1
├── provider[registry.terraform.io/hashicorp/helm] ~> 2.2
├── provider[registry.terraform.io/hashicorp/random] ~> 3.0
├── provider[registry.terraform.io/hashicorp/kubernetes] ~> 2.0
├── module.eks
│   ├── provider[registry.terraform.io/hashicorp/aws] >= 3.44.0
│   ├── provider[registry.terraform.io/rancher/rancher2] >= 1.12.0
│   ├── provider[registry.terraform.io/hashicorp/kubernetes]
│   ├── module.vpc
│   │   └── provider[registry.terraform.io/hashicorp/aws] >= 3.63.0
│   └── module.eks
│       ├── provider[registry.terraform.io/hashicorp/cloudinit] >= 2.0.0
│       ├── provider[registry.terraform.io/terraform-aws-modules/http] >= 2.4.1
│       ├── provider[registry.terraform.io/hashicorp/aws] >= 3.56.0
│       ├── provider[registry.terraform.io/hashicorp/local] >= 1.4.0
│       ├── provider[registry.terraform.io/hashicorp/kubernetes] >= 1.11.1
│       ├── module.fargate
│           └── provider[registry.terraform.io/hashicorp/aws] >= 3.40.0
│       └── module.node_groups
│           ├── provider[registry.terraform.io/hashicorp/aws] >= 3.56.0
│           └── provider[registry.terraform.io/hashicorp/cloudinit] >= 2.0.0
└── module.subnets

Reproduction

Steps to reproduce the behavior:

Createan EKS cluster with enabled creation of aws-auth using the latest versions of the EKS module and kubernetes provider.

Code Snippet to Reproduce

module "eks" {
  source                                = "terraform-aws-modules/eks/aws"
  version                               = "~> 17.0"
  cluster_name                          = var.cluster_name
  cluster_version                       = var.cluster_version
  subnets                               = module.vpc.private_subnets
  vpc_id                                = module.vpc.vpc_id
  enable_irsa                           = true
  write_kubeconfig                      = true
  kubeconfig_output_path                = ".kube/config"
  cluster_enabled_log_types             = var.cluster_enabled_log_types
  cluster_log_retention_in_days         = var.cluster_log_retention
  manage_aws_auth                       = true
  worker_create_initial_lifecycle_hooks = true
  map_roles = ...
  node_groups = ...
}

(Omitted in the snippet is the map_roles and node_groups parameter values)

Expected behavior

Actual behavior

Terminal Output Screenshot(s)

 tg plan -out tf.plan
╷
│ Error: Provider configuration: cannot load Kubernetes client config
│
│   with module.eks.provider["registry.terraform.io/hashicorp/kubernetes"],
│   on .terraform/modules/eks/main.tf line 69, in provider "kubernetes":
│   69: provider "kubernetes" {
│
│ context "eks" does not exist
╵
╷
│ Error: error reading EKS Cluster (eks): couldn't find resource
│
│   with module.eks.module.eks.module.node_groups.data.aws_eks_cluster.default[0],
│   on .terraform/modules/eks.eks/modules/node_groups/locals.tf line 1, in data "aws_eks_cluster" "default":
│    1: data "aws_eks_cluster" "default" {
│
╵
ERRO[0018] 1 error occurred:
	* exit status 1

Additional context

Fixing kubernetes provider to <2.6.0 fixes the issue:

Providers required by configuration:
.
├── provider[registry.terraform.io/gavinbunney/kubectl] ~> 1.7
├── provider[registry.terraform.io/hashicorp/aws] ~> 3.15
├── provider[registry.terraform.io/rancher/rancher2] < 1.20.0
├── provider[registry.terraform.io/gitlabhq/gitlab] ~> 3.1
├── provider[registry.terraform.io/hashicorp/helm] ~> 2.2
├── provider[registry.terraform.io/hashicorp/random] ~> 3.0
├── provider[registry.terraform.io/hashicorp/kubernetes] ~> 2.0, < 2.6.0
├── module.eks
│   ├── provider[registry.terraform.io/hashicorp/aws] >= 3.44.0
│   ├── provider[registry.terraform.io/rancher/rancher2] >= 1.12.0
│   ├── provider[registry.terraform.io/hashicorp/kubernetes]
│   ├── module.vpc
│   │   └── provider[registry.terraform.io/hashicorp/aws] >= 3.63.0
│   └── module.eks
│       ├── provider[registry.terraform.io/hashicorp/cloudinit] >= 2.0.0
│       ├── provider[registry.terraform.io/terraform-aws-modules/http] >= 2.4.1
│       ├── provider[registry.terraform.io/hashicorp/aws] >= 3.56.0
│       ├── provider[registry.terraform.io/hashicorp/local] >= 1.4.0
│       ├── provider[registry.terraform.io/hashicorp/kubernetes] >= 1.11.1
│       ├── module.fargate
│           └── provider[registry.terraform.io/hashicorp/aws] >= 3.40.0
│       └── module.node_groups
│           ├── provider[registry.terraform.io/hashicorp/aws] >= 3.56.0
│           └── provider[registry.terraform.io/hashicorp/cloudinit] >= 2.0.0
└── module.subnets

[daroga0002]

could you paste terraform version for problematic run?

[daroga0002]

seems be related to hashicorp/terraform-provider-kubernetes#1469

[daroga0002]

and can be potentially related to hashicorp/terraform-provider-kubernetes#1467

[daroga0002]

Looks they released 2.6.1 (there is just a tag without github release) so you can try it

[jabbrwcky]

Issue unfornuately still persists with 2.6.1, but it already fails on plan:

╷
│ Error: Provider configuration: cannot load Kubernetes client config
│
│   with module.eks.provider["registry.terraform.io/hashicorp/kubernetes"],
│   on .terraform/modules/eks/main.tf line 69, in provider "kubernetes":
│   69: provider "kubernetes" {
│
│ context "eks" does not exist
╵
ERRO[0017] 1 error occurred:
	* exit status 1

terraform version
Terraform v1.0.9
on darwin_amd64
+ provider registry.terraform.io/gavinbunney/kubectl v1.13.0
+ provider registry.terraform.io/gitlabhq/gitlab v3.7.0
+ provider registry.terraform.io/hashicorp/aws v3.63.0
+ provider registry.terraform.io/hashicorp/cloudinit v2.2.0
+ provider registry.terraform.io/hashicorp/helm v2.3.0
+ provider registry.terraform.io/hashicorp/kubernetes v2.6.1
+ provider registry.terraform.io/hashicorp/local v2.1.0
+ provider registry.terraform.io/hashicorp/random v3.1.0
+ provider registry.terraform.io/rancher/rancher2 v1.17.2
+ provider registry.terraform.io/terraform-aws-modules/http v2.4.1

[daroga0002]

please paste how you initialize kubernetes provider in terraform and check does you have in kube config context context "eks" does not exist

[jabbrwcky]

Apart from the module invocation I use the code from the Module documentation:

data "aws_eks_cluster" "eks" {
  name = module.eks.cluster_id
}

data "aws_eks_cluster_auth" "eks" {
  name = module.eks.cluster_id
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.eks.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.eks.certificate_authority[0].data)
  token                  = data.aws_eks_cluster_auth.eks.token
}

module "eks" {
 ...
}

[daroga0002]

So I tested this and using following providers:

Initializing provider plugins...
- Reusing previous version of hashicorp/kubernetes from the dependency lock file
- Reusing previous version of hashicorp/cloudinit from the dependency lock file
- Reusing previous version of terraform-aws-modules/http from the dependency lock file
- Reusing previous version of hashicorp/aws from the dependency lock file
- Reusing previous version of hashicorp/local from the dependency lock file
- Reusing previous version of hashicorp/random from the dependency lock file
- Using previously-installed terraform-aws-modules/http v2.4.1
- Using previously-installed hashicorp/aws v3.63.0
- Using previously-installed hashicorp/local v2.1.0
- Using previously-installed hashicorp/random v3.1.0
- Using previously-installed hashicorp/kubernetes v2.6.1
- Using previously-installed hashicorp/cloudinit v2.2.0

on our example https://github.com/terraform-aws-modules/terraform-aws-eks/tree/master/examples/launch_templates_with_managed_node_groups and no issue observed

where kubernetes provider is initialized:

terraform-aws-eks/examples/launch_templates_with_managed_node_groups/main.tf

Lines 97 to 101 in 967403b

	provider "kubernetes" {
	host = data.aws_eks_cluster.cluster.endpoint
	cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority[0].data)
	token = data.aws_eks_cluster_auth.cluster.token
	}

[jabbrwcky]

Just encountered the same issue again with a different cluster. A downgrade from 2.6.1 to 2.5.1 resolved the issue

[github-actions]

This issue has been automatically marked as stale because it has been open 30 days
with no activity. Remove stale label or comment or this issue will be closed in 10 days

[github-actions]

This issue was automatically closed because of stale in 10 days