docs: Update Terraform getting started guide to include recent changes #2913
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -20,7 +20,9 @@ Follow the clean-up instructions to reduce any charges. | |
|
|
||
| ## Install | ||
|
|
||
| Karpenter is installed in clusters with a helm chart. In this example, an EKS Managed Node group | ||
| is provisioned to support Karpenter. For running Karpenter on EKS Fargate Profiles, see | ||
| [this example](https://github.com/terraform-aws-modules/terraform-aws-eks/tree/master/examples/karpenter). | ||
|
|
||
| Karpenter additionally requires IAM Roles for Service Accounts (IRSA). IRSA | ||
| permits Karpenter (within the cluster) to make privileged requests to AWS (as | ||
|
|
@@ -33,10 +35,10 @@ Install these tools before proceeding: | |
| 1. [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2-linux.html) | ||
| 2. `kubectl` - [the Kubernetes CLI](https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/) | ||
| 3. `terraform` - [infrastructure-as-code tool made by HashiCorp](https://learn.hashicorp.com/tutorials/terraform/install-cli) | ||
|
|
||
| [Configure the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-quickstart.html) | ||
|
There was a problem hiding this comment. Copied from eksctl getting started guide |
||
| with a user that has sufficient privileges to create an EKS cluster. Verify that the CLI can | ||
| authenticate properly by running `aws sts get-caller-identity`. | ||
|
|
||
| ### Setting up Variables | ||
|
|
||
|
|
@@ -78,54 +80,61 @@ locals { | |
|
|
||
| # Used to determine correct partition (i.e. - `aws`, `aws-gov`, `aws-cn`, etc.) | ||
| partition = data.aws_partition.current.partition | ||
|
|
||
| vpc_cidr = "10.0.0.0/16" | ||
| azs = slice(data.aws_availability_zones.available.names, 0, 3) | ||
| } | ||
|
|
||
| data "aws_partition" "current" {} | ||
| data "aws_availability_zones" "available" {} | ||
| data "aws_ecrpublic_authorization_token" "token" {} | ||
| ``` | ||
|
|
||
| ### Create a Cluster | ||
|
|
||
| We're going to use three different Terraform modules to create our cluster | ||
| 1. [`eks`](https://registry.terraform.io/modules/terraform-aws-modules/eks/aws/latest) which creates the EKS cluster and associated cluster resources | ||
| 2. [`karpenter`](https://registry.terraform.io/modules/terraform-aws-modules/eks/aws/latest/submodules/karpenter) which creates Karpenter IAM role(s), instance profile, SQS queue, and EvnetBridge rules | ||
| 3. [`vpc`](https://registry.terraform.io/modules/terraform-aws-modules/vpc/aws/latest) which creates a VPC suitable for provisioning our cluster | ||
|
|
||
|
|
||
| Add the following to your `main.tf` to create the VPC and EKS cluster. | ||
|
|
||
| ```hcl | ||
| module "vpc" { | ||
| # https://registry.terraform.io/modules/terraform-aws-modules/vpc/aws/latest | ||
| source = "terraform-aws-modules/vpc/aws" | ||
| version = "3.18.1" | ||
|
|
||
| name = local.cluster_name | ||
| cidr = local.vpc_cidr | ||
|
|
||
| azs = local.azs | ||
| private_subnets = [for k, v in local.azs : cidrsubnet(local.vpc_cidr, 4, k)] | ||
| public_subnets = [for k, v in local.azs : cidrsubnet(local.vpc_cidr, 8, k + 48)] | ||
|
|
||
| enable_nat_gateway = true | ||
| single_nat_gateway = true | ||
| enable_dns_hostnames = true | ||
|
|
||
| public_subnet_tags = { | ||
| "kubernetes.io/role/elb" = 1 | ||
| } | ||
|
|
||
| private_subnet_tags = { | ||
| "kubernetes.io/role/internal-elb" = 1 | ||
| # Tags subnets for Karpenter auto-discovery | ||
| "karpenter.sh/discovery" = "true" | ||
| } | ||
| } | ||
|
|
||
| module "eks" { | ||
| # https://registry.terraform.io/modules/terraform-aws-modules/eks/aws/latest | ||
| source = "terraform-aws-modules/eks/aws" | ||
| version = "18.31.0" | ||
|
|
||
| cluster_name = local.cluster_name | ||
| cluster_version = "1.24" | ||
|
|
||
| vpc_id = module.vpc.vpc_id | ||
| subnet_ids = module.vpc.private_subnets | ||
|
|
@@ -148,7 +157,7 @@ module "eks" { | |
| # NOTE - if creating multiple security groups with this module, only tag the | ||
| # security group that Karpenter should utilize with the following tag | ||
| # (i.e. - at most, only one security group should have this tag in your account) | ||
| "karpenter.sh/discovery" = local.cluster_name | ||
| } | ||
|
|
||
| # Only need one node to get Karpenter up and running. | ||
|
|
@@ -164,23 +173,13 @@ module "eks" { | |
| min_size = 2 | ||
| max_size = 3 | ||
| desired_size = 2 | ||
| } | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| At this point, go ahead and apply what we've done to create the VPC and | ||
| EKS cluster. This will take approximately 12-15 minutes to create. | ||
|
|
||
| ```bash | ||
| terraform init | ||
|
|
@@ -197,65 +196,37 @@ aws iam create-service-linked-role --aws-service-name spot.amazonaws.com | |
| # An error occurred (InvalidInput) when calling the CreateServiceLinkedRole operation: Service role name AWSServiceRoleForEC2Spot has been taken in this account, please try a different suffix. | ||
| ``` | ||
|
|
||
| ### Create the Karpenter AWS Resources | ||
|
|
||
| The EKS module creates an IAM role for the EKS managed node group nodes. We'll use that for | ||
| Karpenter (so we don't have to reconfigure the aws-auth ConfigMap for a new role), but we need | ||
| to create an instance profile we can reference. We will utilize the `karpenter` sub-module to | ||
| create the necessary AWS resources required by Karpenter, including those necessary to support | ||
| node termination handling (SQS queue and EventBridge rules). | ||
|
|
||
| Add the following to your `main.tf` to create: | ||
| - AWS IAM instance profile Karpenter will assign to nodes created | ||
| - AWS IAM role for service accounts (IRSA) used by the Karpenter controller | ||
| - AWS SQS queue and EventBridge rules for node termination handling | ||
|
|
||
| ```hcl | ||
| module "karpenter" { | ||
| source = "terraform-aws-modules/eks/aws//modules/karpenter" | ||
| version = "18.31.0" | ||
|
|
||
| cluster_name = module.eks.cluster_name | ||
|
|
||
| irsa_oidc_provider_arn = module.eks.oidc_provider_arn | ||
| irsa_namespace_service_accounts = ["karpenter:karpenter"] | ||
|
|
||
| # Since Karpenter is running on an EKS Managed Node group, | ||
| # we can re-use the role that was created for the node group | ||
| create_iam_role = false | ||
| iam_role_arn = module.eks.eks_managed_node_groups["initial"].iam_role_arn | ||
| } | ||
| ``` | ||
|
|
||
| Go ahead and apply the changes. | ||
|
|
||
| ```bash | ||
| terraform init | ||
|
|
@@ -264,11 +235,10 @@ terraform apply | |
|
|
||
| ### Install Karpenter Helm Chart | ||
|
|
||
| We are going to use the `helm_release` Terraform resource to do the deploy and pass | ||
| in the cluster details and IAM role Karpenter needs to assume. | ||
|
|
||
| Add the following to your `main.tf` to provision Karpenter via its Helm chart. | ||
|
|
||
| ```hcl | ||
| provider "helm" { | ||
|
|
@@ -279,7 +249,7 @@ provider "helm" { | |
| exec { | ||
| api_version = "client.authentication.k8s.io/v1beta1" | ||
| command = "aws" | ||
| args = ["eks", "get-token", "--cluster-name", module.eks.cluster_name] | ||
| } | ||
| } | ||
| } | ||
|
|
@@ -288,29 +258,36 @@ resource "helm_release" "karpenter" { | |
| namespace = "karpenter" | ||
| create_namespace = true | ||
|
|
||
| name = "karpenter" | ||
| repository = "oci://public.ecr.aws/karpenter" | ||
| repository_username = data.aws_ecrpublic_authorization_token.token.user_name | ||
|
There was a problem hiding this comment. uses data source to pull creds to ensure chart can be pulled from ECR |
||
| repository_password = data.aws_ecrpublic_authorization_token.token.password | ||
| chart = "karpenter" | ||
| version = "{{< param "latest_release_version" >}}" | ||
|
|
||
| set { | ||
| name = "settings.aws.clusterName" | ||
|
There was a problem hiding this comment. we will want to rely on Nothing required here other than where most instances of |
||
| value = module.eks.cluster_name | ||
| } | ||
|
|
||
| set { | ||
| name = "settings.aws.clusterEndpoint" | ||
| value = module.eks.cluster_endpoint | ||
| } | ||
|
|
||
| set { | ||
| name = "serviceAccount.annotations.eks\\.amazonaws\\.com/role-arn" | ||
| value = module.karpenter.irsa_arn | ||
| } | ||
|
|
||
| set { | ||
| name = "settings.aws.defaultInstanceProfile" | ||
| value = module.karpenter.instance_profile_name | ||
| } | ||
|
|
||
| set { | ||
| name = "settings.aws.interruptionQueueName" | ||
| value = module.karpenter.queue_name | ||
| } | ||
| } | ||
| ``` | ||
|
|
@@ -358,32 +335,47 @@ provider "kubectl" { | |
| exec { | ||
| api_version = "client.authentication.k8s.io/v1beta1" | ||
| command = "aws" | ||
| args = ["eks", "get-token", "--cluster-name", module.eks.cluster_name] | ||
| } | ||
| } | ||
|
|
||
| resource "kubectl_manifest" "karpenter_provisioner" { | ||
| yaml_body = <<-YAML | ||
| apiVersion: karpenter.sh/v1alpha5 | ||
| kind: Provisioner | ||
| metadata: | ||
| name: default | ||
| spec: | ||
| requirements: | ||
| - key: karpenter.sh/capacity-type | ||
| operator: In | ||
| values: ["spot"] | ||
| limits: | ||
| resources: | ||
| cpu: 1000 | ||
| providerRef: | ||
| name: default | ||
| ttlSecondsAfterEmpty: 30 | ||
| YAML | ||
|
|
||
| depends_on = [ | ||
| helm_release.karpenter | ||
| ] | ||
| } | ||
|
|
||
| resource "kubectl_manifest" "karpenter_node_template" { | ||
|
There was a problem hiding this comment. tags were removed from node group to de-emphasize the launch template usage of Karpenter and switch to the node template which has been added here |
||
| yaml_body = <<-YAML | ||
| apiVersion: karpenter.k8s.aws/v1alpha1 | ||
| kind: AWSNodeTemplate | ||
| metadata: | ||
| name: default | ||
| spec: | ||
| subnetSelector: | ||
| karpenter.sh/discovery: "true" | ||
| securityGroupSelector: | ||
| karpenter.sh/discovery: ${module.eks.cluster_name} | ||
| tags: | ||
| karpenter.sh/discovery: ${module.eks.cluster_name} | ||
| YAML | ||
|
|
||
| depends_on = [ | ||
|
|
@@ -434,7 +426,7 @@ spec: | |
| terminationGracePeriodSeconds: 0 | ||
| containers: | ||
| - name: inflate | ||
| image: public.ecr.aws/eks-distro/kubernetes/pause:3.7 | ||
| resources: | ||
| requests: | ||
| cpu: 1 | ||
|
|
@@ -477,8 +469,4 @@ created LaunchTemplates. | |
| kubectl delete deployment inflate | ||
| kubectl delete node -l karpenter.sh/provisioner-name=default | ||
| terraform destroy | ||
| ``` | ||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not required since this is handled through Terraform via Helm's SDK