Make MSK Cluster create timeouts configurable

[Stratouklos]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

New AWS MSK Clusters can routinely take close to 60 minutes or more to be created. This causes timeouts and in subsequent runs the resource shows as tainted requiring manual untainting.

New or Affected Resource(s)

• aws_msk_cluster

Potential Terraform Configuration

resource "aws_msk_cluster" "kafka" {
  cluster_name           = local.service
  kafka_version          = var.kafka_version
  number_of_broker_nodes = var.number_of_broker_nodes

  broker_node_group_info {
    instance_type   = var.instance_type
    ebs_volume_size = var.ebs_volume_size
    client_subnets  = flatten([for s in data.aws_subnet_ids.private : s.ids])
    security_groups = [aws_security_group.kafka.id]
  }

  configuration_info {
    arn      = aws_msk_configuration.kafka.arn
    revision = aws_msk_configuration.kafka.latest_revision
  }

  encryption_info {
    encryption_at_rest_kms_key_arn = aws_kms_key.kafka_key.arn

    encryption_in_transit {
      client_broker = "TLS"
      in_cluster    = true
    }
  }

  client_authentication {
    sasl {
      scram = true
    }
  }

  open_monitoring {
    prometheus {
      jmx_exporter {
        enabled_in_broker = true
      }
      node_exporter {
        enabled_in_broker = true
      }
    }
  }

  logging_info {
    broker_logs {
      cloudwatch_logs {
        enabled   = true
        log_group = aws_cloudwatch_log_group.broker_logs.name
      }
    }
  }

  timeouts {
    create = "2h" # As of Feb 2021 MSK clusters can take 1h or more to be created 
  }
}

References

• NA

[ewbankkit]

For creation (1h):

terraform-provider-aws/aws/resource_aws_msk_cluster.go

Lines 393 to 397 in 27f1e21

	func waitForMskClusterCreation(conn *kafka.Kafka, arn string) error {
	input := &kafka.DescribeClusterInput{
	ClusterArn: aws.String(arn),
	}
	err := resource.Retry(60*time.Minute, func() *resource.RetryError {

but for other cluster operations (2h):

terraform-provider-aws/aws/resource_aws_msk_cluster.go

Lines 1196 to 1202 in 27f1e21

	func waitForMskClusterOperation(conn *kafka.Kafka, clusterOperationARN string) error {
	stateConf := &resource.StateChangeConf{
	Pending: []string{"PENDING", "UPDATE_IN_PROGRESS"},
	Target: []string{"UPDATE_COMPLETE"},
	Refresh: mskClusterOperationRefreshFunc(conn, clusterOperationARN),
	Timeout: 2 * time.Hour,
	}

[mgusiew-guide]

FTR there is another issue describing this problem: #11736

[github-actions]

This functionality has been released in v3.60.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.