[Umbrella] ISO-friendly Tagging

[YakDriver]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Relates #18593

Considerations

Five (at least) considerations relate to the problem of tagging support in non-standard AWS partitions:

• Tag on Create - When this is available, it's highly desireable. I.e., we want to prioritize its use.
• Tagging support (at all) - Not working at all contrasts with not supporting tag on create.
• Default tags - Ideally, practitioners will be able to use default tags even if not all resources in non-standard partitions support tagging.
• Changing AWS functionality - Ideally, we will not need to change the code when AWS adds levels of tagging support to other partitions. Features will simply start working.
• Avoid eating errors that should be fatal - Eventual consistency or actual tagging problems should result in errors reported to practitioners. Thus the challenge is to narrowly eat errors that specifically relate to lack of tagging support.

Design

A design to handle these considerations:

Tag on Create

Attempt tag on create

Tags?	DTags?	Error?	Result
0	0	0	👍
0	0	1	Don't ToC if len(tags) == 0
0	1	0	👍
0	1	1	Attempt post-create tag
1	0	0	👍
1	0	1	Attempt post-create tag
1	1	0	👍
1	1	1	Attempt post-create tag

Tag after Create

Only if tag on create fails, check error and, if appropriate, try create without tags.

Tags?	DTags?	Error?	Result
0	1	0	👍
0	1	1	Log error, continue
1	0	0	👍
1	0	1	Fatal error (explicit setting of tags when tagging not possible)
1	1	0	👍
1	1	1	Fatal error (explicit setting of tags when tagging not possible)

Read

1. Attempt to read tags
2. If tag read fails, check error and, if appropriate, skip tag read.

Tags?	DTags?	Error?	Result
0	0	0	👍
0	0	1	Regular, non-tag error
0	1	0	👍
0	1	1	Log error, continue

Update

1. Attempt tag update
2. If update fails, check error and, if appropriate, skip tag update.

Tags?	DTags?	Error?	Result
0	0	0	👍
0	0	1	Regular, non-tag error
0	1	0	👍
0	1	1	Log error, continue

New or Affected Resource(s)

• ds/aws_ecr_repository (ecr: ISO-friendly tagging #22535)
• ds/aws_iam_role (iam: ISO-friendly tagging #22544)
• ds/aws_iam_user (iam: ISO-friendly tagging #22544)
• ds/aws_lb (elbv2: ISO-friendly tagging #22551)
• ds/aws_lb_listener (elbv2: ISO-friendly tagging #22551)
• ds/aws_lb_target_group (elbv2: ISO-friendly tagging #22551)
• ds/aws_sqs_queue (sqs/queue: Make ISO-friendly tags #22516)
• r/aws_cloudwatch_event_rule (eventbridge: ISO-friendly tagging #22550)
• r/aws_cloudwatch_event_bus (eventbridge: ISO-friendly tagging #22550)
• r/aws_cloudwatch_composite_alarm (cloudwatch: ISO-friendly tagging #22556)
• r/aws_cloudwatch_metric_alarm (cloudwatch: ISO-friendly tagging #22556)
• r/aws_cloudwatch_metric_stream (cloudwatch: ISO-friendly tagging #22556)
• r/aws_ecr_repository (ecr: ISO-friendly tagging #22535)
• r/aws_ecs_capacity_provider (ecs: ISO-friendly tagging #22529)
• r/aws_ecs_cluster (ecs: ISO-friendly tagging #22529)
• r/aws_ecs_service (ecs: ISO-friendly tagging #22529)
• r/aws_ecs_task_definition (ecs: ISO-friendly tagging #22529)
• r/aws_ecs_task_set (ecs: ISO-friendly tagging #22529)
• r/aws_iam_group (no tags)
• r/aws_iam_role (iam: ISO-friendly tagging #22544)
• r/aws_iam_user (iam: ISO-friendly tagging #22544)
• r/aws_lb (elbv2: ISO-friendly tagging #22551)
• r/aws_lb_listener (elbv2: ISO-friendly tagging #22551)
• r/aws_lb_listener_rule (elbv2: ISO-friendly tagging #22551)
• r/aws_lb_target_group (elbv2: ISO-friendly tagging #22551)
• r/aws_sns_topic (sns/topic: ISO-friendly tagging #22511)
• r/aws_sqs_queue (sqs/queue: Make ISO-friendly tags #22516)

References

• Support aws-iso, aws-iso-b, aws-iso-e, aws-iso-f partitions #18593

[blafrisch]

aws_cloudwatch_event_rule and aws_lb_listener are additional resources that we know of to have tagging issues in the ISO regions.

[YakDriver]

@blafrisch @lorengordon We're doing our best to anticipate the errors but since they often vary from service to service, we are not 100% confident these fixes will guard against the exact errors for all these services. This is attempt #1. 🤞

Now with the guard structure in place, updating for the correct error is pretty trivial. But, as we cannot test ourselves, any help with that would be much appreciated. The error code and message would be ideal.

Here are the error codes that v3.72.0 will be guarding against:

CloudWatch:

• AccessDenied
• InternalServiceFault

ECR:

• AccessDenied
• InvalidParameter
• Validation

ECS:

• AccessDenied
• InvalidParameter
• UnsupportedFeature

ELBv2:

• AccessDenied
• InvalidConfigurationRequest
• OperationNotPermitted

EventBridge (CloudWatch Events):

• AccessDenied
• InternalException
• OperationDisabled

IAM

• AccessDenied
• InvalidInput
• ServiceFailure

SNS:

• AccessDenied
• AuthorizationError
• InvalidAction

SQS:

• AccessDenied
• AuthorizationError
• InvalidAction
• UnsupportedOperation

[egon010]

Is there a test suite yet to gather results from various partitions/regions?

[YakDriver]

Yes and no. Formally, we only run daily tests across aws (standard partition) and aws-us-gov (GovCloud), and mostly us-west-2 and us-gov-west-1. We track those results. We don't have access to iso regions in house. There are some similarities between the China and iso partitions. We don't have information from AWS on whether they are the same but that could be a potential avenue for testing.

[lorengordon]

We'll give it a go, thanks for the heads up!

[YakDriver]

I'll keep this issue open until we hear back on success and/or failure.

[github-actions]

This functionality has been released in v3.72.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[blafrisch]

@YakDriver we were able to deploy v3.72.0 and test with the aws_cloudwatch_event_rule and aws_lb_listener resources; both still experienced errors. Note that these messages are hand-transcribed and have some info redacted:

For aws_cloudwatch_event_rule:

Error: error listing tags for EventBridge Rule (arn:aws-iso:events:<REDACTED>:rule/<REDACTED>): UnknownOperationException Operation is disabled in this region
status code: 400

For aws_lb_listener:

Error: error listing tags for (arn:aws-iso:elasticloadbalancing:<REDACTED>:listener/<REDACTED>): ValidationError: Resource type 'listener' does not support tagging
status code: 400

[spavuluri]

@YakDriver can confirm the same error with v3.72.0 in aws-iso.

In the console, the resource appears like its created, at least in the case of cloudwatch event rule. But the apply reports above error.

Testing continues.

[YakDriver]

These are services we've heard back about where tags are now not causing problems.

@lorengordon @spavuluri @blafrisch Let us know if you run into problems! Or, if these are working...

(These may differ between iso and iso-b.)

• CloudWatch (ISO Tags: Improve error handling #22717)
• ECR
• ECS
• ELBv2 (ISO Tags: Improve error handling #22717)
• EventBridge (CloudWatch Events) (ISO Tags: Improve error handling #22717)
• IAM (may not have ever been an issue, relying on old docs to know there was a problem)
• SNS
• SQS

[blafrisch]

@YakDriver I can confirm that the most recent changes were successful in aws-iso for aws_cloudwatch_event_rule and aws_lb_listener. 🎉

[spavuluri]

@YakDriver no issues encountered for aws_sqs_queue, aws_sns_topic, aws_cloudwatch_metric_alarm, aws_ecr_repository in aws-iso

[YakDriver]

@spavuluri @blafrisch Millions of thanks!!

We have heard back that ECS continues to be a problem but still need to find out what the exact error message is. If anyone has the AWS error code (e.g., UnknownOperationException) and message, that would be super helpful! Once we have that, we can get a fix out quickly.

[YakDriver]

ECS handles tagging differently than (most?) every other service. As a result, it took a different approach than the others but hopefully it should be working now with #23030. If we don't hear back for anyone in the next 30 days on more tagging problems in ISO, I'll close this issue on or after March 8, 2022. Thanks for all the help!

[lorengordon]

@YakDriver I see the PR fixing ECS tags was marked for the v4 release... Does that mean it won't make it into a v3 release?

[YakDriver]

That is likely correct. Any estimate of showstoppers with v4 and ISO?

[lorengordon]

Just makes it harder to test, is all, with all the breaking changes. Folks might need some time to rewrite configs to account for that, before they can report that things are or are not working.

[caleb-devops]

@YakDriver v3.74.0 did fix most of the errors relating to tagging, however I am receiving the following error during terraform apply for aws_iam_instance_profile resources:

error updating tags for IAM Instance Profile (<REDACTED>): Error tagging resource (<REDACTED>): InvalidAction: Unsupported operation.

[YakDriver]

The ECS fixes have been backported to v3.74 and are available in v3.74.3.

[AbrohamLincoln]

I ran into this this today with the following resources:
aws_elasticache_subnet_group
aws_elasticache_replication_group

[YakDriver]

@AbrohamLincoln Thank you for letting us know. We'll work on a fix.

[YakDriver]

@AbrohamLincoln For ElastiCache fixes, see #23980. When 4.9 is released (next week I believe), please give it a test and let us know if it works. We are unable to test in iso or iso-b on our own.

[cvockrodt]

I've seen some differing information here regarding IAM tags

You cannot tag the following resources in aws-iso (not sure about aws-iso-b:

• IAM SAML identity providers
• Instance Profiles
• OpenID Connect (OIDC) identity providers
• Policies
• Server certificates
• Virtual MFA devices

This affects the ability to use provider level default tags when managing any of the above resources.

[github-actions]

Marking this issue as stale due to inactivity. This helps our maintainers find and focus on the active issues. If this issue receives no comments in the next 30 days it will automatically be closed. Maintainers can also remove the stale label.

If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.