CFE-1131: AWS Tags DAY2 Update #297

anirudhAgniRedhat · 2024-10-11T11:25:26Z

This PR introduces a custom EBSVolumeTagController that monitors the OpenShift Infrastructure resource for changes in AWS ResourceTags. When tags are updated, the controller automatically fetches all AWS EBS-backed PersistentVolumes (PVs) in the cluster, retrieves their volume IDs, and updates the associated EBS tags in AWS.

Key Changes:

Monitors Infrastructure resource for AWS ResourceTags updates.
Directly fetches all PVs using the AWS EBS CSI driver (ebs.csi.aws.com).
Updates AWS EBS tags by merging new and existing tags using the AWS SDK.

openshift-ci-robot · 2024-10-11T11:25:29Z

@anirudhAgniRedhat: This pull request references CFE-1131 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.18.0" version, but no target version was set.

In response to this:

This PR introduces a custom EBSVolumeTagController that monitors the OpenShift Infrastructure resource for changes in AWS ResourceTags. When tags are updated, the controller automatically fetches all AWS EBS-backed PersistentVolumes (PVs) in the cluster, retrieves their volume IDs, and updates the associated EBS tags in AWS.

Key Changes:

Monitors Infrastructure resource for AWS ResourceTags updates.
Directly fetches all PVs using the AWS EBS CSI driver (ebs.csi.aws.com).
Updates AWS EBS tags by merging new and existing tags using the AWS SDK.
Graceful operator restart on tag changes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

anirudhAgniRedhat · 2024-10-11T11:25:54Z

/hold
Please don't review now currently WIP

openshift-ci · 2024-10-11T11:29:03Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: anirudhAgniRedhat
Once this PR has been reviewed and has the lgtm label, please assign tsmetana for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci-robot · 2024-10-14T15:35:16Z

@anirudhAgniRedhat: This pull request references CFE-1131 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.18.0" version, but no target version was set.

In response to this:

This PR introduces a custom EBSVolumeTagController that monitors the OpenShift Infrastructure resource for changes in AWS ResourceTags. When tags are updated, the controller automatically fetches all AWS EBS-backed PersistentVolumes (PVs) in the cluster, retrieves their volume IDs, and updates the associated EBS tags in AWS.

Key Changes:

Monitors Infrastructure resource for AWS ResourceTags updates.
Directly fetches all PVs using the AWS EBS CSI driver (ebs.csi.aws.com).
Updates AWS EBS tags by merging new and existing tags using the AWS SDK.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

anirudhAgniRedhat · 2024-10-15T18:07:36Z

/unhold
open for reviews

TrilokGeer · 2024-10-16T07:35:47Z

cmd/aws-ebs-csi-driver-operator/main.go

+
+	go ebsTagsController.Run(ctx)
+
+	klog.Info("EBS Volume Tag Controller is running")


nit: TBD - cleanup info logs before merge.

TrilokGeer · 2024-10-16T10:35:48Z

pkg/driver/aws-ebs/aws_ebs_tags_controller.go

+	klog.Infof("Updating EBS tags for volume ID %s with tags: %v", volumeID, mergedTags)
+
+	// Create or update the tags
+	_, err = ec2Client.CreateTags(&ec2.CreateTagsInput{


What is the behaviour when no.of tags > 50?

Hey @TrilokGeer I tested the scenario the AWS client gives a bad request error which states Error updating tags for volume vol-08d7a36e4b077873f: TagLimitExceeded: More than 50 tags specified status code: 400, request id: 364561fd-6319-40e2-b3d1-ff110954e73c

We are not reconciling again on this request. What else you think we should do other than this??

An error event or message must be set on the operator status to indicate the error.

anirudhAgniRedhat · 2024-10-21T04:16:29Z

/cc @jsafrane
PTAL!!

jsafrane

Please update cluster-storage-operator to add token-minter sidecar.

cmd/aws-ebs-csi-driver-operator/main.go

pkg/driver/aws-ebs/aws_ebs_tags_controller.go

jsafrane · 2024-10-21T14:55:31Z

pkg/driver/aws-ebs/aws_ebs_tags_controller.go

+			if err != nil {
+				klog.Errorf("Error updating tags for volume %s: %v", volumeID, err)


There should be retry with exp. backoff. Esp. when CreateTags calls are throttled by AWS.
That probably implies a queue of PVs.

pkg/driver/aws-ebs/aws_ebs_tags_controller.go

gnufied · 2024-10-22T19:42:36Z

pkg/driver/aws-ebs/aws_ebs_tags_controller.go

+	infraInformer := c.commonClient.ConfigInformers.Config().V1().Infrastructures().Informer()
+
+	// Add event handler to process updates only when ResourceTags change
+	infraInformer.AddEventHandler(cache.ResourceEventHandlerFuncs{


We don't have do this. If we are using factory.WithInformers, we will reconcile this within Sync function.

Hey @gnufied I was bit confused with this!! basically I don't want to run reconciliation on every change on InfraStructure resource. I need to run reconciliation only if there is a any change in infra.Status.PlatformStatus.AWS.ResourceTags.

Can you suggest a better way to do this so that we can remove the unnecessarily computes.

How will this work when controller is restarted? You are also doing WithInformers below and hence any change in infra object will still trigger Sync.

So - most bulletproof way of ensuring that, we don't unnecessarily process all PVs is to store the information that we have processed these PVs somewhere in a persistent way.

So, what we have currently is worst of both the worlds. If I were to design this, I will probably make a hash of sorted tags and annotate PV with tag hash. If tag hash annotation in PV and computed tag hash don't change, then I will not update the PV or else I will.

How will this work when controller is restarted? You are also doing WithInformers below and hence any change in infra object will still trigger Sync.

What I am thinking is on every restart I would like to run a sync function and update the volumes if there is a change and further On each change in resource-tags we would again run the sync.

So, what we have currently is worst of both the worlds. If I were to design this, I will probably make a hash of sorted tags and annotate PV with tag hash. If tag hash annotation in PV and computed tag hash don't change, then I will not update the PV or else I will.

This looks easy way to manage this!! I will then add a new field in controller struct which will have the updated hash for the sorted map of tags! now in each reconciliation I will update tags only if the hash is different from the other one? Does this sounds better to you??

But that is not what I said. I said, we should store hash of sorted tags in PV objects as annotation and compare those with current tags we are about to apply. We should only apply tags with AWS if hashes change.

Storing them just in-memory doesn't help us much.

Added Thanks For Suggestion!!

gnufied · 2024-10-23T13:28:01Z

pkg/driver/aws-ebs/aws_ebs_tags_controller.go

+	if err != nil {
+		return err
+	}
+	err = c.processInfrastructure(infra, ec2Client)


So, does this controller needs to be an opt-in or a default controller? I don't assume every OCP customer wants this feature and if tagging were to fail after OCP upgrade, their clusters will be degraded and we will have support nightmare.

cc @jsafrane

It is presented as enabled by default + no opt out for all HyperShift clusters in the enhancement.

gnufied · 2024-10-23T13:28:24Z

pkg/driver/aws-ebs/aws_ebs_tags_controller.go

+}
+
+// startFailedQueueWorker runs a worker that processes failed batches independently
+func (c *EBSVolumeTagsController) startFailedQueueWorker(ctx context.Context) {


I am not sure if we need this function at all.

So the reason I brought this change is I would like to retry the the batches which have failed to update tags!! Here I would like to update tags in a serial order(one By one) Discussuion link for the volumes. I cannot use the similar sync function here or else you could suggest a better way for this!!

If we are going to per-pv hash, then we are not going to need this right?

The points I want to make sure here are

I definitely need to batch volumes in order to not to hit throttling condition.

Now on failure we would need to retry!!

Since we are using batch APIs, so AWS SDK's APIs will give us error if any one of the VolumeID in the batch hits the any error(May be validation, Auth, Permission etc), All the VolumeIDs in that batch will not be able to update the tags! In this condition I would like to add a worker queue that will handle the the serial update of PV's tags and will retry to update the tags in exponential back-off time-period.

Since We need to figure out the trade-off between the either in first place we should update all PVs using AWS API's in serial order and retry using similar sync function or should we use Batch Volumes to be called in the sync Functions and later retry should be processed with the queue function serially until the queue is empty!!

I definitely need to batch volumes in order to not to hit throttling condition.

That is fine. You are already batching PVs in fetchPVsAndUpdateTags.

Now on failure we would need to retry!!

We are unnecessarily complicating this. The controller is going to resync every 20 minutes anyways, so it will try to tag all the PVs which aren't tagged. So do we even need to keep separate queue for failed PVs? I am also afraid that, your failed worker queue is going to race with normal controller resync.

What is the point of separate failed worker queue when every 20 minutes, we are going to try and sync tag for all PVs which doesn't have matching tag-hash? If you really want a separate worker queue, you will have to redesign the whole thing, so as at least they are not racy. But I don't really understand point of doing it.

Hey @gnufied I completely agree that this retrying the failure is unnecessarily making the changes complicated.

@jsafrane I had a chat with @TrilokGeer regarding this, IMHO we should drop the idea to add degraded condition on tags update failure. As the cluster should not be degraded for failure to apply Tags. as the sync will anyhow retry to tag the volumes withing resync period, in this way we will not immediately require to retry the failures and can remove this queue worker.
Also anyways we are emitting the warning events from the this controller if the tags update is failed. so user will know that the tags update has been failed due the the certain reason also can think of alerts based on that.

/cc @TrilokGeer
Can you also put your views on this!!

BTW are we planning to backport this PR to older releases?

@gnufied I guess we would need to backport this to 4.17. Slack Thread.

So I noticed that you removed failed worker logic. The thing is - I was talking to @jsafrane offline and he has me convinced that, we do need some kind of additional logic so as we can try retagging of "failed" PVs one-by-one, rather than in a batch. This will ensure that, one bad apple in a batch doesn't prevent tagging of rest of the PVs.

But - we need to be careful when doing this.

We should make sure that, PVs which will be tagged via failed worker, doesn't get processed via regular controller resync (so no race).

I would move the entire failed worker code in a separate file.

Ack;
I believe we can remove the ResyncEvery parameter from the controller builder as this is a overkill for us why do we want to run the sync function in every 20 mins if nothing has changed in the resource Tags.

Alternatively, If you really think resync is important here then, I think that we can add the handle the race condition by using another annotation in PVs for Tagging status and filtering based on tagshash and status in the annotation. But this will also cost us some volume Update calls and further need to handle cases where we are not able to update the status within batches. WDYT?

pkg/driver/aws-ebs/aws_ebs_tags_controller.go

anirudhAgniRedhat · 2024-10-26T14:22:40Z

/retest

gnufied · 2024-10-26T15:26:39Z

pkg/driver/aws-ebs/aws_ebs_tags_controller.go

+		failedQueue:   workqueue.NewRateLimitingQueue(workqueue.DefaultControllerRateLimiter()),
+	}
+
+	go c.startFailedQueueWorker(ctx)


Problem with starting a worker here is, we are not waiting for caches to be synced before starting stuff. We shouldn't be starting workers at this stage.

I have removed the worker from here and we are initializing the worker in postStartHook after the caches are synced and controller is started.

pkg/driver/aws-ebs/aws_ebs_tags_controller.go

anirudhAgniRedhat · 2024-11-07T08:10:03Z

Please update cluster-storage-operator to add token-minter sidecar.

Added PR to add token minter sidecar PTAL!
openshift/cluster-storage-operator#528

anirudhAgniRedhat · 2024-11-07T11:53:55Z

/retest
/test e2e-openstack-cinder-csi

gnufied · 2024-11-07T21:32:56Z

pkg/driver/aws-ebs/aws_ebs_tags_controller.go

+}
+
+// getInfrastructure retrieves the Infrastructure resource in OpenShift
+func (c *EBSVolumeTagsController) getInfrastructure(ctx context.Context) (*configv1.Infrastructure, error) {


why are we reading an object from informer with exponential backoff? See how we do this in other operators - https://github.com/openshift/vmware-vsphere-csi-driver-operator/blob/master/pkg/operator/vspherecontroller/vspherecontroller.go#L167

gnufied · 2024-11-07T21:37:16Z

pkg/driver/aws-ebs/aws_ebs_tags_controller.go

+	if infra.Status.PlatformStatus != nil && infra.Status.PlatformStatus.AWS != nil {
+		infraRegion = infra.Status.PlatformStatus.AWS.Region
+	}
+	ec2Client, err := c.getEC2Client(ctx, infraRegion)


IMO, we shouldn't create ec2Client here until we know we are going to need to tag PV objects. Since we are planning to backport this feature, for most infra objects, there may not even be any tags that needded to be synced on AWS. So, this controller will be a NO-OP for many clusters.

This means at very minimum, I would check if infra object has non-nil infra.Status.PlatformStatus.AWS.ResourceTags, before going ahead with ec2 client creation.

gnufied · 2024-11-07T21:40:50Z

pkg/driver/aws-ebs/aws_ebs_tags_controller.go

+	return nil
+}
+
+func (c *EBSVolumeTagsController) listPersistentVolumesWithRetry(ctx context.Context) ([]*v1.PersistentVolume, error) {


same comment as before. There is no need to list PV objects from a informer with an exponential backoff. Informer has cached objects, it isn't supposed to fail.

gnufied · 2024-11-07T21:45:04Z

pkg/driver/aws-ebs/aws_ebs_tags_controller.go

+func newAndUpdatedTags(resourceTags []configv1.AWSResourceTag) []*ec2.Tag {
+	// Convert map back to slice of ec2.Tag
+	var tags []*ec2.Tag
+	for _, tag := range resourceTags {


I am confused. The function definition says - it is supposed to "merge" tags, but all I see is, this function returns configv1.AWSResourceTag as ec2.Tag objects. I do not see any "merge" happening.

gnufied · 2024-11-07T21:47:05Z

pkg/driver/aws-ebs/aws_ebs_tags_controller.go

+func (c *EBSVolumeTagsController) handleTagUpdateFailure(batch []*v1.PersistentVolume, error error) {
+	errorMessage := fmt.Sprintf("Error updating tags for volume %v: %v", batch, error)
+	for _, pv := range batch {
+		klog.Errorf("Error updating volume %v tags: %v", pv.Name, errorMessage)


so, I do not think dumping full PV objects in error log is a good idea here. This will make logs very noisy. Also, didn't we add full PV object already in error object?

Please do not use error as a variable name, since error is also an interface. This confuses tools.

gnufied · 2024-11-07T21:48:06Z

pkg/driver/aws-ebs/aws_ebs_tags_controller.go

+	})
+
+	if err != nil {
+		return fmt.Errorf("error creating tags for volume %v: %v", pvBatch, err)


Aren't we appending full PV objects again and again to same error message?

gnufied · 2024-11-07T21:51:50Z

pkg/driver/aws-ebs/aws_ebs_tags_controller.go

+	}
+
+	// Set or update the tag hash annotation
+	pv.Annotations[tagHashAnnotationKey] = hash


We should get DeepCopy before modifying object we retrieved from informer. We do not want to modify object we retrieved from informers directly.

gnufied · 2024-11-07T21:54:43Z

pkg/driver/aws-ebs/aws_ebs_tags_controller.go

+			// Update the PV with the new annotations
+			err = c.updateVolumeWithRetry(ctx, volume)
+			if err != nil {
+				klog.Errorf("Error updating PV annotations for volume %s: %v", volume.Name, err)


If we are already going to handle these PVs via failedWorkerQueue, why does the code in updateVOlumeWithRetry needs to have its own local exponential backoff?

gnufied · 2024-11-07T21:55:30Z

pkg/driver/aws-ebs/aws_ebs_tags_controller.go

+		return nil
+	}
+
+	// Process the volumes in batches


I mentioned in my original comment that, the main controller should not try and update PVs which will be processed by failed worker queue. I do not see that scenairo handled here.

gnufied · 2024-11-07T21:56:03Z

pkg/driver/aws-ebs/aws_ebs_tags_retry_worker.go

+			return errors.New("context canceled, stopping failed queue worker for EBS Volume Tags") // Stop the goroutine when the context is canceled
+		default:
+			// Get the next failed volume from the queue
+			item, quit := c.failedQueue.Get()


This code is very hard to read, can you split this into multiple smaller functions please.

openshift-ci · 2024-11-09T07:11:37Z

@anirudhAgniRedhat: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/security	`869a428`	link	false	`/test security`
ci/prow/e2e-azurestack-csi	`869a428`	link	false	`/test e2e-azurestack-csi`
ci/prow/e2e-openstack-manila-csi	`869a428`	link	true	`/test e2e-openstack-manila-csi`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Oct 11, 2024

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Oct 11, 2024

openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Oct 11, 2024

openshift-ci bot requested review from dobsonj and RomanBednar October 11, 2024 11:28

anirudhAgniRedhat force-pushed the AWS_DAY2_TAGS_RECONCILIATION branch from 4d92e01 to 14ecc0e Compare October 14, 2024 12:42

anirudhAgniRedhat force-pushed the AWS_DAY2_TAGS_RECONCILIATION branch from d68d71a to 44e34e7 Compare October 14, 2024 16:00

anirudhAgniRedhat changed the title ~~[WIP] CFE-1131: AWS Tags DAY2 Update~~ CFE-1131: AWS Tags DAY2 Update Oct 14, 2024

openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Oct 14, 2024

openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Oct 15, 2024

TrilokGeer reviewed Oct 16, 2024

View reviewed changes

openshift-ci bot requested a review from jsafrane October 21, 2024 04:16

jsafrane reviewed Oct 21, 2024

View reviewed changes

anirudhAgniRedhat force-pushed the AWS_DAY2_TAGS_RECONCILIATION branch 2 times, most recently from 146634c to 858b442 Compare October 22, 2024 18:47

gnufied reviewed Oct 22, 2024

View reviewed changes

anirudhAgniRedhat force-pushed the AWS_DAY2_TAGS_RECONCILIATION branch from 858b442 to fe7b17e Compare October 23, 2024 08:31

gnufied reviewed Oct 23, 2024

View reviewed changes

gnufied reviewed Oct 24, 2024

View reviewed changes

pkg/driver/aws-ebs/aws_ebs_tags_controller.go Outdated Show resolved Hide resolved

anirudhAgniRedhat force-pushed the AWS_DAY2_TAGS_RECONCILIATION branch 3 times, most recently from a9b8b54 to 42579da Compare October 25, 2024 11:40

gnufied reviewed Oct 26, 2024

View reviewed changes

openshift-ci bot requested a review from TrilokGeer October 28, 2024 06:13

gnufied reviewed Oct 29, 2024

View reviewed changes

pkg/driver/aws-ebs/aws_ebs_tags_controller.go Outdated Show resolved Hide resolved

anirudhAgniRedhat force-pushed the AWS_DAY2_TAGS_RECONCILIATION branch 7 times, most recently from e70bfeb to c0e7dd8 Compare November 4, 2024 13:02

anirudhAgniRedhat mentioned this pull request Nov 5, 2024

CFE-1132: EFS Access Point Tags Update DAY2 #313

Open

anirudhAgniRedhat force-pushed the AWS_DAY2_TAGS_RECONCILIATION branch from c1ae14d to c62693a Compare November 7, 2024 05:18

anirudhAgniRedhat requested review from jsafrane and gnufied November 7, 2024 07:28

Added tag controller for tags reconciliation

869a428

anirudhAgniRedhat force-pushed the AWS_DAY2_TAGS_RECONCILIATION branch from c62693a to 869a428 Compare November 7, 2024 08:07

gnufied reviewed Nov 7, 2024

View reviewed changes

jsafrane mentioned this pull request Nov 19, 2024

CFE-1162: Updates enhancement to reflect hcp usecase and with latest information on aws tags support openshift/enhancements#1700

Open


		go ebsTagsController.Run(ctx)

		klog.Info("EBS Volume Tag Controller is running")

		if err != nil {
		klog.Errorf("Error updating tags for volume %s: %v", volumeID, err)

CFE-1131: AWS Tags DAY2 Update #297

Are you sure you want to change the base?

CFE-1131: AWS Tags DAY2 Update #297

Conversation

anirudhAgniRedhat commented Oct 11, 2024 • edited Loading

openshift-ci-robot commented Oct 11, 2024 • edited by openshift-ci bot Loading

anirudhAgniRedhat commented Oct 11, 2024

openshift-ci bot commented Oct 11, 2024

openshift-ci-robot commented Oct 14, 2024 • edited by openshift-ci bot Loading

anirudhAgniRedhat commented Oct 15, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

anirudhAgniRedhat commented Oct 21, 2024

jsafrane left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gnufied Oct 26, 2024 • edited Loading

Choose a reason for hiding this comment

anirudhAgniRedhat Oct 28, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

anirudhAgniRedhat Oct 30, 2024 • edited Loading

Choose a reason for hiding this comment

anirudhAgniRedhat commented Oct 26, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

anirudhAgniRedhat commented Nov 7, 2024

anirudhAgniRedhat commented Nov 7, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

openshift-ci bot commented Nov 9, 2024

anirudhAgniRedhat commented Oct 11, 2024 •

edited

Loading

openshift-ci-robot commented Oct 11, 2024 •

edited by openshift-ci bot

Loading

openshift-ci-robot commented Oct 14, 2024 •

edited by openshift-ci bot

Loading

gnufied Oct 26, 2024 •

edited

Loading

anirudhAgniRedhat Oct 28, 2024 •

edited

Loading

anirudhAgniRedhat Oct 30, 2024 •

edited

Loading