Add deletes to change feeds by deads2k · Pull Request #6058 · Azure/ARO-HCP

David Eads (deads2k) · 2026-07-13T19:57:16Z

Add deletes to changefeeds

Copilot

Pull request overview

This PR updates Cosmos-backed informers/controllers and supporting tooling to propagate resource deletions through “latest version” Cosmos change feeds by implementing soft-delete (deletionTimestamp + TTL) and by treating instanceVersion as the informer resourceVersion.

Changes:

Implement soft-delete semantics in Cosmos CRUD (deletionTimestamp + short TTL) and filter soft-deleted docs from Get/List paths.
Introduce/extend change-feed client plumbing across DB clients, informers, mocks, and integration tests to deliver timely Deleted events.
Update tooling and logging to understand changefeed log entries and gate event application by instanceVersion; bump azcosmos dependency.

Reviewed changes

Copilot reviewed 130 out of 141 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
tooling/hcpctl/cmd/datadump-to-git/cmd.go	Parse changefeed logs; instanceVersion gating
tooling/cleanup-sweeper/go.sum	Dependency checksum updates
test/go.sum	Dependency checksum updates
test/go.mod	Bump azcosmos dependency
test-integration/kube-applier/framework/framework.go	Update informer wiring signatures
test-integration/go.sum	Dependency checksum updates
test-integration/go.mod	Bump azcosmos dependency
test-integration/frontend/informer_test.go	Update informer wiring signatures
test-integration/backend/unionkubeapplierinformers/integration_test.go	Update informer factory signature usage
test-integration/backend/launch/notification_test.go	Update backend informer constructor usage
test-integration/backend/controllers/mismatches/delete_orphaned_cosmos_test.go	Wrapper implements new QueueForInformers
sessiongate/go.sum	Dependency checksum updates
mgmt-agent/go.sum	Dependency checksum updates
kube-applier/pkg/controllers/read_desire_kubernetes/controller.go	Add informer ObjectDescription
kube-applier/pkg/app/kube_applier.go	Pass changefeed client into informers
kube-applier/pkg/app/cosmos_wiring.go	Remove unused ctx parameter
kube-applier/go.sum	Dependency checksum updates
kube-applier/go.mod	Bump azcosmos dependency
kube-applier/cmd/root.go	Remove ctx plumbed into options wiring
internal/validation/validators.go	Guard nil CP versions during validation
internal/utils/context.go	Add AddResourceTypes helper
internal/go.sum	Dependency checksum updates
internal/go.mod	Bump azcosmos dependency
internal/databasetesting/mock_resources_global_lister.go	Filter soft-deleted docs in mocks
internal/databasetesting/mock_resources_db_client.go	Add mock changefeed support
internal/databasetesting/mock_resources_crud.go	Soft-delete behavior in mock CRUD
internal/databasetesting/mock_resources_changefeed.go	In-memory changefeed implementation
internal/databasetesting/mock_kube_applier_client.go	Add changefeed support to kube-applier mock
internal/database/unioninformers/kubeapplier/union_informers_test.go	Update informer constructor signature usage
internal/database/unioninformers/kubeapplier/factory.go	Pass changefeed client into informers
internal/database/unioninformers/kubeapplier/controller_test.go	Update informer constructor signature usage
internal/database/types_typeddocument.go	Add deletionTimestamp to typed envelope
internal/database/kube_applier_client.go	Expose changefeed APIs; adjust constructors
internal/database/informers/types.go	Plumb ChangeFeedClient into informers
internal/database/informers/informers.go	Switch desire informers to changefeed listwatch
internal/database/informers/informers_test.go	Update informer constructor signature usage
internal/database/informers/fleet_informers.go	Add informer ObjectDescription fields
internal/database/global_lister.go	Filter out soft-deleted docs in queries
internal/database/database.go	Add ChangeFeedClient interface to DB client
internal/database/crud_helpers.go	Implement soft-delete replace + TTL
internal/database/crud_hcpcluster.go	Filter out soft-deleted active operations
internal/api/types_runtime.go	Set ResourceVersion from InstanceVersion
internal/api/kubeapplier/types_runtime.go	Set ResourceVersion from InstanceVersion
internal/api/fleet/types_runtime.go	Set ResourceVersion from InstanceVersion
internal/api/arm/types_runtime.go	Set ResourceVersion from InstanceVersion
frontend/pkg/frontend/node_pool.go	Remove delete-time datadump logging
frontend/pkg/frontend/external_auth.go	Remove delete-time datadump logging
frontend/pkg/frontend/cluster.go	Remove delete-time datadump logging
frontend/go.sum	Dependency checksum updates
frontend/go.mod	Bump azcosmos dependency
fleet/pkg/controllers/base/stamp_watching_controller.go	Provide handler logger in notifier wiring
fleet/go.sum	Dependency checksum updates
fleet/go.mod	Bump azcosmos dependency
docs/cosmos-resource-deletion.md	Document soft-delete approach
backend/pkg/informers/types.go	Pass ResourcesDBClient as changefeed client
backend/pkg/informers/informers_test.go	Update informer constructors in tests
backend/pkg/controllers/validationcontrollers/nodepool_validation_controller.go	Remove cooldown checker plumbing
backend/pkg/controllers/validationcontrollers/nodepool_validation_controller_test.go	Update tests for cooldown removal
backend/pkg/controllers/validationcontrollers/cluster_validation_controller.go	Remove cooldown checker plumbing
backend/pkg/controllers/upgradecontrollers/trigger_node_pool_upgrade_controller.go	Remove cooldown checker plumbing
backend/pkg/controllers/upgradecontrollers/trigger_control_plane_upgrade_controller.go	Active create detection via ActiveOperationID
backend/pkg/controllers/upgradecontrollers/trigger_control_plane_upgrade_controller_test.go	Update tests for new gating logic
backend/pkg/controllers/upgradecontrollers/nodepool_version_controller.go	Queue SPC informer; NeedsWork requires CP versions
backend/pkg/controllers/upgradecontrollers/nodepool_version_controller_test.go	Update tests for signature/logic changes
backend/pkg/controllers/upgradecontrollers/nodepool_active_version_real_cosmos_test.go	Update tests for cooldown removal
backend/pkg/controllers/upgradecontrollers/nodepool_active_version_controller.go	Disable cooldown gating (nil checker)
backend/pkg/controllers/upgradecontrollers/nodepool_active_version_controller_test.go	Update tests for cooldown removal
backend/pkg/controllers/upgradecontrollers/control_plane_desired_version_controller.go	Active create detection via ActiveOperationID
backend/pkg/controllers/upgradecontrollers/control_plane_desired_version_controller_test.go	Update tests for new gating logic
backend/pkg/controllers/upgradecontrollers/control_plane_active_version_controller.go	Remove cooldown checker plumbing
backend/pkg/controllers/upgradecontrollers/control_plane_active_version_controller_test.go	Update tests for cooldown removal
backend/pkg/controllers/statuscontrollers/nodepool_degraded_aggregator.go	Handle NotFound on replace after delete
backend/pkg/controllers/statuscontrollers/nodepool_degraded_aggregator_test.go	Update tests for cooldown removal
backend/pkg/controllers/statuscontrollers/externalauth_degraded_aggregator.go	Handle NotFound on replace after delete
backend/pkg/controllers/statuscontrollers/externalauth_degraded_aggregator_test.go	Update tests for cooldown removal
backend/pkg/controllers/statuscontrollers/cluster_degraded_aggregator.go	Handle NotFound on replace after delete
backend/pkg/controllers/statuscontrollers/cluster_degraded_aggregator_test.go	Update tests for cooldown removal
backend/pkg/controllers/statuscontrollers/aggregator_testhelpers_test.go	Remove unused cooldown test helper
backend/pkg/controllers/operationcontrollers/generic_operation.go	Add QueueForInformers (panic stub)
backend/pkg/controllers/nodepooldeletion/node_pool_deletion_controller.go	Remove cooldown checker plumbing
backend/pkg/controllers/nodepooldeletion/node_pool_deletion_controller_test.go	Update tests for cooldown removal
backend/pkg/controllers/nodepooldeletion/node_pool_cluster_service_id_clearer.go	Remove cooldown checker plumbing
backend/pkg/controllers/nodepooldeletion/node_pool_cluster_service_id_clearer_test.go	Update tests for cooldown removal
backend/pkg/controllers/nodepooldeletion/node_pool_cluster_service_delete_dispatch_controller.go	Remove cooldown checker plumbing
backend/pkg/controllers/nodepooldeletion/node_pool_cluster_service_delete_dispatch_controller_test.go	Update tests for cooldown removal
backend/pkg/controllers/nodepooldeletion/node_pool_child_resources_cleanup_controller.go	Remove cooldown checker plumbing
backend/pkg/controllers/nodepooldeletion/node_pool_child_resources_cleanup_controller_test.go	Update tests for cooldown removal
backend/pkg/controllers/nodepoolcreationcontrollers/node_pool_cluster_service_create_controller.go	Remove cooldown checker plumbing
backend/pkg/controllers/nodepoolcreationcontrollers/node_pool_cluster_service_create_controller_test.go	Update tests for cooldown removal
backend/pkg/controllers/mismatchcontrollers/delete_orphaned_cosmos.go	Add QueueForInformers (panic stub)
backend/pkg/controllers/mismatchcontrollers/cluster_service_cluster_matching.go	Suppress “no healthy upstream” failure
backend/pkg/controllers/managementclustercontrollers/management_cluster_placement_sync.go	Remove cooldown checker plumbing
backend/pkg/controllers/managementclustercontrollers/management_cluster_placement_sync_test.go	Update tests for cooldown removal
backend/pkg/controllers/externalauthdeletion/external_auth_deletion_controller.go	Remove cooldown checker plumbing
backend/pkg/controllers/externalauthdeletion/external_auth_deletion_controller_test.go	Update tests for cooldown removal
backend/pkg/controllers/externalauthdeletion/external_auth_cluster_service_id_clearer.go	Remove cooldown checker plumbing
backend/pkg/controllers/externalauthdeletion/external_auth_cluster_service_id_clearer_test.go	Update tests for cooldown removal
backend/pkg/controllers/externalauthdeletion/external_auth_cluster_service_delete_dispatch_controller.go	Remove cooldown checker plumbing
backend/pkg/controllers/externalauthdeletion/external_auth_cluster_service_delete_dispatch_controller_test.go	Update tests for cooldown removal
backend/pkg/controllers/externalauthdeletion/external_auth_child_resources_cleanup_controller.go	Remove cooldown checker plumbing
backend/pkg/controllers/externalauthdeletion/external_auth_child_resources_cleanup_controller_test.go	Update tests for cooldown removal
backend/pkg/controllers/do_nothing.go	Add QueueForInformers (panic stub)
backend/pkg/controllers/datadumpcontrollers/dump_subscription_non_cluster.go	Use time-based cooldown only
backend/pkg/controllers/datadumpcontrollers/dump_cluster_recursive.go	Remove cooldown checker plumbing
backend/pkg/controllers/datadumpcontrollers/cs_state_dump.go	Simplify cooldown checker usage
backend/pkg/controllers/datadumpcontrollers/cs_state_dump_test.go	Update tests for cooldown changes
backend/pkg/controllers/datadumpcontrollers/billing_dump.go	Simplify cooldown checker usage
backend/pkg/controllers/datadumpcontrollers/billing_dump_test.go	Update tests for cooldown changes
backend/pkg/controllers/create_nodepool_scoped_read_desires_controller.go	Remove cooldown checker plumbing
backend/pkg/controllers/create_cluster_scoped_read_desires_controller.go	Remove cooldown checker plumbing
backend/pkg/controllers/controllerutils/util.go	Controller interface adds QueueForInformers
backend/pkg/controllers/controllerutils/nodepool_watching_controller.go	Remove syncer CooldownChecker requirement
backend/pkg/controllers/controllerutils/generic_watching_controller.go	Add handler loggers; nil cooldown handling
backend/pkg/controllers/controllerutils/external_auth_watching_controller.go	Remove syncer CooldownChecker requirement
backend/pkg/controllers/controllerutils/cluster_watching_controller.go	Remove syncer CooldownChecker requirement
backend/pkg/controllers/clusterpropertiescontroller/identity_migration.go	Remove cooldown checker plumbing
backend/pkg/controllers/clusterpropertiescontroller/identity_migration_test.go	Update tests for cooldown removal
backend/pkg/controllers/clusterpropertiescontroller/desired_control_plane_size_sync.go	Remove cooldown checker plumbing
backend/pkg/controllers/clusterpropertiescontroller/desired_control_plane_size_sync_test.go	Update tests for cooldown removal
backend/pkg/controllers/clusterpropertiescontroller/cluster_properties_test_helpers_test.go	Remove unused cooldown test helper
backend/pkg/controllers/clusterpropertiescontroller/cluster_properties_sync.go	Remove cooldown checker plumbing
backend/pkg/controllers/clusterpropertiescontroller/cluster_properties_sync_test.go	Update tests for cooldown removal
backend/pkg/controllers/clusterpropertiescontroller/cluster_base_domain_prefix_sync.go	Remove cooldown checker plumbing
backend/pkg/controllers/clusterpropertiescontroller/cluster_base_domain_prefix_sync_test.go	Update tests for cooldown removal
backend/pkg/controllers/clusterdeletion/cluster_deletion_controller.go	Remove cooldown checker plumbing
backend/pkg/controllers/clusterdeletion/cluster_deletion_controller_test.go	Update tests for cooldown removal
backend/pkg/controllers/clusterdeletion/cluster_cluster_service_id_clearer.go	Remove cooldown checker plumbing
backend/pkg/controllers/clusterdeletion/cluster_cluster_service_id_clearer_test.go	Update tests for cooldown removal
backend/pkg/controllers/clusterdeletion/cluster_cluster_service_delete_dispatch_controller.go	Remove cooldown checker plumbing
backend/pkg/controllers/clusterdeletion/cluster_cluster_service_delete_dispatch_controller_test.go	Update tests for cooldown removal
backend/pkg/controllers/clusterdeletion/cluster_child_resources_cleanup_controller.go	Remove cooldown checker plumbing
backend/pkg/controllers/clusterdeletion/cluster_child_resources_cleanup_controller_test.go	Update tests for cooldown removal
backend/pkg/controllers/billingcontrollers/orphaned_billing_cleanup.go	Add QueueForInformers (panic stub)
backend/pkg/app/backend.go	Pass DB client into backend informers
backend/go.sum	Dependency checksum updates
backend/go.mod	Bump azcosmos dependency
admin/server/go.sum	Dependency checksum updates
admin/server/go.mod	Bump azcosmos dependency

Copilot

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 3 comments.

Copilot

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 1 comment.

David Eads (deads2k) · 2026-07-14T15:33:56Z

+	var typedDoc TypedDocument
+	if err := json.Unmarshal(responseItem.Value, &typedDoc); err != nil {
+		return fmt.Errorf("failed to unmarshal document for soft delete: %w", err)
+	}


I wonder if this is true. I bet the bot is wrong since this mostly worked last e2e run.

David Eads (deads2k) · 2026-07-14T19:04:10Z

pods not running again. Need to reach the next level on that.

/retest

Manyanda Chitimbo (machi1990)

left two suggestions, other lgtm

Matthew Barnes (mbarnes) · 2026-07-15T12:49:24Z

LGTM as well, nothing to add beyond Manyanda Chitimbo (@machi1990)'s comments.

Copilot

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 1 comment.

Copilot

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 2 comments.

David Eads (deads2k) · 2026-07-15T18:40:56Z

comments addressed. apply label base don't on previous comment and in slack
/label lgtm

Instead of hard-deleting documents with DeleteItem (which the changefeed cannot observe), Delete now sets a DeletionTimestamp and a 30-second TTL on the document via a conditional Replace. This surfaces deletions on the changefeed so controllers receive timely watch.Deleted notifications. Get/GetByID return 404 for soft-deleted documents. All list queries (scoped, global, and ListActiveOperations) filter them out with NOT IS_DEFINED(c.deletionTimestamp). The changefeed watcher checks DeletionTimestamp before shouldDeliverItemFn and emits watch.Deleted for previously-seen resources. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

openshift-ci · 2026-07-15T19:59:58Z

New changes are detected. LGTM label has been removed.

David Eads (deads2k) · 2026-07-15T20:03:37Z

/label lgtm

Copilot

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 2 comments.

+	SetSoftDeleteFields(&doc, time.Now())
+
+	modified, err := json.Marshal(doc)
+	if err != nil {
+		return fmt.Errorf("failed to marshal soft-deleted document: %w", err)
+	}


+	ts := metav1.NewTime(now)
+	doc.DeletionTimestamp = &ts
+	doc.TimeToLive = SoftDeleteTTLSeconds
+


Manyanda Chitimbo (machi1990) · 2026-07-16T04:45:06Z

/lgtm

openshift-ci · 2026-07-16T04:45:16Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: deads2k, machi1990

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~docs/OWNERS~~ [deads2k]
~~internal/OWNERS~~ [deads2k]
~~test-integration/OWNERS~~ [deads2k]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Manyanda Chitimbo (machi1990) · 2026-07-16T04:51:11Z

/retest

The test failed with

{fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:291]: failed to create pull-secret-test pod in pullsecret-image-test
Unexpected error:
    <*errors.StatusError | 0xc0011fa280>: 
    pods "pull-secret-test" is forbidden: error looking up service account pullsecret-image-test/default: serviceaccount "default" not found
    ...
occurred  fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:291]: failed to create pull-secret-test pod in pullsecret-image-test
Unexpected error:
    <*errors.StatusError | 0xc0011fa280>: 
    pods "pull-secret-test" is forbidden: error looking up service account pullsecret-image-test/default: serviceaccount "default" not found
    ...
occurred}

perhaps we need something like https://github.com/Azure/ARO-HCP/pull/5973/changes I'll open a separate PR for that

This reverts commit a35dcb5, reversing changes made to 5ca348d.

…ddeletes Revert "Merge pull request #6058 from deads2k/cs-193-changefeed-delete"

Copilot AI review requested due to automatic review settings July 13, 2026 19:57

openshift-ci Bot requested review from Gerd Oberlechner (geoberle) and Jan-Hendrik Boll (janboll) July 13, 2026 19:57

openshift-ci Bot added the approved label Jul 13, 2026

Copilot started reviewing on behalf of David Eads (deads2k) July 13, 2026 19:58 View session

Copilot AI reviewed Jul 13, 2026

View reviewed changes

Rael Garcia (raelga) mentioned this pull request Jul 14, 2026

chore(config): add stgGlobalV2.billingKusto{Name,Sku} to de-hardcode STG billing Kusto (AROSLSRE-1482) #6065

Merged

Copilot AI review requested due to automatic review settings July 14, 2026 12:18

David Eads (deads2k) force-pushed the cs-193-changefeed-delete branch from 1407ce7 to 37f0d66 Compare July 14, 2026 12:18

Copilot started reviewing on behalf of David Eads (deads2k) July 14, 2026 12:19 View session

Copilot AI reviewed Jul 14, 2026

View reviewed changes

Comment thread internal/database/informers/changefeed_list_watch.go

Comment thread internal/database/crud_helpers.go

Comment thread test-integration/backend/changefeed/changefeed_test.go Outdated

Copilot AI review requested due to automatic review settings July 14, 2026 15:07

Copilot started reviewing on behalf of David Eads (deads2k) July 14, 2026 15:08 View session

Copilot AI reviewed Jul 14, 2026

View reviewed changes

Manyanda Chitimbo (machi1990) reviewed Jul 15, 2026

View reviewed changes

Comment thread internal/databasetesting/mock_resources_crud.go

Manyanda Chitimbo (machi1990) reviewed Jul 15, 2026

View reviewed changes

Comment thread internal/database/crud_helpers.go Outdated

Manyanda Chitimbo (machi1990) reviewed Jul 15, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings July 15, 2026 17:46

Copilot started reviewing on behalf of David Eads (deads2k) July 15, 2026 17:46 View session

Copilot AI reviewed Jul 15, 2026

View reviewed changes

Comment thread internal/database/crud_helpers.go

Copilot AI review requested due to automatic review settings July 15, 2026 18:20

Copilot started reviewing on behalf of David Eads (deads2k) July 15, 2026 18:21 View session

Copilot AI reviewed Jul 15, 2026

View reviewed changes

Comment thread internal/database/crud_helpers.go

Comment thread internal/databasetesting/mock_resources_crud.go

openshift-ci Bot added the lgtm label Jul 15, 2026

Copilot AI review requested due to automatic review settings July 15, 2026 19:59

David Eads (deads2k) force-pushed the cs-193-changefeed-delete branch from 570849b to b860da6 Compare July 15, 2026 19:59

openshift-ci Bot removed the lgtm label Jul 15, 2026

Copilot started reviewing on behalf of David Eads (deads2k) July 15, 2026 20:00 View session

openshift-ci Bot added the lgtm label Jul 15, 2026

Copilot AI reviewed Jul 15, 2026

View reviewed changes

openshift-ci Bot assigned Manyanda Chitimbo (machi1990) Jul 16, 2026

openshift-merge-bot Bot merged commit a35dcb5 into Azure:main Jul 16, 2026
16 checks passed

Miguel Soriano (miguelsorianod) mentioned this pull request Jul 16, 2026

Revert "Merge pull request #6058 from deads2k/cs-193-changefeed-delete" #6115

Merged

Steve Kuznetsov (stevekuznetsov) pushed a commit that referenced this pull request Jul 16, 2026

Revert "Merge pull request #6058 from deads2k/cs-193-changefeed-delete"

398ea7d

This reverts commit a35dcb5, reversing changes made to 5ca348d.

Steve Kuznetsov (stevekuznetsov) added a commit that referenced this pull request Jul 16, 2026

Merge pull request #6115 from miguelsorianod/revert-pr-6058-changefee…

10f898c

…ddeletes Revert "Merge pull request #6058 from deads2k/cs-193-changefeed-delete"

Conversation

David Eads (deads2k) commented Jul 13, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

David Eads (deads2k) Jul 14, 2026

Choose a reason for hiding this comment

Uh oh!

David Eads (deads2k) commented Jul 14, 2026

Uh oh!

Uh oh!

Uh oh!

Manyanda Chitimbo (machi1990) left a comment

Choose a reason for hiding this comment

Uh oh!

Matthew Barnes (mbarnes) commented Jul 15, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

David Eads (deads2k) commented Jul 15, 2026

Uh oh!

openshift-ci Bot commented Jul 15, 2026

Uh oh!

David Eads (deads2k) commented Jul 15, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Manyanda Chitimbo (machi1990) commented Jul 16, 2026

Uh oh!

openshift-ci Bot commented Jul 16, 2026

Uh oh!

Manyanda Chitimbo (machi1990) commented Jul 16, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants