Skip to content

Stacks operations got interrupted on pod termination #795

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
project0 opened this issue Jan 21, 2025 · 4 comments · Fixed by #803
Closed

Stacks operations got interrupted on pod termination #795

project0 opened this issue Jan 21, 2025 · 4 comments · Fixed by #803
Assignees
@EronWright
Labels
kind/bug Some behavior is incorrect or out of spec resolution/fixed This issue was fixed

Comments

@project0
Copy link

What happened?

Stack operations got interrupted on pod termination, this can lead to orphan resources as stack operation are not finished. This is especially crucial on ephemeral resources that should be cleaned up on stack deletion, currently it just leaves the resources orphan -.-

"grpc_log": true}
2025-01-21T14:28:14.019Z	INFO	pulumi	^C received; cancelling. If you would like to terminate immediately, press ^C again.
2025-01-21T14:28:14.020Z	INFO	pulumi	Note that terminating immediately may lead to orphaned resources and other 


2025-01-21T14:26:24.292Z	INFO	pulumi	Resources:
2025-01-21T14:26:24.292Z	INFO	pulumi	    + 20 created
2025-01-21T14:26:24.292Z	INFO	pulumi	
2025-01-21T14:26:24.292Z	INFO	pulumi	Duration: 36s
2025-01-21T14:26:26.251Z	INFO	server	up completed	{"summary": {"version":0,"kind":"update","startTime":"2025-01-21T14:25:48.000Z","message":"\"Stack Update (up)\"","environment":{"exec.agent":"pulumi-kubernetes-operator/v2.0.0-beta.3","exec.kind":"auto.local","pulumi.arch":"amd64","pulumi.env.PULUMI_BACKEND_URL":"set","pulumi.env.PULUMI_CONFIG_PASSPHRASE":"set","pulumi.env.PULUMI_DEBUG_COMMANDS":"true","pulumi.flag.diff":"true","pulumi.flag.event-log":"set","pulumi.flag.exec-agent":"set","pulumi.flag.exec-kind":"set","pulumi.flag.message":"set","pulumi.flag.non-interactive":"true","pulumi.flag.skip-preview":"true","pulumi.flag.stack":"set","pulumi.flag.suppress-progress":"true","pulumi.flag.yes":"true","pulumi.os":"linux","pulumi.version":"v3.146.0","stack.environments":"[]","updatePlan":"false"},"config":{"aws:defaultTags":{"Value":"{\"tags\":{\"pulumi-eks-managed\":\"eks-dev\",\"meta:managed-by\":\"pulumi\"}}","Secret":false},"aws:region":{"Value":"eu-central-1","Secret":false},"base:clusterName":{"Value":"eks-dev","Secret":false},"base:domainName":{"Value":"eks-dev.aws.flowers-software.com","Secret":false},"base:envName":{"Value":"dev-env-pr-12","Secret":false}},"result":"succeeded","endTime":"2025-01-21T14:26:24.000Z","resourceChanges":{"create":20}}}
2025-01-21T14:26:26.253Z	INFO	cmd.serve.grpc	finished streaming call with code OK	{"grpc.start_time": "2025-01-21T14:25:47Z", "system": "grpc", "span.kind": "server", "grpc.service": "agent.AutomationService", "grpc.method": "Up", "user.name": "system:serviceaccount:pulumi-kubernetes-operator:controller-manager", "peer.address": "10.32.1.102:58240", "auth.mode": "kubernetes", "user.id": "06eaadc0-3908-44cd-996b-cbd8746e990f", "grpc.code": "OK", "grpc.time_ms": 38864.97}
2025-01-21T14:28:10.513Z	INFO	server	selected a stack	{"name": "dev-env-pr-12-base-shared"}
2025-01-21T14:28:11.369Z	INFO	cmd.serve.grpc	finished unary call with code OK	{"grpc.start_time": "2025-01-21T14:28:10Z", "system": "grpc", "span.kind": "server", "grpc.service": "agent.AutomationService", "grpc.method": "SelectStack", "peer.address": "10.32.1.102:35688", "auth.mode": "kubernetes", "user.id": "06eaadc0-3908-44cd-996b-cbd8746e990f", "user.name": "system:serviceaccount:pulumi-kubernetes-operator:controller-manager", "grpc.code": "OK", "grpc.time_ms": 1151.835}
2025-01-21T14:28:12.489Z	INFO	pulumi	Destroying (dev-env-pr-12-base-shared):
2025-01-21T14:28:12.615Z	INFO	pulumi	
2025-01-21T14:28:13.788Z	INFO	pulumi	@ destroying....
2025-01-21T14:28:13.788Z	INFO	pulumi	 -  aws:cloudwatch:EventTarget flowers-api-event-target-1 deleting (0s) 
2025-01-21T14:28:13.788Z	INFO	pulumi	 -  aws:cloudwatch:EventTarget rm-event-target-1 deleting (0s) 
2025-01-21T14:28:13.788Z	INFO	pulumi	 -  aws:cloudwatch:EventTarget rma-event-target-1 deleting (0s) 
2025-01-21T14:28:14.017Z	INFO	cmd.serve.grpc	shutting down the server
2025-01-21T14:28:14.018Z	INFO	cmd.serve.grpc	[core][Server #1 ListenSocket #2]ListenSocket deleted	{"system": "grpc", "grpc_log": true}
2025-01-21T14:28:14.019Z	INFO	pulumi	^C received; cancelling. If you would like to terminate immediately, press ^C again.
2025-01-21T14:28:14.020Z	INFO	pulumi	Note that terminating immediately may lead to orphaned resources and other inconsistencies.
2025-01-21T14:28:14.020Z	INFO	pulumi	
2025-01-21T14:28:14.061Z	INFO	pulumi	 -  aws:cloudwatch:EventTarget flowers-api-event-target-1 deleted (0.27s) 
2025-01-21T14:28:14.197Z	INFO	pulumi	 -  aws:cloudwatch:EventTarget rma-event-target-1 deleted (0.41s) 
2025-01-21T14:28:14.319Z	INFO	pulumi	 -  aws:cloudwatch:EventTarget rm-event-target-1 deleted (0.53s) 
2025-01-21T14:28:14.813Z	INFO	pulumi	@ destroying....
2025-01-21T14:28:14.814Z	INFO	pulumi	    pulumi:pulumi:Stack base-dev-env-pr-12-base-shared  
2025-01-21T14:28:14.814Z	INFO	pulumi	Resources:
2025-01-21T14:28:14.815Z	INFO	pulumi	    - 3 deleted
2025-01-21T14:28:14.815Z	INFO	pulumi	
2025-01-21T14:28:14.816Z	INFO	pulumi	Duration: 3s

Example

not relevant, that happens with any program with Stack CRD in my case.

Output of pulumi about

n/a

Additional context

I am using argocd applicationset to ramp up a namespace and the program/stack. I guess the pod is terminated by argo in this case. However, there should never be a situation that operations leave undesired stack states (especially on delete).

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).
@project0 project0 added kind/bug Some behavior is incorrect or out of spec needs-triage Needs attention from the triage team labels Jan 21, 2025
@project0 project0 mentioned this issue Jan 23, 2025
Open
@project0
Copy link
Author

project0 commented Jan 23, 2025 •
edited
Loading

I have been digging further into this problem, it looks like the pulumi operator itself replaces/updates the workspace/statefulset during active operations :-(

During the destroy operation it get interrupted by updating the statefulset during the destroy operation, marks it even as fail, and then removes the finalizer from the stack. :-(

Operator log 


2025-01-23T23:13:10.234Z	INFO	Reconciling Stack	{"controller": "stack-controller", "namespace": "flowers-env-dev-env-pr-3", "name": "base-shared", "reconcileID": "db7e2825-5f6b-4349-b376-fb962c9335bd", "revision": "6850741"}
2025-01-23T23:13:10.282Z	INFO	Reconciling Workspace	{"controller": "workspace-controller", "controllerGroup": "auto.pulumi.com", "controllerKind": "Workspace", "Workspace": {"name":"base-shared","namespace":"flowers-env-dev-env-pr-3"}, "namespace": "flowers-env-dev-env-pr-3", "name": "base-shared", "reconcileID": "16c29213-ff2f-418b-8c0e-58e2e3fa16e0", "revision": "6850755"}
2025-01-23T23:13:10.283Z	INFO	Applying StatefulSet	{"controller": "workspace-controller", "controllerGroup": "auto.pulumi.com", "controllerKind": "Workspace", "Workspace": {"name":"base-shared","namespace":"flowers-env-dev-env-pr-3"}, "namespace": "flowers-env-dev-env-pr-3", "name": "base-shared", "reconcileID": "16c29213-ff2f-418b-8c0e-58e2e3fa16e0", "revision": "6850755", "hash": "4369d226bc0e19e69a64bbc09eac8683", "source": {"Generation":3,"ForceRequest":"","Git":null,"Flux":{"Url":"http://pulumi-kubernetes-operator.pulumi-kubernetes-operator:80/programs/flowers-env-dev-env-pr-3/base","Digest":"sha256:881b01cd29493dea25238286408d40305d41083bccdbd467ad0111beb4b9a086","Dir":""},"Local":null}}
2025-01-23T23:13:10.303Z	INFO	Status updated	{"controller": "stack-controller", "namespace": "flowers-env-dev-env-pr-3", "name": "base-shared", "reconcileID": "db7e2825-5f6b-4349-b376-fb962c9335bd", "revision": "6850756", "observedGeneration": 3, "observedReconcileRequest": "", "lastUpdate": {"generation":2,"name":"base-shared-194956831f9","type":"up","state":"succeeded","lastAttemptedCommit":"sha256:881b01cd29493dea25238286408d40305d41083bccdbd467ad0111beb4b9a086","lastSuccessfulCommit":"sha256:881b01cd29493dea25238286408d40305d41083bccdbd467ad0111beb4b9a086","lastResyncTime":"2025-01-23T23:04:15Z"}, "currentUpdate": null, "conditions": [{"type":"Ready","status":"False","lastTransitionTime":"2025-01-23T23:13:10Z","reason":"NotReadyInProgress","message":"reconciliation is in progress"},{"type":"Reconciling","status":"True","lastTransitionTime":"2025-01-23T23:13:10Z","reason":"StackProcessing","message":"waiting for workspace readiness"}]}
2025-01-23T23:13:10.319Z	INFO	Connecting to workspace pod	{"controller": "workspace-controller", "controllerGroup": "auto.pulumi.com", "controllerKind": "Workspace", "Workspace": {"name":"base-shared","namespace":"flowers-env-dev-env-pr-3"}, "namespace": "flowers-env-dev-env-pr-3", "name": "base-shared", "reconcileID": "16c29213-ff2f-418b-8c0e-58e2e3fa16e0", "revision": "6850755", "addr": "base-shared-workspace.flowers-env-dev-env-pr-3:50051"}
2025-01-23T23:13:10.328Z	INFO	Connected to workspace pod	{"controller": "workspace-controller", "controllerGroup": "auto.pulumi.com", "controllerKind": "Workspace", "Workspace": {"name":"base-shared","namespace":"flowers-env-dev-env-pr-3"}, "namespace": "flowers-env-dev-env-pr-3", "name": "base-shared", "reconcileID": "16c29213-ff2f-418b-8c0e-58e2e3fa16e0", "revision": "6850755", "addr": "base-shared-workspace.flowers-env-dev-env-pr-3:50051"}
2025-01-23T23:13:10.328Z	INFO	Ready	{"controller": "workspace-controller", "controllerGroup": "auto.pulumi.com", "controllerKind": "Workspace", "Workspace": {"name":"base-shared","namespace":"flowers-env-dev-env-pr-3"}, "namespace": "flowers-env-dev-env-pr-3", "name": "base-shared", "reconcileID": "16c29213-ff2f-418b-8c0e-58e2e3fa16e0", "revision": "6850755"}
2025-01-23T23:13:10.350Z	INFO	Status updated	{"controller": "workspace-controller", "controllerGroup": "auto.pulumi.com", "controllerKind": "Workspace", "Workspace": {"name":"base-shared","namespace":"flowers-env-dev-env-pr-3"}, "namespace": "flowers-env-dev-env-pr-3", "name": "base-shared", "reconcileID": "16c29213-ff2f-418b-8c0e-58e2e3fa16e0", "revision": "6850757", "observedGeneration": 3, "address": "base-shared-workspace.flowers-env-dev-env-pr-3:50051", "conditions": [{"type":"Ready","status":"True","observedGeneration":3,"lastTransitionTime":"2025-01-23T23:03:22Z","reason":"Succeeded","message":""}]}
2025-01-23T23:13:10.352Z	INFO	Reconciling Stack	{"controller": "stack-controller", "namespace": "flowers-env-dev-env-pr-3", "name": "base-shared", "reconcileID": "8efa0ffe-bb59-427e-a2f1-15dd630cb302", "revision": "6850756"}
2025-01-23T23:13:10.402Z	INFO	Status updated	{"controller": "stack-controller", "namespace": "flowers-env-dev-env-pr-3", "name": "base-shared", "reconcileID": "8efa0ffe-bb59-427e-a2f1-15dd630cb302", "revision": "6850760", "observedGeneration": 3, "observedReconcileRequest": "", "lastUpdate": {"generation":2,"name":"base-shared-194956831f9","type":"up","state":"succeeded","lastAttemptedCommit":"sha256:881b01cd29493dea25238286408d40305d41083bccdbd467ad0111beb4b9a086","lastSuccessfulCommit":"sha256:881b01cd29493dea25238286408d40305d41083bccdbd467ad0111beb4b9a086","lastResyncTime":"2025-01-23T23:04:15Z"}, "currentUpdate": {"generation":3,"name":"base-shared-1949570e0eb","commit":"sha256:881b01cd29493dea25238286408d40305d41083bccdbd467ad0111beb4b9a086"}, "conditions": [{"type":"Ready","status":"False","lastTransitionTime":"2025-01-23T23:13:10Z","reason":"NotReadyInProgress","message":"reconciliation is in progress"},{"type":"Reconciling","status":"True","lastTransitionTime":"2025-01-23T23:13:10Z","reason":"StackProcessing","message":"stack is being processed"}]}
2025-01-23T23:13:10.412Z	INFO	Reconciling Update	{"controller": "update-controller", "controllerGroup": "auto.pulumi.com", "controllerKind": "Update", "Update": {"name":"base-shared-1949570e0eb","namespace":"flowers-env-dev-env-pr-3"}, "namespace": "flowers-env-dev-env-pr-3", "name": "base-shared-1949570e0eb", "reconcileID": "6d33e81b-e5d9-46b7-9e16-7740573a493a", "revision": "6850762"}
2025-01-23T23:13:10.412Z	INFO	Updating the status	{"controller": "update-controller", "controllerGroup": "auto.pulumi.com", "controllerKind": "Update", "Update": {"name":"base-shared-1949570e0eb","namespace":"flowers-env-dev-env-pr-3"}, "namespace": "flowers-env-dev-env-pr-3", "name": "base-shared-1949570e0eb", "reconcileID": "6d33e81b-e5d9-46b7-9e16-7740573a493a", "revision": "6850762"}
2025-01-23T23:13:10.438Z	INFO	Status updated	{"controller": "update-controller", "controllerGroup": "auto.pulumi.com", "controllerKind": "Update", "Update": {"name":"base-shared-1949570e0eb","namespace":"flowers-env-dev-env-pr-3"}, "namespace": "flowers-env-dev-env-pr-3", "name": "base-shared-1949570e0eb", "reconcileID": "6d33e81b-e5d9-46b7-9e16-7740573a493a", "revision": "6850762", "revision": "6850763", "observedGeneration": 1, "message": "", "conditions": [{"type":"Progressing","status":"True","observedGeneration":1,"lastTransitionTime":"2025-01-23T23:13:10Z","reason":"Progressing","message":""},{"type":"Failed","status":"False","observedGeneration":1,"lastTransitionTime":"2025-01-23T23:13:10Z","reason":"Progressing","message":""},{"type":"Complete","status":"False","observedGeneration":1,"lastTransitionTime":"2025-01-23T23:13:10Z","reason":"Progressing","message":""}]}
2025-01-23T23:13:10.438Z	INFO	Connecting	{"controller": "update-controller", "controllerGroup": "auto.pulumi.com", "controllerKind": "Update", "Update": {"name":"base-shared-1949570e0eb","namespace":"flowers-env-dev-env-pr-3"}, "namespace": "flowers-env-dev-env-pr-3", "name": "base-shared-1949570e0eb", "reconcileID": "6d33e81b-e5d9-46b7-9e16-7740573a493a", "revision": "6850762", "revision": "6850763", "addr": "base-shared-workspace.flowers-env-dev-env-pr-3:50051"}
2025-01-23T23:13:10.442Z	INFO	Selecting the stack	{"controller": "update-controller", "controllerGroup": "auto.pulumi.com", "controllerKind": "Update", "Update": {"name":"base-shared-1949570e0eb","namespace":"flowers-env-dev-env-pr-3"}, "namespace": "flowers-env-dev-env-pr-3", "name": "base-shared-1949570e0eb", "reconcileID": "6d33e81b-e5d9-46b7-9e16-7740573a493a", "revision": "6850762", "revision": "6850763", "stackName": "dev-env-pr-3-base-shared"}
2025-01-23T23:13:11.680Z	INFO	Applying the update	{"controller": "update-controller", "controllerGroup": "auto.pulumi.com", "controllerKind": "Update", "Update": {"name":"base-shared-1949570e0eb","namespace":"flowers-env-dev-env-pr-3"}, "namespace": "flowers-env-dev-env-pr-3", "name": "base-shared-1949570e0eb", "reconcileID": "6d33e81b-e5d9-46b7-9e16-7740573a493a", "revision": "6850762", "revision": "6850763", "type": "destroy"}
2025-01-23T23:13:11.680Z	INFO	Executing destroy operation	{"controller": "update-controller", "controllerGroup": "auto.pulumi.com", "controllerKind": "Update", "Update": {"name":"base-shared-1949570e0eb","namespace":"flowers-env-dev-env-pr-3"}, "namespace": "flowers-env-dev-env-pr-3", "name": "base-shared-1949570e0eb", "reconcileID": "6d33e81b-e5d9-46b7-9e16-7740573a493a", "revision": "6850762", "request": "message:\"Stack Update (destroy)\""}
2025-01-23T23:13:12.134Z	INFO	Reconciling Workspace	{"controller": "workspace-controller", "controllerGroup": "auto.pulumi.com", "controllerKind": "Workspace", "Workspace": {"name":"base-shared","namespace":"flowers-env-dev-env-pr-3"}, "namespace": "flowers-env-dev-env-pr-3", "name": "base-shared", "reconcileID": "4d805630-1fb0-42cf-b681-c36908c96d9f", "revision": "6850869"}
2025-01-23T23:13:12.149Z	INFO	Status updated	{"controller": "workspace-controller", "controllerGroup": "auto.pulumi.com", "controllerKind": "Workspace", "Workspace": {"name":"base-shared","namespace":"flowers-env-dev-env-pr-3"}, "namespace": "flowers-env-dev-env-pr-3", "name": "base-shared", "reconcileID": "4d805630-1fb0-42cf-b681-c36908c96d9f", "revision": "6850870", "observedGeneration": 4, "address": "base-shared-workspace.flowers-env-dev-env-pr-3:50051", "conditions": [{"type":"Ready","status":"False","observedGeneration":4,"lastTransitionTime":"2025-01-23T23:03:22Z","reason":"Deleting","message":"Workspace is being deleted"}]}
2025-01-23T23:13:13.735Z	INFO	Reconciling Workspace	{"controller": "workspace-controller", "controllerGroup": "auto.pulumi.com", "controllerKind": "Workspace", "Workspace": {"name":"base-shared","namespace":"flowers-env-dev-env-pr-3"}, "namespace": "flowers-env-dev-env-pr-3", "name": "base-shared", "reconcileID": "29d9c6e0-76ad-43c0-8934-47735fdda3aa", "revision": "6850870"}
2025-01-23T23:13:14.182Z	INFO	Reconciling Update	{"controller": "update-controller", "controllerGroup": "auto.pulumi.com", "controllerKind": "Update", "Update": {"name":"base-shared-194956831f9","namespace":"flowers-env-dev-env-pr-3"}, "namespace": "flowers-env-dev-env-pr-3", "name": "base-shared-194956831f9", "reconcileID": "e80338fe-2b30-49c8-a293-5fca588ccb1d", "revision": "6850934"}
2025-01-23T23:13:15.538Z	INFO	Update complete	{"controller": "update-controller", "controllerGroup": "auto.pulumi.com", "controllerKind": "Update", "Update": {"name":"base-shared-1949570e0eb","namespace":"flowers-env-dev-env-pr-3"}, "namespace": "flowers-env-dev-env-pr-3", "name": "base-shared-1949570e0eb", "reconcileID": "6d33e81b-e5d9-46b7-9e16-7740573a493a", "revision": "6850762", "result": "stdout:\"Destroying (dev-env-pr-3-base-shared):\\n\\n@ destroying....\\n -  aws:cloudwatch:EventTarget flowers-api-event-target-1 deleting (0s) \\n -  aws:cloudwatch:EventTarget rma-event-target-1 deleting (0s) \\n -  aws:cloudwatch:EventTarget rm-event-target-1 deleting (0s) \\n -  aws:cloudwatch:EventTarget flowers-api-event-target-1 deleted (0.22s) \\n -  aws:cloudwatch:EventTarget rma-event-target-1 deleted (0.28s) \\n -  aws:cloudwatch:EventTarget rm-event-target-1 deleted (0.39s) \\n^C received; cancelling. If you would like to terminate immediately, press ^C again.\\nNote that terminating immediately may lead to orphaned resources and other inconsistencies.\\n\\n    pulumi:pulumi:Stack base-dev-env-pr-3-base-shared  \\nResources:\\n    - 3 deleted\\n\\nDuration: 2s\\n\\nThe resources in the stack have been deleted, but the history and configuration associated with the stack are still maintained. \\nIf you want to remove the stack completely, run `pulumi stack rm dev-env-pr-3-base-shared`.\\n\"  summary:{start_time:{seconds:1737673992}  end_time:{seconds:1737673994}  result:\"succeeded\"  message:\"\\\"Stack Update (destroy)\\\"\"}"}
2025-01-23T23:13:15.546Z	ERROR	Reconciler error	{"controller": "update-controller", "controllerGroup": "auto.pulumi.com", "controllerKind": "Update", "Update": {"name":"base-shared-1949570e0eb","namespace":"flowers-env-dev-env-pr-3"}, "namespace": "flowers-env-dev-env-pr-3", "name": "base-shared-1949570e0eb", "reconcileID": "6d33e81b-e5d9-46b7-9e16-7740573a493a", "error": "updating status: Operation cannot be fulfilled on updates.auto.pulumi.com \"base-shared-1949570e0eb\": the object has been modified; please apply your changes to the latest version and try again"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:324
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222
2025-01-23T23:13:15.546Z	INFO	Reconciling Update	{"controller": "update-controller", "controllerGroup": "auto.pulumi.com", "controllerKind": "Update", "Update": {"name":"base-shared-1949570e0eb","namespace":"flowers-env-dev-env-pr-3"}, "namespace": "flowers-env-dev-env-pr-3", "name": "base-shared-1949570e0eb", "reconcileID": "7aa9010d-1a59-4d28-9251-d4158cd7b33f", "revision": "6850911"}
2025-01-23T23:13:15.546Z	INFO	was progressing; marking as failed	{"controller": "update-controller", "controllerGroup": "auto.pulumi.com", "controllerKind": "Update", "Update": {"name":"base-shared-1949570e0eb","namespace":"flowers-env-dev-env-pr-3"}, "namespace": "flowers-env-dev-env-pr-3", "name": "base-shared-1949570e0eb", "reconcileID": "7aa9010d-1a59-4d28-9251-d4158cd7b33f", "revision": "6850911"}
2025-01-23T23:13:15.564Z	INFO	Status updated	{"controller": "update-controller", "controllerGroup": "auto.pulumi.com", "controllerKind": "Update", "Update": {"name":"base-shared-1949570e0eb","namespace":"flowers-env-dev-env-pr-3"}, "namespace": "flowers-env-dev-env-pr-3", "name": "base-shared-1949570e0eb", "reconcileID": "7aa9010d-1a59-4d28-9251-d4158cd7b33f", "revision": "6850911", "revision": "6850970", "observedGeneration": 2, "message": "", "conditions": [{"type":"Progressing","status":"False","observedGeneration":2,"lastTransitionTime":"2025-01-23T23:13:10Z","reason":"Failed","message":""},{"type":"Failed","status":"True","observedGeneration":2,"lastTransitionTime":"2025-01-23T23:13:10Z","reason":"Aborted","message":""},{"type":"Complete","status":"True","observedGeneration":2,"lastTransitionTime":"2025-01-23T23:13:10Z","reason":"Failed","message":""}]}
2025-01-23T23:13:15.564Z	INFO	Reconciling Update	{"controller": "update-controller", "controllerGroup": "auto.pulumi.com", "controllerKind": "Update", "Update": {"name":"base-shared-1949570e0eb","namespace":"flowers-env-dev-env-pr-3"}, "namespace": "flowers-env-dev-env-pr-3", "name": "base-shared-1949570e0eb", "reconcileID": "f7dae7c0-f0ec-4d53-a754-f6aeefe1332c", "revision": "6850911"}
2025-01-23T23:13:15.564Z	INFO	was progressing; marking as failed	{"controller": "update-controller", "controllerGroup": "auto.pulumi.com", "controllerKind": "Update", "Update": {"name":"base-shared-1949570e0eb","namespace":"flowers-env-dev-env-pr-3"}, "namespace": "flowers-env-dev-env-pr-3", "name": "base-shared-1949570e0eb", "reconcileID": "f7dae7c0-f0ec-4d53-a754-f6aeefe1332c", "revision": "6850911"}
2025-01-23T23:13:15.564Z	INFO	Reconciling Stack	{"controller": "stack-controller", "namespace": "flowers-env-dev-env-pr-3", "name": "base-shared", "reconcileID": "f7422f88-b784-4058-b918-79237a1b28a7", "revision": "6850760"}
2025-01-23T23:13:15.564Z	INFO	Failed to update Stack	{"controller": "stack-controller", "namespace": "flowers-env-dev-env-pr-3", "name": "base-shared", "reconcileID": "f7422f88-b784-4058-b918-79237a1b28a7", "revision": "6850760", "Stack.Name": "dev-env-pr-3-base-shared", "Message": ""}
2025-01-23T23:13:15.573Z	ERROR	Reconciler error	{"controller": "update-controller", "controllerGroup": "auto.pulumi.com", "controllerKind": "Update", "Update": {"name":"base-shared-1949570e0eb","namespace":"flowers-env-dev-env-pr-3"}, "namespace": "flowers-env-dev-env-pr-3", "name": "base-shared-1949570e0eb", "reconcileID": "f7dae7c0-f0ec-4d53-a754-f6aeefe1332c", "error": "updating status: Operation cannot be fulfilled on updates.auto.pulumi.com \"base-shared-1949570e0eb\": the object has been modified; please apply your changes to the latest version and try again"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:324
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222
2025-01-23T23:13:15.573Z	INFO	Reconciling Update	{"controller": "update-controller", "controllerGroup": "auto.pulumi.com", "controllerKind": "Update", "Update": {"name":"base-shared-1949570e0eb","namespace":"flowers-env-dev-env-pr-3"}, "namespace": "flowers-env-dev-env-pr-3", "name": "base-shared-1949570e0eb", "reconcileID": "28b5c7c0-c965-4da4-a4d3-1eeea06d300a", "revision": "6850970"}
2025-01-23T23:13:15.575Z	INFO	Stack was destroyed; finalizing now.	{"controller": "stack-controller", "namespace": "flowers-env-dev-env-pr-3", "name": "base-shared", "reconcileID": "f7422f88-b784-4058-b918-79237a1b28a7", "revision": "6850760"}
2025-01-23T23:13:15.578Z	INFO	Reconciling Update	{"controller": "update-controller", "controllerGroup": "auto.pulumi.com", "controllerKind": "Update", "Update": {"name":"base-shared-1949570e0eb","namespace":"flowers-env-dev-env-pr-3"}, "namespace": "flowers-env-dev-env-pr-3", "name": "base-shared-1949570e0eb", "reconcileID": "52a8362b-7ffd-462d-8b8f-ca1f7acf5fb3", "revision": "6850970"}
2025-01-23T23:13:15.596Z	INFO	Status updated	{"controller": "stack-controller", "namespace": "flowers-env-dev-env-pr-3", "name": "base-shared", "reconcileID": "f7422f88-b784-4058-b918-79237a1b28a7", "revision": "6850972", "observedGeneration": 3, "observedReconcileRequest": "", "lastUpdate": {"generation":3,"name":"base-shared-1949570e0eb","type":"destroy","state":"failed","lastAttemptedCommit":"sha256:881b01cd29493dea25238286408d40305d41083bccdbd467ad0111beb4b9a086","lastSuccessfulCommit":"sha256:881b01cd29493dea25238286408d40305d41083bccdbd467ad0111beb4b9a086","lastResyncTime":"2025-01-23T23:13:15Z"}, "currentUpdate": null, "conditions": [{"type":"Ready","status":"False","lastTransitionTime":"2025-01-23T23:13:10Z","reason":"NotReadyInProgress","message":"reconciliation is in progress"},{"type":"Reconciling","status":"True","lastTransitionTime":"2025-01-23T23:13:10Z","reason":"RetryingAfterFailure","message":"0 update failure(s)"}]}
2025-01-23T23:13:15.628Z	INFO	Reconciling Stack	{"controller": "stack-controller", "namespace": "flowers-env-dev-env-pr-3", "name": "base-shared", "reconcileID": "7b9d71c5-591b-4a37-a827-acea35007f75", "revision": "6850974"}
2025-01-23T23:13:16.009Z	INFO	Reconciling Stack	{"controller": "stack-controller", "namespace": "flowers-env-dev-env-pr-3", "name": "base-shared", "reconcileID": "ccf0a920-154b-4a8f-90e1-f1ef73a8d2c9", "revision": "6850974"}
2025-01-23T23:13:16.215Z	INFO	Reconciling Program	{"controller": "program-controller", "controllerGroup": "pulumi.com", "controllerKind": "Program", "Program": {"name":"base","namespace":"flowers-env-dev-env-pr-3"}, "namespace": "flowers-env-dev-env-pr-3", "name": "base", "reconcileID": "29eeeb1b-57bf-428d-ae14-3318e2645036"}
2025-01-23T23:13:16.215Z	INFO	Calculating digest hash for Program artifact	{"controller": "program-controller", "controllerGroup": "pulumi.com", "controllerKind": "Program", "Program": {"name":"base","namespace":"flowers-env-dev-env-pr-3"}, "namespace": "flowers-env-dev-env-pr-3", "name": "base", "reconcileID": "29eeeb1b-57bf-428d-ae14-3318e2645036"}
2025-01-23T23:13:16.219Z	INFO	Updating Program status	{"controller": "program-controller", "controllerGroup": "pulumi.com", "controllerKind": "Program", "Program": {"name":"base","namespace":"flowers-env-dev-env-pr-3"}, "namespace": "flowers-env-dev-env-pr-3", "name": "base", "reconcileID": "29eeeb1b-57bf-428d-ae14-3318e2645036"}
2025-01-23T23:13:16.236Z	ERROR	unable to update Program status	{"controller": "program-controller", "controllerGroup": "pulumi.com", "controllerKind": "Program", "Program": {"name":"base","namespace":"flowers-env-dev-env-pr-3"}, "namespace": "flowers-env-dev-env-pr-3", "name": "base", "reconcileID": "29eeeb1b-57bf-428d-ae14-3318e2645036", "error": "Operation cannot be fulfilled on programs.pulumi.com \"base\": StorageError: invalid object, Code: 4, Key: /registry/pulumi.com/programs/flowers-env-dev-env-pr-3/base, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 858daff8-cc1d-4f7e-a4fa-d685f71941cd, UID in object meta: "}
github.com/pulumi/pulumi-kubernetes-operator/v2/operator/internal/controller/pulumi.(*ProgramReconciler).Reconcile
	/go/operator/internal/controller/pulumi/program_controller.go:255
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222
2025-01-23T23:13:16.236Z	ERROR	Reconciler error	{"controller": "program-controller", "controllerGroup": "pulumi.com", "controllerKind": "Program", "Program": {"name":"base","namespace":"flowers-env-dev-env-pr-3"}, "namespace": "flowers-env-dev-env-pr-3", "name": "base", "reconcileID": "29eeeb1b-57bf-428d-ae14-3318e2645036", "error": "Operation cannot be fulfilled on programs.pulumi.com \"base\": StorageError: invalid object, Code: 4, Key: /registry/pulumi.com/programs/flowers-env-dev-env-pr-3/base, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 858daff8-cc1d-4f7e-a4fa-d685f71941cd, UID in object meta: "}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:324
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222
2025-01-23T23:13:16.239Z	INFO	Reconciling Program	{"controller": "program-controller", "controllerGroup": "pulumi.com", "controllerKind": "Program", "Program": {"name":"base","namespace":"flowers-env-dev-env-pr-3"}, "namespace": "flowers-env-dev-env-pr-3", "name": "base", "reconcileID": "39869809-65e8-45b6-b9b2-61c320d36185"}
2025-01-23T23:13:16.241Z	INFO	Reconciling Program	{"controller": "program-controller", "controllerGroup": "pulumi.com", "controllerKind": "Program", "Program": {"name":"base","namespace":"flowers-env-dev-env-pr-3"}, "namespace": "flowers-env-dev-env-pr-3", "name": "base", "reconcileID": "b516e754-921c-41f2-aabd-5237200bbfcd"}

workspace log 

2025-01-23T23:13:10.811Z	INFO	server	selected a stack	{"name": "dev-env-pr-3-base-shared"}
2025-01-23T23:13:11.679Z	INFO	cmd.serve.grpc	finished unary call with code OK	{"grpc.start_time": "2025-01-23T23:13:10Z", "system": "grpc", "span.kind": "server", "grpc.service": "agent.AutomationService", "grpc.method": "SelectStack", "user.id": "06eaadc0-3908-44cd-996b-cbd8746e990f", "user.name": "system:serviceaccount:pulumi-kubernetes-operator:controller-manager", "peer.address": "10.32.1.123:35814", "auth.mode": "kubernetes", "grpc.code": "OK", "grpc.time_ms": 1236.626}
2025-01-23T23:13:12.776Z	INFO	pulumi	Destroying (dev-env-pr-3-base-shared):
2025-01-23T23:13:12.857Z	INFO	pulumi
2025-01-23T23:13:13.971Z	INFO	pulumi	@ destroying....
2025-01-23T23:13:13.972Z	INFO	pulumi	 -  aws:cloudwatch:EventTarget flowers-api-event-target-1 deleting (0s)
2025-01-23T23:13:13.972Z	INFO	pulumi	 -  aws:cloudwatch:EventTarget rma-event-target-1 deleting (0s)
2025-01-23T23:13:13.973Z	INFO	pulumi	 -  aws:cloudwatch:EventTarget rm-event-target-1 deleting (0s)
2025-01-23T23:13:14.184Z	INFO	pulumi	 -  aws:cloudwatch:EventTarget flowers-api-event-target-1 deleted (0.22s)
2025-01-23T23:13:14.254Z	INFO	pulumi	 -  aws:cloudwatch:EventTarget rma-event-target-1 deleted (0.28s)
2025-01-23T23:13:14.362Z	INFO	pulumi	 -  aws:cloudwatch:EventTarget rm-event-target-1 deleted (0.39s)
2025-01-23T23:13:14.577Z	INFO	cmd.serve.grpc	shutting down the server
2025-01-23T23:13:14.577Z	INFO	cmd.serve.grpc	[core][Server #1 ListenSocket #2]ListenSocket deleted	{"system": "grpc", "grpc_log": true}
2025-01-23T23:13:14.580Z	INFO	pulumi	^C received; cancelling. If you would like to terminate immediately, press ^C again.
2025-01-23T23:13:14.580Z	INFO	pulumi	Note that terminating immediately may lead to orphaned resources and other inconsistencies.
2025-01-23T23:13:14.581Z	INFO	pulumi
2025-01-23T23:13:14.734Z	INFO	pulumi	    pulumi:pulumi:Stack base-dev-env-pr-3-base-shared
2025-01-23T23:13:14.735Z	INFO	pulumi	Resources:
2025-01-23T23:13:14.735Z	INFO	pulumi	    - 3 deleted
2025-01-23T23:13:14.736Z	INFO	pulumi
2025-01-23T23:13:14.736Z	INFO	pulumi	Duration: 2s
2025-01-23T23:13:14.736Z	INFO	pulumi
2025-01-23T23:13:14.904Z	INFO	pulumi	The resources in the stack have been deleted, but the history and configuration associated with the stack are still maintained.
2025-01-23T23:13:14.904Z	INFO	pulumi	If you want to remove the stack completely, run `pulumi stack rm dev-env-pr-3-base-shared`.
2025-01-23T23:13:15.537Z	INFO	server	destroy completed	{"summary": {"version":0,"kind":"destroy","startTime":"2025-01-23T23:13:12.000Z","message":"\"Stack Update (destroy)\"","environment":{"exec.agent":"pulumi-kubernetes-operator/v2.0.0-beta.3","exec.kind":"auto.local","pulumi.arch":"arm64","pulumi.env.PULUMI_BACKEND_URL":"set","pulumi.env.PULUMI_CONFIG_PASSPHRASE":"set","pulumi.env.PULUMI_DEBUG_COMMANDS":"true","pulumi.flag.event-log":"set","pulumi.flag.exec-agent":"set","pulumi.flag.exec-kind":"set","pulumi.flag.message":"set","pulumi.flag.non-interactive":"true","pulumi.flag.skip-preview":"true","pulumi.flag.stack":"set","pulumi.flag.yes":"true","pulumi.os":"linux","pulumi.version":"v3.147.0","stack.environments":"[]","updatePlan":"false"},"config":{"aws:defaultTags":{"Value":"{\"tags\":{\"pulumi-eks-managed\":\"eks-dev\",\"meta:managed-by\":\"pulumi\"}}","Secret":false},"aws:region":{"Value":"eu-central-1","Secret":false},"base:clusterName":{"Value":"eks-dev","Secret":false},"base:domainName":{"Value":"eks-dev.aws.flowers-software.com","Secret":false},"base:envName":{"Value":"dev-env-pr-3","Secret":false}},"result":"succeeded","endTime":"2025-01-23T23:13:14.000Z","resourceChanges":{"delete":3}}}
2025-01-23T23:13:15.538Z	INFO	cmd.serve.grpc	finished streaming call with code OK	{"grpc.start_time": "2025-01-23T23:13:11Z", "system": "grpc", "span.kind": "server", "grpc.service": "agent.AutomationService", "grpc.method": "Destroy", "peer.address": "10.32.1.123:35814", "auth.mode": "kubernetes", "user.id": "06eaadc0-3908-44cd-996b-cbd8746e990f", "user.name": "system:serviceaccount:pulumi-kubernetes-operator:controller-manager", "grpc.code": "OK", "grpc.time_ms": 3856.216}
2025-01-23T23:13:15.539Z	INFO	cmd.serve	server stopped

@rquitales
Copy link
Member

Thanks for looking into and reporting this issue and providing detailed logs. We'll look further into resolving this issue.

@rquitales rquitales removed the needs-triage Needs attention from the triage team label Jan 29, 2025
@EronWright
Copy link
Contributor

I wonder if this has the same root cause as #786. The stack controller generally does nothing when there's an active update, so a more low-level perturbation of the workspace seems to be happening. Thanks again for the tremendous reports, stay tuned.

@EronWright EronWright self-assigned this Feb 4, 2025
@EronWright EronWright mentioned this issue Feb 5, 2025
Merged
@EronWright
Copy link
Contributor

@project0 I have a PR ready, feel free to comment.

@pulumi-bot pulumi-bot added resolution/fixed This issue was fixed labels Feb 6, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@EronWright EronWright

Labels
kind/bug Some behavior is incorrect or out of spec resolution/fixed This issue was fixed
Projects
None yet
Milestone
No milestone
4 participants
@EronWright @project0 @rquitales @pulumi-bot