Skip to content
This repository has been archived by the owner on Jan 8, 2024. It is now read-only.

If deployment destroy errors, waypoint should forget the deployment #4464

Open
izaaklauer opened this issue Jan 30, 2023 · 0 comments
Open
Labels
bug Something isn't working core jira Will add an Issue to Jira

Comments

@izaaklauer
Copy link
Contributor

izaaklauer commented Jan 30, 2023

Describe the bug

If waypoint deployment destroy fails (example: #4461), waypoint deletes the record of the deployment, forgetting the deployment exists and orphaning those resources on the platform.

Example:

$ waypoint deployment destroy 11

» Destroying deployments for acmeapp1

» 1 deployments will be destroyed.
  Destroying deployment: 01GQV017QJ421ZX88CWK3TVRGR

» Operation is queued waiting for job "01GR1PCP8V1NZK9DGKTR3Z91E8". Waiting for runner assignment...
  If you interrupt this command, the job will still run in the background.
  Performing operation on "aws-ecs" with runner profile "ecs-01GQJJP4GH4PXEDHNZPY8ZA6W7"

» Cloning data from Git
         URL: https://api.hashicorp.cloud/waypoint/2022-04-21/github/clone/hashicorp/waypoint-learn-terraform-aws-ecs
         Ref: main
        Auth: username/password
  Git Commit: 511720182f674ce9fc06eea5d528c64ffaee682c
   Timestamp: 2023-01-28 01:34:45 +0000 UTC
     Message: better cpu value
  
  mem is probably wrong though!
  
✓ Running deployment destroy v11
❌ Destroying ecs deployment...
✓ Deleted service acmeapp1-01GQV017X6Q8YXP367KCWG1
❌ Deleting target group acmeapp1-01GQV017X6Q8YXP367KCWG1
! Error destroying deployment 11: rpc error: code = Internal desc = failed to destroy all resources for deployment: rpc error: code = Internal desc = failed to delete target group
acmeapp1-01GQV017X6Q8YXP367KCWG1 (ARN: "arn:aws:elasticloadbalancing:us-east-1:863953081260:targetgroup/acmeapp1-01GQV017X6Q8YXP367KCWG1/586fac67bd05f7f3"): ResourceInUse: Target group
'arn:aws:elasticloadbalancing:us-east-1:863953081260:targetgroup/acmeapp1-01GQV017X6Q8YXP367KCWG1/586fac67bd05f7f3' is currently in use by a listener or a rule
  	status code: 400, request id: 46ac44b6-905f-439b-ad57-44e132d6deea

$ waypoint deployment list

» acmeapp1
No deployments found for application "acmeapp1"

Steps to Reproduce

  • Make a deployment
  • Force the deployment to fail, either by exploiting a bug, making an alpha build that always throws an error, or manipulating the platform to make it impossible for waypoint to destroy (e.g. turn deletion protection on for an important resource)
  • Observe that despite the error, the deployment is gone in waypoint!

Expected behavior

It would be much better if waypoint left the deployment around, so I could attempt to delete it again at a future date, after maybe addressing the error in some way.

If waypoint tries to delete the resources and they don't exist, it should throw some kind of warning, but succeed in deleting the deployment

There are times when a user might need the existing behavior, if for example there's a persistent bug in destroying deployments and they just want the record gone from the db. In that case, we could in the future add some kind of waypoint deployment delete -force or -forget flag, but I don't think this should be the default case.

Waypoint Platform Versions
Additional version and platform information to help triage the issue if
applicable:

  • Waypoint CLI Version: 0.10.5
  • Waypoint Server Platform and Version: HCP
  • Waypoint Plugin: demonstrated with ecs, but likely core behavior needs to change
@izaaklauer izaaklauer added new jira Will add an Issue to Jira core bug Something isn't working and removed new labels Jan 30, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Something isn't working core jira Will add an Issue to Jira
Projects
None yet
Development

No branches or pull requests

1 participant