(aws-cdk): changeset fails with Description attribute mismatch

[shellscape]

Upon deploying a stack which has not seen any changes to the local CDK code via cdk deploy, we receive a message of:

ChangeSet cdk-deploy-change-set cannot be created due to a mismatch with existing attribute Description

My first stop was to synth and output the template for the failing stack. The template only had one resource which had a Description field. The value in the template matched perfectly to what had been in the code for some time prior, and the Description on the resource in the AWS Console.

Next, I went into the CloudFormation Stack for the failing stack and took a look at the changesets. For each of our failing stacks, there was a pending changeset. Each changeset had JSON changes which looked like this:

[
  {
    "resourceChange": {
      "logicalResourceId": "CDKMetadata",
      "action": "Modify",
      "physicalResourceId": "6fffac30-9bfd-11eb-9c43-0e0748486565",
      "resourceType": "AWS::CDK::Metadata",
      "replacement": "Conditional",
      "moduleInfo": null,
      "details": [
        {
          "target": {
            "name": "Analytics",
            "requiresRecreation": "Conditionally",
            "attribute": "Properties"
          },
          "causingEntity": null,
          "evaluation": "Static",
          "changeSource": "DirectModification"
        }
      ],
      "changeSetId": null,
      "scope": [
        "Properties"
      ]
    },
    "hookInvocationCount": null,
    "type": "Resource"
  }
]

My initial thought was to delete the pending changesets for the failing services, thinking that maybe they were left over from some interrupted deploy and were causing the conflict. Deleting the changesets just resulted in new pending changesets from subsequent deploy attempts.

Next, I figured I would manually execute the changesets in the failing stacks to see what happened. Every changeset in every failing stack, suffering from the error above, executed successfully when triggered manually in the AWS Console. After manually executing, all failing stacks succeeded with (no changes) in their success messages via cdk deploy

The only possible correlation that I could find in the code here was:

aws-cdk/packages/aws-cdk/lib/api/deploy-stack.ts

Line 250 in 9e421e5

Description: `CDK Changeset for execution ${executionId}`,

Other than that, I'm at a loss.

Reproduction Steps

I'm honestly not sure how to reproduce this. After we initially solved the problem by manually executing the changesets, everything seemed to work as expected.

What did you expect to happen?

Successful deploys.

What actually happened?

Please see above

Environment

• CDK CLI Version : 1.98.0
• Framework Version: 1.98.0
• Node.js Version: 14.15.3
• OS : MacOS
• Language (Version): TS 4.1.3

Other

---

This is 🐛 Bug Report

[shellscape]

It's worth noting that this is occurring in a monorepo with multiple parallel deployments. The monorepo in question has 14 services and we started seeing this issue with more than one deployment being run at once. This may be a red herring, but that added context may be useful. I suspect that there is some issue with the executionId and parallel deploys, but I could find no evidence for that to be the case.

[rix0rrr]

This error seems to occur when two CDK invocations create the change set at the same time. They'll try to create a change set with the same name, which is causing this error.

Pass --change-set-name to potentially parallel invocations to make sure they create different change sets.

Only one of them will be able to deploy successfully anyway, but at least the error message will be different :)

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[shellscape]

@rix0rrr appreciate the reply here, but seems like a premature close. Passing the buck to the users for a race/context conflict comes across as (forgive me, I can't think of another word, and I don't mean to be overly negative here) lazy. For a project with as much reach as this one, it seems like an important problem to solve without requiring user intervention.

At the very least, this should be codified in documentation explicitly, no?

[cristianrat]

We are having the exact issue with parallel deployments, which use some shared resources.
So we'll have to extract those resources and deploy them at an earlier stage, and not do parallel

[elliot-smith]

For those that run into this issue and are trying to deploy into multiple regions or potentially accounts in separate shells, you can stop this error from occurring by specifying an output directory for the cdk stack.

i.e. Shell 1
export AWS_REGION=us-west-2 && cdk deploy --all --output /my/custom/cdk-out/directory/1

Shell 2
export AWS_REGION=ap-southeast-2 && cdk deploy --all --output /my/custom/cdk-out/directory/2

[shellscape]

@elliot-smith thank you for sharing this. it's wild that CDK doesn't do this by default.