Updating a BucketDeployment's memoryLimit causes stack update failure

[tedinski]

Changing the memoryLimit of an existing BucketDeployment (@aws-cdk/aws-s3-deployment) causes the CloudFormation deploy of the update to fail with the error "Modifying service token is not allowed."

Reproduction Steps

Create a bucket deployment:

const deployment = new BucketDeployment(this, 'BucketDeployment', {
  destinationBucket: myDestinationBucket,
  sources: [Source.bucket(myBucket, myS3Obj)],
});

Then add memoryLimit: 1792, (for example) to its properties and redeploy. The CFN changeset will fail to deploy and rollback.

Error Log

CloudFormation reports the error "Modifying service token is not allowed." for the "BucketDeploymentCustomResource."

Environment

• CLI Version : 1.30
• Framework Version: 1.23
• OS : Mac/Linux
• Language : Typescript

Workaround

For anyone struggling with this issue, here's how I worked around the problem:

1. Remove your bucket deployment construct.
2. Deploy to all environments. The contents of the S3 bucket might be stale momentarily, but they aren't deleted or anything.
3. Add it back with the correct memoryLimit.
4. Deploy to all environments.

In retrospect, just changing the construct name (add 2 to the end or something) to destroy the old and create the new would probably also have worked. I haven't tested that.

Other

I noticed this was noted on the original pull request by someone else as well: #4204 (comment)

Additional request

I strongly believe the default memoryLimit should be set to 1792, which is the documented number for a full vCPU for a Lambda.

Our experience with even a small 2MB zip file showed we had a 40s deploy at 128 MB and a 3s deploy at 1792 MB. That's 13X faster at 14X resources, a basically linear speedup, which means it's nearly free in terms of cost. I strongly believe there's no reason whatsoever for this lambda to ever have a smaller memory limit, which means this is a very bad default.

Of course, to safely update that default, this bug needs fixing first. Otherwise everyone's deploy will fail.

---

This is 🐛 Bug Report

[iliapolo]

Hi @tedinski - thanks for reporting this.

Writing down my initial investigation for reference to when we work on the resolution:

What happens is that we use the lambda function arn as the serviceToken of the custom resource. When the memory configuration changes, it is reflected in the function arn so that a new lambda with the new memory config will be created. This in turn causes the custom resource to have a different service token, which is not allowed.

Proposed solution:

Incorporate lambda config in the custom resource id so that a new custom resource will be created, with a new service token.

[Khalian]

Whats a proposed workaround for this? I dont want to go blow up my stack, only to go update bucket deployments.

[iliapolo]

@Khalian Apologies for the delayed response. You can change the BucketDeployment construct id. This will replace the underlying custom resource and deploy the new one with the updated config. It doesn't require blowing up the stack.

[ryanwilliams83]

I suspect this lambda does not detect when the awscli command terminates due to OOM error.
I have seen this lambda (when configured at the default 128MB) execute for the full 15 minutes even though cloudwatch output abruptly stops at 62 seconds.

The problem mainly occurs when trying to BucketDeploy 161 local files totalling 133MB using the default memoryLimit

[masterchop]

i dont understand why uploading files through the CDK is so painful in compare with the SDK. SDK just works and faster.

[masterchop]

i dont understand why uploading files through the CDK is so painful in compare with the SDK. SDK just works and faster.