(aws-lamdba-nodejs): esbuild + yarn berry monorepo + bundling.nodeModules fails in CI

[blimmer]

What is the problem?

When using a yarn monorepo layout (as described in the Reference Architecture), an issue occurs on CI platforms when esbuild tries to bundle the lambda function.

Upon bundling an error like this appears:

➤ YN0000: [pkg1]: ➤ YN0028: │ The lockfile would have been modified by this install, which is explicitly forbidden.
➤ YN0000: [pkg1]: ➤ YN0000: └ Completed in 2s 749ms
➤ YN0000: [pkg1]: ➤ YN0000: Failed with errors in 2s 770ms
➤ YN0000: [pkg1]:
➤ YN0000: [pkg1]: /Users/blimmer/code/cdk-bug-reports/node_modules/@aws-cdk/core/lib/asset-staging.ts:398
➤ YN0000: [pkg1]:       throw new Error(`Failed to bundle asset ${this.node.path}, bundle output is located at ${bundleErrorDir}: ${err}`);
➤ YN0000: [pkg1]:             ^
➤ YN0000: [pkg1]: Error: Failed to bundle asset Pkg1Stack/Lambda/Code/Stage, bundle output is located at /Users/blimmer/code/cdk-bug-reports/packages/pkg1/cdk.out/bundling-temp-5161160113442bc15e6e0b6db63161298f2b3d8fd5504f589c1ebb6f3e99008a-error: Error: bash exited with status 1
➤ YN0000: [pkg1]:     at AssetStaging.bundle (/Users/blimmer/code/cdk-bug-reports/node_modules/@aws-cdk/core/lib/asset-staging.ts:398:13)
➤ YN0000: [pkg1]:     at AssetStaging.stageByBundling (/Users/blimmer/code/cdk-bug-reports/node_modules/@aws-cdk/core/lib/asset-staging.ts:246:10)
➤ YN0000: [pkg1]:     at stageThisAsset (/Users/blimmer/code/cdk-bug-reports/node_modules/@aws-cdk/core/lib/asset-staging.ts:137:35)
➤ YN0000: [pkg1]:     at Cache.obtain (/Users/blimmer/code/cdk-bug-reports/node_modules/@aws-cdk/core/lib/private/cache.ts:24:13)
➤ YN0000: [pkg1]:     at new AssetStaging (/Users/blimmer/code/cdk-bug-reports/node_modules/@aws-cdk/core/lib/asset-staging.ts:162:44)
➤ YN0000: [pkg1]:     at new Asset (/Users/blimmer/code/cdk-bug-reports/node_modules/@aws-cdk/aws-s3-assets/lib/asset.ts:68:21)
➤ YN0000: [pkg1]:     at AssetCode.bind (/Users/blimmer/code/cdk-bug-reports/node_modules/@aws-cdk/aws-lambda/lib/code.ts:183:20)
➤ YN0000: [pkg1]:     at new Function (/Users/blimmer/code/cdk-bug-reports/node_modules/@aws-cdk/aws-lambda/lib/function.ts:338:29)
➤ YN0000: [pkg1]:     at new NodejsFunction (/Users/blimmer/code/cdk-bug-reports/node_modules/@aws-cdk/aws-lambda-nodejs/lib/function.ts:53:5)
➤ YN0000: [pkg1]:     at new Pkg1Stack (/Users/blimmer/code/cdk-bug-reports/packages/pkg1/lib/pkg1-stack.ts:9:5)
➤ YN0000: [pkg1]: Subprocess exited with error 1

The problem is because yarn automatically goes into "immutable" mode on CI platforms (see https://github.com/yarnpkg/berry/blob/dd043a8ab1618c0b02e28b2b40743bd336d63437/packages/plugin-essentials/sources/index.ts#L88-L92).

Then, when bundling, the monorepo's yarn.lock file gets copied into the bundling temporary directory:

 pwd
/Users/blimmer/code/cdk-bug-reports/packages/pkg1/cdk.out/bundling-temp-5161160113442bc15e6e0b6db63161298f2b3d8fd5504f589c1ebb6f3e99008a-error
 ll
total 244
-rw-r--r-- 1 blimmer staff   1536 Oct 20 14:14 index.js
-rw-r--r-- 1 blimmer staff     37 Oct 20 14:14 package.json
-rw-r--r-- 1 blimmer staff 239334 Oct 20 14:14 yarn.lock

The package.json file contains the nodeModules I've specified in the bundlingProps. For instance, this construct instantiation:

new lambda.NodejsFunction(this, 'Lambda', {
  bundling: {
    nodeModules: ['axios']
  }
})

produces this package.json file in the bundling directory

 pwd
/Users/blimmer/code/cdk-bug-reports/packages/pkg1/cdk.out/bundling-temp-5161160113442bc15e6e0b6db63161298f2b3d8fd5504f589c1ebb6f3e99008a-error
 cat package.json
{"dependencies":{"axios":"^0.23.0"}}

Herein lies the problem. The yarn.lock file is out of sync for this limited package.json file because it contains all the dependencies for the entire monorepo. And, because yarn automatically goes into immutable mode on CI platforms, the bundling (when it runs yarn install) fails.

Reproduction Steps

1. Clone my sample repo (on the bug/yarn-lockfile-in-ci branch) git clone https://github.com/blimmer/cdk-bug-reports.git --branch=bug/yarn-lockfile-in-ci
2. Install dependencies (yarn install).
3. Set the yarn installs to be immutable using the environment variable (export YARN_ENABLE_IMMUTABLE_INSTALLS=true). This happens automatically in CI environments, but you can simulate it locally.
4. cd packages/pkg1. this is the workspace with a NodejsFunction that specifies a bundling.nodeModules package.
5. Run yarn cdk synth.

You'll observe the lockfile error. esbuild will print the bundling temporary directory where you can see the state described in the section above.

Alternatively, you can check out the failure in GitHub actions here: https://github.com/blimmer/cdk-bug-reports/runs/3956431715. github actions log output - just in case GitHub logs time out for https://github.com/blimmer/cdk-bug-reports/runs/3956431715

What did you expect to happen?

Because I followed the Reference Architecture for monorepos, I expected bundling in my CI environment to work without throwing an error.

What actually happened?

I received an error during bundling that the lockfile would be changed, so my NodejsFunction failed to synth.

github actions log output - just in case GitHub logs time out for https://github.com/blimmer/cdk-bug-reports/runs/3956431715

CDK CLI Version

1.128.0 (build 1d3883a)

Framework Version

No response

Node.js Version

14.x

OS

macos, github actions CI (linux)

Language

Typescript

Language Version

TypeScript 3.9.10 | Yarn Berry v3 | Esbuild 0.13.x

Other information

Proposed Solution

I think the easiest way to fix this would be to flag yarn to not use immutability when bundling local bundling.nodeModules-specified dependencies:

yarn install --no-immutable

That would avoid this issue automatically for all users. There's more details on this flag here.

Workaround

In case you find this bug report and you need to work around this issue, you can set this environment variable in your CI platform before running cdk synth or cdk deploy:

export YARN_ENABLE_IMMUTABLE_INSTALLS=false

I recommend only setting this before the cdk synth or cdk deploy command so when you're installing your project dependencies, immutability is still respected.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.