Skip to content

Commit ccab485

Browse files
authored
fix(s3-assets): cannot publish a file without extension (#30597)
### Issue # (if applicable) Closes #30471 ### ### Reason for this change Publishing a file with no extension using the `Asset` class with `BundlingOutput.SINGLE_FILE` and `AssetHashType.SOURCE`(default), as shown below, will result in an error `fail: EISDIR: illegal operation on a directory, read`, and publishing will fail. ```ts export class AssetTestStack extends Stack { constructor(scope: Construct, id: string, props?: StackProps) { super(scope, id, props); const asset = new s3assets.Asset(this, 'asset', { path: path.join(__dirname, '../assets/main'), bundling: { image: DockerImage.fromRegistry('golang:1.21'), entrypoint: ["bash", "-c"], command: ["echo 123 > /asset-output/main"], // a file without extension outputType: BundlingOutput.SINGLE_FILE, }, }); new CfnOutput(this, 'AssetHash', { value: asset.assetHash }); } } ``` This is because the path in `*.asset.json` is different from the actual file path. The `*.asset.json` expects the file to be in `asset.bead5b2c0d128650228f146d2326d5f3cbfb36738a9383fc6a09b1e9278803f0`, but when I check the `cdk.out` directory, I see that `asset.bead5b2c0d128650228f146d2326d5f3cbfb36738a9383fc6a09b1e9278803f0` is a directory, not a file. ```json { "version": "36.0.0", "files": { "bead5b2c0d128650228f146d2326d5f3cbfb36738a9383fc6a09b1e9278803f0": { "source": { "path": "asset.bead5b2c0d128650228f146d2326d5f3cbfb36738a9383fc6a09b1e9278803f0", "packaging": "file" }, "destinations": { "current_account-us-east-1": { "bucketName": "cdk-hnb659fds-assets-${AWS::AccountId}-us-east-1", "objectKey": "bead5b2c0d128650228f146d2326d5f3cbfb36738a9383fc6a09b1e9278803f0.bead5b2c0d128650228f146d2326d5f3cbfb36738a9383fc6a09b1e9278803f0", "region": "us-east-1", "assumeRoleArn": "arn:${AWS::Partition}:iam::${AWS::AccountId}:role/cdk-hnb659fds-file-publishing-role-${AWS::AccountId}-us-east-1" } } } } ``` ```bash cdk.out ├── SampleStack.assets.json ├── SampleStack.template.json ├── asset.bead5b2c0d128650228f146d2326d5f3cbfb36738a9383fc6a09b1e9278803f0 │   └── main ├── cdk.out ├── manifest.json └── tree.json ``` If I change it to a file with an extension, as shown below, I see that the file with the extension is staged under `cdk.out` dir, and the asset is published successfully. ```ts export class AssetTestStack extends Stack { constructor(scope: Construct, id: string, props?: StackProps) { super(scope, id, props); const asset = new s3assets.Asset(this, 'asset', { path: path.join(__dirname, '../assets/main'), bundling: { image: DockerImage.fromRegistry('golang:1.21'), entrypoint: ["bash", "-c"], command: ["echo 123 > /asset-output/main.bin"], // a file with an extension outputType: BundlingOutput.SINGLE_FILE, }, }); new CfnOutput(this, 'AssetHash', { value: asset.assetHash }); } } ``` ```bash cdk.out ├── SampleStack.assets.json ├── SampleStack.template.json ├── asset.dc5ce447844d7490834e46df016edc7f671b4fae19ab55b6c78973dcb5af98f8 │   └── main.bin ├── asset.dc5ce447844d7490834e46df016edc7f671b4fae19ab55b6c78973dcb5af98f8.bin # !! staged file here !! ├── cdk.out ├── manifest.json └── tree.json ``` ```json { "version": "36.0.0", "files": { "dc5ce447844d7490834e46df016edc7f671b4fae19ab55b6c78973dcb5af98f8": { "source": { "path": "asset.dc5ce447844d7490834e46df016edc7f671b4fae19ab55b6c78973dcb5af98f8.bin", "packaging": "file" }, "destinations": { "current_account-us-east-1": { "bucketName": "cdk-hnb659fds-assets-${AWS::AccountId}-us-east-1", "objectKey": "dc5ce447844d7490834e46df016edc7f671b4fae19ab55b6c78973dcb5af98f8.bin", "region": "us-east-1", "assumeRoleArn": "arn:${AWS::Partition}:iam::${AWS::AccountId}:role/cdk-hnb659fds-file-publishing-role-${AWS::AccountId}-us-east-1" } } } } ``` Files without extensions must be staged correctly in order to be published correctly. ### Description of changes The directory to write the bundling output is usually `cdk.out/asset.{asset hash}`. If the extension exists, it will be renamed from `cdk.out/asset.{asset hash}/{asset file name}` to `cdk.out/{asset hash}.{asset file extension}`. https://github.com/aws/aws-cdk/blob/c826d8faaeb310623eb9a1a1c82930b679768007/packages/aws-cdk-lib/core/lib/asset-staging.ts#L392 If the extension does not exist, the file name `cdk.out/asset.{asset hash}` (without extension) will be the same as the directory where bundling output is written. Therefore, the file is already considered staged and will not be staged correctly. https://github.com/aws/aws-cdk/blob/c826d8faaeb310623eb9a1a1c82930b679768007/packages/aws-cdk-lib/core/lib/asset-staging.ts#L383 Therefore, in such cases, I fix to change the file name by adding a suffix such as `noext` after the file name so that the file is correctly renamed. ### Description of how you validated changes unit tests and integ test. ### Checklist - [x] My code adheres to the [CONTRIBUTING GUIDE](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md) and [DESIGN GUIDELINES](https://github.com/aws/aws-cdk/blob/main/docs/DESIGN_GUIDELINES.md) ---- *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
1 parent a279b98 commit ccab485

File tree

13 files changed

+224
-14
lines changed

13 files changed

+224
-14
lines changed

Diff for: packages/@aws-cdk-testing/framework-integ/test/aws-s3-assets/test/integ.assets.file-bundling.lit.js.snapshot/IntegTestDefaultTestDeployAssertE3E7D2A4.assets.json

+1-1
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

Diff for: packages/@aws-cdk-testing/framework-integ/test/aws-s3-assets/test/integ.assets.file-bundling.lit.js.snapshot/asset.eb4b352e81f47c1880938665c15111e9aa20db0d42e929187a1b592a593ea713/main

Whitespace-only changes.

Diff for: packages/@aws-cdk-testing/framework-integ/test/aws-s3-assets/test/integ.assets.file-bundling.lit.js.snapshot/asset.eb4b352e81f47c1880938665c15111e9aa20db0d42e929187a1b592a593ea713noext

+1
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

Diff for: packages/@aws-cdk-testing/framework-integ/test/aws-s3-assets/test/integ.assets.file-bundling.lit.js.snapshot/cdk-integ-assets-bundling.assets.json

+16-3
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

Diff for: packages/@aws-cdk-testing/framework-integ/test/aws-s3-assets/test/integ.assets.file-bundling.lit.js.snapshot/cdk.out

+1-1
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

Diff for: packages/@aws-cdk-testing/framework-integ/test/aws-s3-assets/test/integ.assets.file-bundling.lit.js.snapshot/integ.json

+1-1
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

Diff for: packages/@aws-cdk-testing/framework-integ/test/aws-s3-assets/test/integ.assets.file-bundling.lit.js.snapshot/manifest.json

+2-2
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

Diff for: packages/@aws-cdk-testing/framework-integ/test/aws-s3-assets/test/integ.assets.file-bundling.lit.js.snapshot/tree.json

+28-3
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

Diff for: packages/@aws-cdk-testing/framework-integ/test/aws-s3-assets/test/integ.assets.file-bundling.lit.ts

+11
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,17 @@ class TestStack extends Stack {
2626

2727
const user = new iam.User(this, 'MyUser');
2828
asset.grantRead(user);
29+
30+
new assets.Asset(this, 'BundledAssetWithoutExtension', {
31+
path: path.join(__dirname, 'markdown-asset'),
32+
bundling: {
33+
image: DockerImage.fromBuild(path.join(__dirname, 'alpine-markdown')),
34+
outputType: BundlingOutput.SINGLE_FILE,
35+
command: [
36+
'sh', '-c', 'echo 123 > /asset-output/main',
37+
],
38+
},
39+
});
2940
}
3041
}
3142

Diff for: packages/aws-cdk-lib/core/lib/asset-staging.ts

+18-3
Original file line numberDiff line numberDiff line change
@@ -278,9 +278,10 @@ export class AssetStaging extends Construct {
278278
*/
279279
private stageByCopying(): StagedAsset {
280280
const assetHash = this.calculateHash(this.hashType);
281-
const stagedPath = this.stagingDisabled
281+
const targetPath = this.stagingDisabled
282282
? this.sourcePath
283283
: path.resolve(this.assetOutdir, renderAssetFilename(assetHash, getExtension(this.sourcePath)));
284+
const stagedPath = this.renderStagedPath(this.sourcePath, targetPath);
284285

285286
if (!this.sourceStats.isDirectory() && !this.sourceStats.isFile()) {
286287
throw new Error(`Asset ${this.sourcePath} is expected to be either a directory or a regular file`);
@@ -338,7 +339,10 @@ export class AssetStaging extends Construct {
338339
// Calculate assetHash afterwards if we still must
339340
assetHash = assetHash ?? this.calculateHash(this.hashType, bundling, bundledAsset.path);
340341

341-
const stagedPath = path.resolve(this.assetOutdir, renderAssetFilename(assetHash, bundledAsset.extension));
342+
const stagedPath = this.renderStagedPath(
343+
bundledAsset.path,
344+
path.resolve(this.assetOutdir, renderAssetFilename(assetHash, bundledAsset.extension)),
345+
);
342346

343347
this.stageAsset(bundledAsset.path, stagedPath, 'move');
344348

@@ -388,7 +392,7 @@ export class AssetStaging extends Construct {
388392
}
389393

390394
// Moving can be done quickly
391-
if (style == 'move') {
395+
if (style === 'move') {
392396
fs.renameSync(sourcePath, targetPath);
393397
return;
394398
}
@@ -511,6 +515,17 @@ export class AssetStaging extends Construct {
511515
throw new Error('Unknown asset hash type.');
512516
}
513517
}
518+
519+
private renderStagedPath(sourcePath: string, targetPath: string): string {
520+
// Add a suffix to the asset file name
521+
// because when a file without extension is specified, the source directory name is the same as the staged asset file name.
522+
// But when the hashType is `AssetHashType.OUTPUT`, the source directory name begins with `bundling-temp-` and the staged asset file name is different.
523+
// We only need to add a suffix when the hashType is not `AssetHashType.OUTPUT`.
524+
if (this.hashType !== AssetHashType.OUTPUT && path.dirname(sourcePath) === targetPath) {
525+
targetPath = targetPath + '_noext';
526+
}
527+
return targetPath;
528+
}
514529
}
515530

516531
function renderAssetFilename(assetHash: string, extension = '') {

Diff for: packages/aws-cdk-lib/core/test/docker-stub-cp.sh

+11
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,17 @@ set -euo pipefail
66
echo "$@" >> /tmp/docker-stub-cp.input.concat
77
echo "$@" > /tmp/docker-stub-cp.input
88

9+
# create a file without extension to emulate created files, fetch the target path from the "docker cp" command
10+
if cat /tmp/docker-stub-cp.input.concat | grep "DOCKER_STUB_SINGLE_FILE_WITHOUT_EXT"; then
11+
if echo "$@" | grep "cp"| grep "/asset-output"; then
12+
outdir=$(echo "$@" | grep cp | grep "/asset-output" | xargs -n1 | grep "cdk.out" | head -n1 | cut -d":" -f1)
13+
if [ -n "$outdir" ]; then
14+
touch "${outdir}/test" # create a file witout extension
15+
exit 0
16+
fi
17+
fi
18+
fi
19+
920
# create a fake zip to emulate created files, fetch the target path from the "docker cp" command
1021
if echo "$@" | grep "cp"| grep "/asset-output"; then
1122
outdir=$(echo "$@" | grep cp | grep "/asset-output" | xargs -n1 | grep "cdk.out" | head -n1 | cut -d":" -f1)

Diff for: packages/aws-cdk-lib/core/test/docker-stub.sh

+6
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,12 @@ if echo "$@" | grep "DOCKER_STUB_SINGLE_ARCHIVE"; then
3737
exit 0
3838
fi
3939

40+
if echo "$@" | grep "DOCKER_STUB_SINGLE_FILE_WITHOUT_EXT"; then
41+
outdir=$(echo "$@" | xargs -n1 | grep "/asset-output" | head -n1 | cut -d":" -f1)
42+
touch ${outdir}/test # create a file witout extension
43+
exit 0
44+
fi
45+
4046
if echo "$@" | grep "DOCKER_STUB_SINGLE_FILE"; then
4147
outdir=$(echo "$@" | xargs -n1 | grep "/asset-output" | head -n1 | cut -d":" -f1)
4248
touch ${outdir}/test.txt

Diff for: packages/aws-cdk-lib/core/test/staging.test.ts

+128
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@ enum DockerStubCommand {
2020
MULTIPLE_FILES = 'DOCKER_STUB_MULTIPLE_FILES',
2121
SINGLE_ARCHIVE = 'DOCKER_STUB_SINGLE_ARCHIVE',
2222
SINGLE_FILE = 'DOCKER_STUB_SINGLE_FILE',
23+
SINGLE_FILE_WITHOUT_EXT = 'DOCKER_STUB_SINGLE_FILE_WITHOUT_EXT',
2324
VOLUME_SINGLE_ARCHIVE = 'DOCKER_STUB_VOLUME_SINGLE_ARCHIVE',
2425
}
2526

@@ -1450,6 +1451,68 @@ describe('staging', () => {
14501451
expect(staging.isArchive).toEqual(false);
14511452
});
14521453

1454+
test('bundling that produces a single file with SINGLE_FILE_WITHOUT_EXT and hash type SOURCE', () => {
1455+
// GIVEN
1456+
const app = new App({ context: { [cxapi.NEW_STYLE_STACK_SYNTHESIS_CONTEXT]: false } });
1457+
const stack = new Stack(app, 'stack');
1458+
const directory = path.join(__dirname, 'fs', 'fixtures', 'test1');
1459+
1460+
// WHEN
1461+
const staging = new AssetStaging(stack, 'Asset', {
1462+
sourcePath: directory,
1463+
bundling: {
1464+
image: DockerImage.fromRegistry('alpine'),
1465+
command: [DockerStubCommand.SINGLE_FILE_WITHOUT_EXT],
1466+
outputType: BundlingOutput.SINGLE_FILE,
1467+
},
1468+
assetHashType: AssetHashType.SOURCE, // default
1469+
});
1470+
1471+
// THEN
1472+
const assembly = app.synth();
1473+
expect(fs.readdirSync(assembly.directory)).toEqual([
1474+
'asset.ef734136dc22840a94140575a2f98cbc061074e09535589d1cd2c11a4ac2fd75',
1475+
'asset.ef734136dc22840a94140575a2f98cbc061074e09535589d1cd2c11a4ac2fd75_noext',
1476+
'cdk.out',
1477+
'manifest.json',
1478+
'stack.template.json',
1479+
'tree.json',
1480+
]);
1481+
expect(staging.packaging).toEqual(FileAssetPackaging.FILE);
1482+
expect(staging.isArchive).toEqual(false);
1483+
});
1484+
1485+
test('bundling that produces a single file with SINGLE_FILE_WITHOUT_EXT and hash type CUSTOM', () => {
1486+
// GIVEN
1487+
const app = new App({ context: { [cxapi.NEW_STYLE_STACK_SYNTHESIS_CONTEXT]: false } });
1488+
const stack = new Stack(app, 'stack');
1489+
const directory = path.join(__dirname, 'fs', 'fixtures', 'test1');
1490+
1491+
// WHEN
1492+
const staging = new AssetStaging(stack, 'Asset', {
1493+
sourcePath: directory,
1494+
bundling: {
1495+
image: DockerImage.fromRegistry('alpine'),
1496+
command: [DockerStubCommand.SINGLE_FILE_WITHOUT_EXT],
1497+
outputType: BundlingOutput.SINGLE_FILE,
1498+
},
1499+
assetHashType: AssetHashType.CUSTOM,
1500+
assetHash: 'custom',
1501+
});
1502+
1503+
// THEN
1504+
const assembly = app.synth();
1505+
expect(fs.readdirSync(assembly.directory)).toEqual([
1506+
'asset.f81c5ba9e81eebb202881a8e61a83ab4b69f6bee261989eb93625c9cf5d35335',
1507+
'asset.f81c5ba9e81eebb202881a8e61a83ab4b69f6bee261989eb93625c9cf5d35335_noext',
1508+
'cdk.out',
1509+
'manifest.json',
1510+
'stack.template.json',
1511+
'tree.json',
1512+
]);
1513+
expect(staging.packaging).toEqual(FileAssetPackaging.FILE);
1514+
expect(staging.isArchive).toEqual(false);
1515+
});
14531516
});
14541517

14551518
describe('staging with docker cp', () => {
@@ -1517,6 +1580,71 @@ describe('staging with docker cp', () => {
15171580
expect.stringContaining('volume rm assetOutput'),
15181581
]));
15191582
});
1583+
1584+
test('bundling that produces a single file with docker image copy variant and hash type SOURCE', () => {
1585+
// GIVEN
1586+
const app = new App({ context: { [cxapi.NEW_STYLE_STACK_SYNTHESIS_CONTEXT]: false } });
1587+
const stack = new Stack(app, 'stack');
1588+
const directory = path.join(__dirname, 'fs', 'fixtures', 'test1');
1589+
1590+
// WHEN
1591+
const staging = new AssetStaging(stack, 'Asset', {
1592+
sourcePath: directory,
1593+
bundling: {
1594+
image: DockerImage.fromRegistry('alpine'),
1595+
command: [DockerStubCommand.SINGLE_FILE_WITHOUT_EXT],
1596+
outputType: BundlingOutput.SINGLE_FILE,
1597+
bundlingFileAccess: BundlingFileAccess.VOLUME_COPY,
1598+
},
1599+
assetHashType: AssetHashType.SOURCE, // default
1600+
});
1601+
1602+
// THEN
1603+
const assembly = app.synth();
1604+
expect(fs.readdirSync(assembly.directory)).toEqual([
1605+
'asset.93bd4079bff7440a725991ecf249416ae9ad73cb639f4a8d9e8f3ad8d491e89f',
1606+
'asset.93bd4079bff7440a725991ecf249416ae9ad73cb639f4a8d9e8f3ad8d491e89f_noext',
1607+
'cdk.out',
1608+
'manifest.json',
1609+
'stack.template.json',
1610+
'tree.json',
1611+
]);
1612+
expect(staging.packaging).toEqual(FileAssetPackaging.FILE);
1613+
expect(staging.isArchive).toEqual(false);
1614+
});
1615+
1616+
test('bundling that produces a single file with docker image copy variant and hash type CUSTOM', () => {
1617+
// GIVEN
1618+
const app = new App({ context: { [cxapi.NEW_STYLE_STACK_SYNTHESIS_CONTEXT]: false } });
1619+
const stack = new Stack(app, 'stack');
1620+
const directory = path.join(__dirname, 'fs', 'fixtures', 'test1');
1621+
1622+
// WHEN
1623+
const staging = new AssetStaging(stack, 'Asset', {
1624+
sourcePath: directory,
1625+
bundling: {
1626+
image: DockerImage.fromRegistry('alpine'),
1627+
command: [DockerStubCommand.SINGLE_FILE_WITHOUT_EXT],
1628+
outputType: BundlingOutput.SINGLE_FILE,
1629+
bundlingFileAccess: BundlingFileAccess.VOLUME_COPY,
1630+
},
1631+
assetHashType: AssetHashType.CUSTOM,
1632+
assetHash: 'custom',
1633+
});
1634+
1635+
// THEN
1636+
const assembly = app.synth();
1637+
expect(fs.readdirSync(assembly.directory)).toEqual([
1638+
'asset.53a51b4c68874a8e831e24e8982120be2a608f50b2e05edb8501143b3305baa8',
1639+
'asset.53a51b4c68874a8e831e24e8982120be2a608f50b2e05edb8501143b3305baa8_noext',
1640+
'cdk.out',
1641+
'manifest.json',
1642+
'stack.template.json',
1643+
'tree.json',
1644+
]);
1645+
expect(staging.packaging).toEqual(FileAssetPackaging.FILE);
1646+
expect(staging.isArchive).toEqual(false);
1647+
});
15201648
});
15211649

15221650
// Reads a docker stub and cleans the volume paths out of the stub.

0 commit comments

Comments
 (0)