(pipelines.CodePipeline): unable to use additional_inputs

[KnowSQL]

What is the problem?

No matter what name I supply for the additional inputs directory, it fails at:

[Container] 2021/10/28 23:30:47 Running command [[ ! -d "blah-1234" ]] || { echo 'additionalInputs: "blah-1234" must not exist yet. If you want to merge multiple artifacts, use a "cp" command.'; exit 1; } && ln -s -- "$CODEBUILD_SRC_DIR_blahblah_blahblah_Source" "blah-1234"

followed by:

/codebuild/output/tmp/script.sh: 4: [[: not found

followed by:

additionalInputs: "blah-1234" must not exist yet. If you want to merge multiple artifacts, use a "cp" command.

followed by:

[Container] 2021/10/28 23:30:47 Command did not exit successfully [[ ! -d "blah-1234" ]] || { echo 'additionalInputs: "blah-1234" must not exist yet. If you want to merge multiple artifacts, use a "cp" command.'; exit 1; } && ln -s -- "$CODEBUILD_SRC_DIR_vblahblah_blahblah_Source" "blah-1234" exit status 1

causing the build to fail.

Reproduction Steps

My code simply looks like:

        source0 = CodePipelineSource.connection("blahblah/blahblah", "master",
            connection_arn="arn:aws:codestar-connections:us-east-1:<123456789012>:connection/<some-uuid>"
        )

        pipeline = CodePipeline(self, "Abcdef",
            synth=ShellStep("Synth",
                input=CodePipelineSource.connection("blablah/blahblah0", "master",
                    connection_arn="arn:aws:codestar-connections:us-east-1:<123456789012>:connection/<some-uuid>"
                ),
                additional_inputs={
                    "blah-1234": source0,
                },
                commands=[
                    "npm install -g aws-cdk",
                    "pip install -r requirements.txt",
                    "cdk synth"
                ]
            )
        )

I've put blah etc in place of real github sources are they are all private repositories. The code is pulled in fine, but the build/synth step fails for source0 above.

I've tried various combinations like ../blah-1234 (to create it in a sibling directory) etc. but all fail with similar error messages.

I'm suspicious of:

/codebuild/output/tmp/script.sh: 4: [[: not found

and think it's something in the aws script that's failing somewhere?

What did you expect to happen?

I expected the build to pass

What actually happened?

The build failed

CDK CLI Version

1.129.0

Framework Version

No response

Node.js Version

N/A

OS

Arch Linux

Language

Python

Language Version

No response

Other information

No response

[KnowSQL]

I just came across this commit. My repository name used as the source for additional_inputs does contain a hyphen. Not sure if it's anything to do with that.

Ignore that, see below

[ustulation]

/codebuild/output/tmp/script.sh: 4: [[: not found

Hmm this looks like script.sh is not being run as a bash script but just a shell script. [[ ]] is bash construct i think and pure posix shells will not recognise it. Maybe the shebang is #!/bin/sh or whatever it is it's not pointing to bash? Maybe bash is aliased to some pure posix shell?

[ustulation]

The BuildSpec produced by cdk looks something like this for the above:

"BuildSpec": "{\n  \"version\": \"0.2\",\n  \"phases\": {\n    \"install\": {\n      \"commands\": [\n        \"[[ ! -d \\\"../some-repo-2\\\" ]] || { echo 'additionalInputs: \\\"../some-repo-2\\\" must not exist yet. If you want to merge multiple artifacts, use a \\\"cp\\\" command.'; exit 1; } && ln -s -- \\\"$CODEBUILD_SRC_DIR_some_owner_some_repo_2_Source\\\" \\\"../some-repo-2\\\"\"\n      ]\n    },\n    \"build\": {\n      \"commands\": [\n        \"npm install -g aws-cdk\",\n        \"pip install -r requirements.txt\",\n        \"cdk synth\"\n      ]\n    }\n  },\n  \"artifacts\": {\n    \"base-directory\": \"cdk.out\",\n    \"files\": \"**/*\"\n  }\n}",
"Type": "CODEPIPELINE"

Is there a way to modify this from within cdk? Like try using POSIX compliant [ ! -d '../some-repo-2' ] || ... instead of [[ construct ?
Or is there a way to ask ShellStep in synth to use bash shell in-case it's using something else?

[ustulation]

OK I've made this deployment successful. It is the issue with the shell. We cannot use ShellStep if you have additional_inputs. I needed to use it's subclass CodeBuildStep instead and specify via partial_build_spec to use bash as the shell as opposed to whatever was getting used.

pipeline = CodePipeline(self, "...",
            synth=CodeBuildStep("Synth",
                partial_build_spec=BuildSpec.from_object({
                    "version": "0.2",
                    "env": {
                        "shell": "bash"
                    }
                }),
                input=...,
                additional_inputs={ ... },
                commands= [ ... ],
                ...

That solves it.

That also means that this should be treated as a bug using ShellStep seems to make it impossible to have additional_inputs. So either make ShellStep configure the underlying CodeBuild step to use bash by default or change the factory method in the source code to make it POSIX compliant (ie., drop [[ .. and use [ .. instead).

[NGL321]

Hey @ustulation,

It looks from your comments like you have figured out the problem independently. Can we consider this resolved, or does it still need attention from a team member?

[ustulation]

I'll leave that to yous but I would consider this to be a bug. The fact that ShellStep does not set the codebuild env to use bash but then creates a bash construct thus failing shell execution at runtime means it's unusable for additional_inputs

[abend-arg]

I experimented this error and ended downgrading to version 1.128. I needed to move forward with another change and could not verify why this was happening.