feat(cli): regression tests #7060

iliapolo · 2020-03-29T11:18:12Z

This PR adds regression testing capabilities to our CLI integration tests.

yarn integ-cli-backwards fetches the latest published version from github and runs its integration tests against the newest code of the CLI and the framework.
yarn integ-cli-framework fetches the latest published version from github and runs its integration tests against the newest CLI code and latest published framework.

More details here: https://github.com/aws/aws-cdk-rfcs/blob/epolon/compatibility-strategy/text/00110-cli-framework-compatibility-strategy.md#breaking-changes-in-cli

Implementation Notes

We don't currently have support for installing framework packages from NPM when running CLI integ tests.

Locally, we use run-against-repo, which uses a local npm wrapper that always symlinks to the repo code.

In CI, we use run-against-dist, which starts up a verdaccio instance that hosts our locally packed packages.

To support installing the latest published versions, I introduced the USE_PUBLISHED_FRAMEWORK_VERSION env variable. Locally, this causes the npm wrapper to simply pass through the command to npm.

In CI, its a bit more complicated. We can't simply pass through to npm because the CLI and all its dependencies still needs to be installed from the local dist packages. I added some hacky code that basically publishes only the necessary packages (CLI and its deps) to verdaccio, leaving everything else to be served by the npmjs uplink.

Commit Message

added two new cli integ tests:

Previous tests against new CLI and new Framework
Previous tests against new CLI and old Framework

End Commit Message

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

mergify · 2020-03-29T11:18:46Z

Title does not follow the guidelines of Conventional Commits. Please adjust title before merge.

mergify · 2020-03-29T11:18:46Z

Title does not follow the guidelines of Conventional Commits. Please adjust title before merge.

aws-cdk-automation · 2020-03-29T12:05:03Z

AWS CodeBuild CI Report

CodeBuild project: AutoBuildProject6AEA49D1-qxepHUsryhcu
Commit ID: bd30a4d
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

aws-cdk-automation · 2020-03-29T12:11:08Z

AWS CodeBuild CI Report

CodeBuild project: AutoBuildProject6AEA49D1-qxepHUsryhcu
Commit ID: 29d05c6
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

aws-cdk-automation · 2020-03-29T12:17:17Z

AWS CodeBuild CI Report

CodeBuild project: AutoBuildProject6AEA49D1-qxepHUsryhcu
Commit ID: ae423e3
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

aws-cdk-automation · 2020-03-29T12:20:44Z

AWS CodeBuild CI Report

CodeBuild project: AutoBuildProject6AEA49D1-qxepHUsryhcu
Commit ID: 945121c
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

aws-cdk-automation · 2020-03-29T14:22:11Z

AWS CodeBuild CI Report

CodeBuild project: AutoBuildProject6AEA49D1-qxepHUsryhcu
Commit ID: 23dfb21
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

aws-cdk-automation · 2020-03-29T14:24:40Z

AWS CodeBuild CI Report

CodeBuild project: AutoBuildProject6AEA49D1-qxepHUsryhcu
Commit ID: 190793f
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

aws-cdk-automation · 2020-03-29T16:14:48Z

AWS CodeBuild CI Report

CodeBuild project: AutoBuildProject6AEA49D1-qxepHUsryhcu
Commit ID: 7438634
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

packages/aws-cdk/test/integ/cli/common.bash

packages/aws-cdk/test/integ/cli/test-cdk-deploy-nested-stack-with-parameters.sh

tools/prlint/package.json

eladb

The names of the two new tests needs to be changed. I couldn't wrap my head around what each one of these tests is actually doing, even after reading all the code it. I don't mind if these names are 200 characters long but it should be clear what their purpose is.

PR title/description:

Please add some details in the commit message about why each type of new test is needed.
Why is the title feat? I don't think we need to advertise this in the CHANGELOG. Do you?

packages/aws-cdk/.gitignore

packages/aws-cdk/CONTRIBUTING.md

packages/aws-cdk/test/integ/cli.exclusions.js

packages/aws-cdk/test/integ/cli/test-cdk-deploy-nested-stack-with-parameters.sh

packages/aws-cdk/test/integ/test-cli-backwards.sh

tools/prlint/package.json

packages/aws-cdk/test/integ/test-cli-framework.sh

iliapolo · 2020-04-02T15:04:03Z

@eladb

Please add some details in the commit message about why each type of new test is needed

Thats why I added a link to the RFC, is that not sufficient?

https://github.com/aws/aws-cdk-rfcs/blob/epolon/compatibility-strategy/text/00110-cli-framework-compatibility-strategy.md#breaking-changes-in-cli

Why is the title feat? I don't think we need to advertise this in the CHANGELOG. Do you?

Debatable, might be useful for contributors to know about this. But I can go either way here.

… names

aws-cdk-automation · 2020-04-02T17:17:39Z

AWS CodeBuild CI Report

CodeBuild project: AutoBuildProject6AEA49D1-qxepHUsryhcu
Commit ID: 0011205
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

aws-cdk-automation · 2020-04-02T18:32:19Z

AWS CodeBuild CI Report

CodeBuild project: AutoBuildProject6AEA49D1-qxepHUsryhcu
Commit ID: 2f7394c
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

aws-cdk-automation · 2020-04-02T18:41:11Z

AWS CodeBuild CI Report

CodeBuild project: AutoBuildProject6AEA49D1-qxepHUsryhcu
Commit ID: a6d07fe
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

aws-cdk-automation · 2020-04-02T18:45:25Z

AWS CodeBuild CI Report

CodeBuild project: AutoBuildProject6AEA49D1-qxepHUsryhcu
Commit ID: bda1dba
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

aws-cdk-automation · 2020-04-02T18:46:46Z

AWS CodeBuild CI Report

CodeBuild project: AutoBuildProject6AEA49D1-qxepHUsryhcu
Commit ID: 226b95f
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

aws-cdk-automation · 2020-04-02T18:47:53Z

AWS CodeBuild CI Report

CodeBuild project: AutoBuildProject6AEA49D1-qxepHUsryhcu
Commit ID: 714bd0e
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

aws-cdk-automation · 2020-04-02T20:23:19Z

AWS CodeBuild CI Report

CodeBuild project: AutoBuildProject6AEA49D1-qxepHUsryhcu
Commit ID: 786f61a
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

aws-cdk-automation · 2020-04-02T20:40:41Z

AWS CodeBuild CI Report

CodeBuild project: AutoBuildProject6AEA49D1-qxepHUsryhcu
Commit ID: 896c866
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

packages/aws-cdk/test/integ/run-wrappers/repo/npm

aws-cdk-automation · 2020-04-02T22:25:49Z

AWS CodeBuild CI Report

CodeBuild project: AutoBuildProject6AEA49D1-qxepHUsryhcu
Commit ID: b0facdc
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mergify · 2020-04-02T22:26:01Z

Thank you for contributing! Your pull request will be updated from master and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

added regression tests ability

ae423e3

iliapolo added the pr/do-not-merge This PR should not be merged at this time. label Mar 29, 2020

mergify bot added the contribution/core This is a PR that came from AWS. label Mar 29, 2020

iliapolo added 3 commits March 29, 2020 14:19

revert changes to test-cdk-assembly.sh

bd30a4d

remove version patching - not needed anymore

945121c

remove blank line

29d05c6

iliapolo added 2 commits March 29, 2020 16:35

fix npm local wrapper to support using latest published framework

190793f

cleanup

23dfb21

only publish CDK cli when using published framework

7438634

iliapolo added 3 commits March 29, 2020 22:18

sanity check

9a93910

dummy

7b093b9

dummy

6cac037

iliapolo commented Mar 29, 2020

View reviewed changes

packages/aws-cdk/test/integ/cli/common.bash Show resolved Hide resolved

added some logs

ed7cc03

SomayaB assigned shivlaks Mar 30, 2020

iliapolo added 6 commits March 31, 2020 00:00

fix tarball path

0aa39c8

oh my

434c2b5

fix version

f112b71

fix tarball construction

a22c215

perpand parameters value with stack prefix

91536df

only publish CDK cli when using published framework

9d795e7

iliapolo commented Apr 1, 2020

View reviewed changes

packages/aws-cdk/test/integ/cli/test-cdk-deploy-nested-stack-with-parameters.sh Show resolved Hide resolved

RomainMuller reviewed Apr 2, 2020

View reviewed changes

tools/prlint/package.json Show resolved Hide resolved

iliapolo requested a review from a team April 2, 2020 12:17

eladb suggested changes Apr 2, 2020

View reviewed changes

iliapolo added 2 commits April 2, 2020 18:46

added comments about why STACK_NAME_PREFIX is needed in the SNS topic…

60f816b

… names

rename some scripts and more detailed explanation in contrib guide

0011205

iliapolo added 5 commits April 2, 2020 21:01

fix path to scripts

2f7394c

some rephrasing

a6d07fe

typos

bda1dba

typo

226b95f

quote env variable expansion in if statement

714bd0e

iliapolo added 2 commits April 2, 2020 22:53

added bunch of comments around the code

786f61a

more comments

896c866

eladb approved these changes Apr 2, 2020

View reviewed changes

packages/aws-cdk/test/integ/run-wrappers/repo/npm Outdated Show resolved Hide resolved

iliapolo added pr-linter/exempt-readme The PR linter will not require README changes and removed pr/do-not-merge This PR should not be merged at this time. labels Apr 2, 2020

Merge branch 'master' into epolon/cli-regression-tests

b0facdc

mergify bot merged commit f18f3f1 into master Apr 2, 2020

mergify bot deleted the epolon/cli-regression-tests branch April 2, 2020 22:26

feat(cli): regression tests #7060

feat(cli): regression tests #7060

Uh oh!

Conversation

iliapolo commented Mar 29, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Implementation Notes

Commit Message

End Commit Message

Uh oh!

mergify bot commented Mar 29, 2020

Uh oh!

mergify bot commented Mar 29, 2020

Uh oh!

aws-cdk-automation commented Mar 29, 2020

AWS CodeBuild CI Report

Uh oh!

aws-cdk-automation commented Mar 29, 2020

AWS CodeBuild CI Report

Uh oh!

aws-cdk-automation commented Mar 29, 2020

AWS CodeBuild CI Report

Uh oh!

aws-cdk-automation commented Mar 29, 2020

AWS CodeBuild CI Report

Uh oh!

aws-cdk-automation commented Mar 29, 2020

AWS CodeBuild CI Report

Uh oh!

aws-cdk-automation commented Mar 29, 2020

AWS CodeBuild CI Report

Uh oh!

aws-cdk-automation commented Mar 29, 2020

AWS CodeBuild CI Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

eladb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

iliapolo commented Apr 2, 2020

Uh oh!

aws-cdk-automation commented Apr 2, 2020

AWS CodeBuild CI Report

Uh oh!

aws-cdk-automation commented Apr 2, 2020

AWS CodeBuild CI Report

Uh oh!

aws-cdk-automation commented Apr 2, 2020

AWS CodeBuild CI Report

Uh oh!

aws-cdk-automation commented Apr 2, 2020

AWS CodeBuild CI Report

Uh oh!

aws-cdk-automation commented Apr 2, 2020

AWS CodeBuild CI Report

Uh oh!

aws-cdk-automation commented Apr 2, 2020

AWS CodeBuild CI Report

Uh oh!

aws-cdk-automation commented Apr 2, 2020

AWS CodeBuild CI Report

Uh oh!

aws-cdk-automation commented Apr 2, 2020

AWS CodeBuild CI Report

Uh oh!

Uh oh!

aws-cdk-automation commented Apr 2, 2020

AWS CodeBuild CI Report

iliapolo commented Mar 29, 2020 •

edited

Loading