Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: decouple "synth" and "deploy" through cloud assemblies #2636

Merged
merged 31 commits into from
May 29, 2019

Conversation

eladb
Copy link
Contributor

@eladb eladb commented May 24, 2019

Formalize the concept of a cloud assembly to allow decoupling "synth" and "deploy". the main
motivation is to allow "deploy" to run in a controlled environment without needing to execute the app for security purposes.

cdk synth now produces a self-contained assembly in the output directory, which we call a "cloud assembly". this directory includes synthesized templates (similar to current behavior) but also a "manifest.json" file and the asset staging directories.

cdk deploy -a assembly-dir will now skip synthesis and will directly deploy the assembly.

To that end, we modified the behavior of asset staging such that all synth output always goes to the output directory, which is by default cdk.out (can be set with --output). if there's a single template, it is printed to stdout, otherwise the name of output directory is printed.

This PR also includes multiple clean ups and deprecations of stale APIs and modules such as:

  • cdk.AppProps includes various new options.
  • An error is thrown if references between stacks that are not in the same app (mostly in test cases).
  • Added the token creation stack trace for errors emitted by tokens
  • Added ConstructNode.root which returns the tree root.

TESTING: verified that all integration tests passed and added a few tests to verify zip and docker assets as well as cloud assemblies. See: https://github.com/awslabs/cdk-ops/pull/364

Closes #1893
Closes #2093
Closes #1954
Closes #2310
Related #2073
Closes #1245
Closes #341
Closes #956
Closes #233

BREAKING CHANGE: IMPORTANT: apps created with the CDK version 0.33.0 and above cannot be used with an older CLI version.

  • --interactive has been removed
  • --numbered has been removed
  • --staging is now a boolean flag that indicates whether assets should be copied to the --output directory or directly referenced (--no-staging is useful for e.g. local debugging with SAM CLI)
  • Assets (e.g. Lambda code assets) are now referenced relative to the output directory.
  • SynthUtils.templateForStackName has been removed (use SynthUtils.synthesize(stack).template).
  • cxapi.SynthesizedStack renamed to cxapi.CloudFormationStackArtifact with multiple API changes.
  • cdk.App.run() now returns a cxapi.CloudAssembly instead of cdk.ISynthesisSession.
  • cdk.App.synthesizeStack and synthesizeStacks has been removed. The cxapi.CloudAssembly object returned from app.run() can be used as an alternative to explore a synthesized assembly programmatically (resolves App.synthesize methods are marked as deprecated with no replacement #2016).
  • cdk.CfnElement.creationTimeStamp may now return undefined if there is no stack trace captured.
  • cdk.ISynthesizable.synthesize now accepts a cxapi.CloudAssemblyBuilder instead of ISynthesisSession.
  • cdk: The concepts of a synthesis session and session stores have been removed (cdk.FileSystemStore, cdk.InMemoryStore, cdk.SynthesisSession, cdk.ISynthesisSessionandcdk.SynthesisOptions`).
  • No more support for legacy manifests (cdk.ManifestOptions member legacyManifest has been removed)
  • All support for build/bundle manifest have been removed (cxapi.BuildManifest, cxapi.BuildStep, etc).
  • Multiple deprecations and breaking changes in the cxapi module (cxapi.AppRuntime has been renamed to cxapi.RuntimeInfo, cxapi.SynthesizeResponse, cxapi.SynthesizedStack has been removed)
  • The @aws-cdk/applet-js program is no longer available. Use decdk as an alternative.

  • verify that old cli fails with a compatibility error and vice versa
  • run all cli integration tests

Pull Request Checklist

  • Testing
    • Unit test added (prefer not to modify an existing test, otherwise, it's probably a breaking change)
    • CLI change?: coordinate update of integration tests with team
    • cdk-init template change?: coordinated update of integration tests with team
  • Docs
    • jsdocs: All public APIs documented
    • README: README and/or documentation topic updated
    • Design: For significant features, design document added to design folder
  • Title and Description
    • Change type: title prefixed with fix, feat and module name in parens, which will appear in changelog
    • Title: use lower-case and doesn't end with a period
    • Breaking?: last paragraph: "BREAKING CHANGE: <describe what changed + link for details>"
    • Issues: Indicate issues fixed via: "Fixes #xxx" or "Closes #xxx"
  • Sensitive Modules (requires 2 PR approvers)
    • IAM Policy Document (in @aws-cdk/aws-iam)
    • EC2 Security Groups and ACLs (in @aws-cdk/aws-ec2)
    • Grant APIs (only if not based on official documentation with a reference)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license.

Elad Ben-Israel added 2 commits May 23, 2019 22:27
formalize the concept of a cloud assembly to allow decoupling "synth" and "deploy". the main
motivation is to allow "deploy" to run in a controlled environment without needing to execute the app for security purposes.

"cdk synth" now produces a self-contained assembly in the output directory, which we call a "cloud assembly". this directory includes synthesized templates (similar to current behavior) but also a "manifest.json" file and the asset staging directories.

"cdk deploy -a <assembly-dir>" will now skip
synthesis and will directly deploy the assembly.

to that end, we modified the behavior of asset staging such that all synth output always goes to the output directory, which is by default `cdk.out` (can be set with `--output`). if there's a single template, it is printed to stdout, otherwise the name of output directory is printed.

BREAKING CHANGE: apps created with this version cannot be deployed using a previous version of the CLI and vice versa. Both have to be >= 0.32.0.

- @aws-cdk/cdk.App member synthesizeStack has been removed
- @aws-cdk/cdk.App member synthesizeStacks has been removed
- @aws-cdk/cdk.FileSystemStore has been removed
- @aws-cdk/cdk.InMemoryStore has been removed
- @aws-cdk/cdk.SynthesisSession member addBuildStep has been removed
- @aws-cdk/cdk.SynthesisSession member store has been removed
- @aws-cdk/cdk.FileSystemStoreOptions has been removed
- @aws-cdk/cdk.ISessionStore has been removed
- @aws-cdk/cdk.ISynthesisSession member addBuildStep has been removed
- @aws-cdk/cdk.ISynthesisSession member store has been removed
- @aws-cdk/cdk.ManifestOptions member legacyManifest has been removed
- @aws-cdk/cdk.ResolveContext has been removed
- @aws-cdk/cdk.SynthesisOptions member legacyManifest has been removed
- @aws-cdk/cdk.SynthesisOptions member store has been removed
- @aws-cdk/cx-api.BuildManifest has been removed
- @aws-cdk/cx-api.BuildStep has been removed
- @aws-cdk/cx-api.SynthesizeResponse has been removed
- @aws-cdk/cx-api.SynthesizedStack has been removed
- @aws-cdk/cx-api.BuildStepType has been removed
- --interactive has been removed
- --numbered has been removed
@eladb eladb requested a review from a team as a code owner May 24, 2019 21:04
@rix0rrr
Copy link
Contributor

rix0rrr commented May 27, 2019

if there's a single template, it is printed to stdout, otherwise the name of output directory is printed.

I don't like this behavior. It's good for humans but horrible to script around. Let's make our tools easy to automate, please?

the main
motivation is to allow "deploy" to run in a controlled environment without needing to execute the app for security purposes.

I agree with the ultimate refactorings, which we've talked about previously, but I hope this is not a stepping stone towards cdk deploy in a pipeline. I'd still rather we build a pipeline that deploys through CloudFormation Actions, if possible.

@RomainMuller
Copy link
Contributor

if there's a single template, it is printed to stdout, otherwise the name of output directory is printed.

I agree with @rix0rrr on this one. I prefer the tool behaves in exactly one way. If you want to keep the behavior of outputting to stdout for a single template, introduce a CLI flag for it.

packages/@aws-cdk/cdk/lib/synthesis.ts Outdated Show resolved Hide resolved
packages/@aws-cdk/cx-api/lib/cxapi.ts Outdated Show resolved Hide resolved
packages/aws-cdk/lib/api/cxapp/exec.ts Outdated Show resolved Hide resolved
packages/aws-cdk/lib/api/cxapp/stacks.ts Show resolved Hide resolved
Copy link
Contributor

@RomainMuller RomainMuller left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's pretty good overall! I don't have much to say here, besides a couple of things that look a little odd to me (possibly waved away with an explanation?) or having names that I don't like much.

packages/@aws-cdk/applet-js/package.json Show resolved Hide resolved
packages/@aws-cdk/assert/lib/synth-utils.ts Outdated Show resolved Hide resolved
packages/@aws-cdk/assert/lib/synth-utils.ts Outdated Show resolved Hide resolved
packages/@aws-cdk/assert/test/test.assertions.ts Outdated Show resolved Hide resolved
packages/@aws-cdk/cdk/lib/app.ts Show resolved Hide resolved

private validateDeps() {
for (const artifact of this.artifacts) {
ignore(artifact.depends);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this read artifact.dependencies?

packages/@aws-cdk/cx-api/lib/assembly/cloud-assembly.ts Outdated Show resolved Hide resolved
this.assets = this.buildAssets();
}

private buildAssets() {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This sort of suggested to me that the actual build process was happening here - but this really just makes an array... Could call collectAssets instead.

packages/aws-cdk/bin/cdk.ts Show resolved Hide resolved
@@ -88,7 +88,7 @@ function invalidateContext(context: Context, key: string) {
// Unset!
if (context.has(key)) {
context.unset(key);
print(`Context value ${colors.blue(key)} reset. It will be refreshed on the next SDK synthesis run.`);
print(`Context value ${colors.blue(key)} reset. It will be refreshed on next synthesis`);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe this should hint that it means network + credentials are needed on the next execution.

@mergify
Copy link
Contributor

mergify bot commented Sep 23, 2019

Thanks so much for taking the time to contribute to the AWS CDK ❤️

We will shortly assign someone to review this pull request and help get it
merged. In the meantime, please take a minute to make sure you follow this
checklist
:

  • PR title type(scope): text
    • type: fix, feat, refactor go into CHANGELOG, chore is hidden
    • scope: name of module without aws- or cdk- prefix or postfix (e.g. s3 instead of aws-s3-deployment)
    • text: use all lower-case, do not end with a period, do not include issue refs
  • PR Description
    • Rationale: describe rationale of change and approach taken
    • Issues: indicate issues fixed via: fixes #xxx or closes #xxx
    • Breaking?: last paragraph: BREAKING CHANGE: <describe what changed + link for details>
  • Testing
    • Unit test added. Prefer to add a new test rather than modify existing tests
    • CLI or init templates change? Re-run/add CLI integration tests
  • Documentation
    • README: update module README to describe new features
    • API docs: public APIs must be documented. Copy from official AWS docs when possible
    • Design: for significant features, follow design process

This was referenced Dec 12, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
contribution/core This is a PR that came from AWS.
Projects
None yet
4 participants