rfc: cdk toolchain #2093
Closed
rfc: cdk toolchain #2093
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
a design document describing how a modular cdk toolchain will look like, separating the various phases of deploying a cdk app into the AWS cloud.
|
I'll monitor this PR and once it's in the wild I'll update the diagram at https://docs.aws.amazon.com/CDK/latest/userguide/writing_constructs.html#writing_constructs_cdk_lifecycle (which will likely change location). |
| ## Requirements | ||
|
|
||
| * It should be possible to execute each component in the toolchain individually by feeding it the output from the previous step. | ||
| * Given a specific input, the output from each step must be completely reproducible (no side effects). |
There was a problem hiding this comment.
For the initial step (what currently is synth), this assumes the CDK App is fully deterministic. This is not always true (and I don't think it's an issue, really).
|
|
||
| The protocol between the CDK app `cdk-synth`: | ||
|
|
||
| - Context is passed into the app via a JSON map serialized into an environment variable called `CONTEXT_ENV`. |
There was a problem hiding this comment.
Can't help feel those are clumsy. Also, all environment variables used by the CDK should have names starting with CDK_ in my opinion.
|
|
||
| If the file `missing.json` exists in the output, it means that the synthesis output cannot be used and relies on missing context information, which must be populated in the context map. `cdk-synth` is responsible to resolve this context information through the environmental context provider system (for example, query the account for which availability zones are available in a specific region, read SSM parameters, etc). Then, `cdk-synth` must re-invoke the app with this context. Bear in mind that normally context information is cached in `cdk.context.json`, so subsequent executions will not require resolution. | ||
|
|
||
| `cdk-synth` may repeat this process up to 5 times (theoretically there could be missing context that the app will only know that it needs after it used some information from a resolved context from previous iteration). |
There was a problem hiding this comment.
So this is possibly more than 5 times then. Can be any number of times.
There was a problem hiding this comment.
Yeah 5 seems to come out of nowhere. Are you trying to say something like "the toolkit may impose a limit on the amount of reinvocations to guard against programming errors." ?
|
|
||
| ### cdk-bundle | ||
|
|
||
| The next component in the toolchain is called `cdk-pack`. The purpose of this step is to take the synthesis output and produce a ready-to-deploy, self-contained [cloud assembly](https://github.com/awslabs/aws-cdk/blob/master/design/cloud-assembly.md), |
There was a problem hiding this comment.
cdk-bundle or cdk-pack?
| } | ||
| ``` | ||
|
|
||
| The bundler will support a growing set of step types. Initially: |
There was a problem hiding this comment.
Hook for pluggability?
rix0rrr
reviewed
Apr 1, 2019
| └── file1.txt | ||
| ``` | ||
|
|
||
| If the file `missing.json` exists in the output, it means that the synthesis output cannot be used and relies on missing context information, which must be populated in the context map. `cdk-synth` is responsible to resolve this context information through the environmental context provider system (for example, query the account for which availability zones are available in a specific region, read SSM parameters, etc). Then, `cdk-synth` must re-invoke the app with this context. Bear in mind that normally context information is cached in `cdk.context.json`, so subsequent executions will not require resolution. |
There was a problem hiding this comment.
I think we can relax this. Missing data should be an attribute of a stack. If context info for Stack A in account 12345 is missing, that shouldn't mean I can't deploy fully synthesized Stack B in account 67890.
|
|
||
| If the file `missing.json` exists in the output, it means that the synthesis output cannot be used and relies on missing context information, which must be populated in the context map. `cdk-synth` is responsible to resolve this context information through the environmental context provider system (for example, query the account for which availability zones are available in a specific region, read SSM parameters, etc). Then, `cdk-synth` must re-invoke the app with this context. Bear in mind that normally context information is cached in `cdk.context.json`, so subsequent executions will not require resolution. | ||
|
|
||
| `cdk-synth` may repeat this process up to 5 times (theoretically there could be missing context that the app will only know that it needs after it used some information from a resolved context from previous iteration). |
There was a problem hiding this comment.
Yeah 5 seems to come out of nowhere. Are you trying to say something like "the toolkit may impose a limit on the amount of reinvocations to guard against programming errors." ?
eladb
removed
package/awscl
6 tasks
eladb
pushed a commit
that referenced
this pull request
May 29, 2019
Formalize the concept of a cloud assembly to allow decoupling "synth" and "deploy". the main motivation is to allow "deploy" to run in a controlled environment without needing to execute the app for security purposes. `cdk synth` now produces a self-contained assembly in the output directory, which we call a "cloud assembly". this directory includes synthesized templates (similar to current behavior) but also a "manifest.json" file and the asset staging directories. `cdk deploy -a assembly-dir` will now skip synthesis and will directly deploy the assembly. To that end, we modified the behavior of asset staging such that all synth output always goes to the output directory, which is by default `cdk.out` (can be set with `--output`). if there's a single template, it is printed to stdout, otherwise the name of output directory is printed. This PR also includes multiple clean ups and deprecations of stale APIs and modules such as: - `cdk.AppProps` includes various new options. - An error is thrown if references between stacks that are not in the same app (mostly in test cases). - Added the token creation stack trace for errors emitted by tokens - Added `ConstructNode.root` which returns the tree root. **TESTING**: verified that all integration tests passed and added a few tests to verify zip and docker assets as well as cloud assemblies. See: https://github.com/awslabs/cdk-ops/pull/364 Closes #1893 Closes #2093 Closes #1954 Closes #2310 Related #2073 Closes #1245 Closes #341 Closes #956 Closes #233 BREAKING CHANGE: *IMPORTANT*: apps created with the CDK version 0.33.0 and above cannot be used with an older CLI version. - `--interactive` has been removed - `--numbered` has been removed - `--staging` is now a boolean flag that indicates whether assets should be copied to the `--output` directory or directly referenced (`--no-staging` is useful for e.g. local debugging with SAM CLI) - Assets (e.g. Lambda code assets) are now referenced relative to the output directory. - `SynthUtils.templateForStackName` has been removed (use `SynthUtils.synthesize(stack).template`). - `cxapi.SynthesizedStack` renamed to `cxapi.CloudFormationStackArtifact` with multiple API changes. - `cdk.App.run()` now returns a `cxapi.CloudAssembly` instead of `cdk.ISynthesisSession`. - `cdk.App.synthesizeStack` and `synthesizeStacks` has been removed. The `cxapi.CloudAssembly` object returned from `app.run()` can be used as an alternative to explore a synthesized assembly programmatically (resolves #2016). - `cdk.CfnElement.creationTimeStamp` may now return `undefined` if there is no stack trace captured. - `cdk.ISynthesizable.synthesize` now accepts a `cxapi.CloudAssemblyBuilder` instead of `ISynthesisSession`. - `cdk`: The concepts of a synthesis session and session stores have been removed (`cdk.FileSystemStore`, cdk.InMemoryStore`, `cdk.SynthesisSession`, `cdk.ISynthesisSession` and `cdk.SynthesisOptions`). - No more support for legacy manifests (`cdk.ManifestOptions` member `legacyManifest` has been removed) - All support for build/bundle manifest have been removed (`cxapi.BuildManifest`, `cxapi.BuildStep`, etc). - Multiple deprecations and breaking changes in the `cxapi` module (`cxapi.AppRuntime` has been renamed to `cxapi.RuntimeInfo`, `cxapi.SynthesizeResponse`, `cxapi.SynthesizedStack` has been removed) - The `@aws-cdk/applet-js` program is no longer available. Use [decdk](https://github.com/awslabs/aws-cdk/tree/master/packages/decdk) as an alternative.
This was referenced Aug 22, 2019
|
Thanks so much for taking the time to contribute to the AWS CDK ❤️ We will shortly assign someone to review this pull request and help get it
|
This was referenced Dec 12, 2019
This was referenced Dec 13, 2019
This was referenced Jan 20, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
A design document describing how a modular cdk toolchain
will look like, separating the various phases of deploying
a cdk app into the AWS cloud.
Fixes #1893