RFC: 0049 Continuous Delivery for CDK Apps

[eladb]

This PR includes the RFC for supporting CI/CD for CDK apps of any complexity.

It also includes the RFC for cdk-assets which is a tool for publishing CDK assets, and is part of the CI/CD solution.

Tracking issue: aws/aws-cdk-rfcs#49
Addresses: #1312

---

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

[rhboyd]

The Pipeline will be CICD as well? How is that deployed? its own pipeline? (Answered later)

New Environments: Can we leverage some functionality in AWS Organizations to "pre-trust" the pipeline?

Controlled deployment environment: I'm not sure what this is getting at. Is this talking about the target environment? (Answered Later i think. It's using a CDK-team managed build infrastructure?)

There’s a one-to-one mapping between an app and a pipeline: One-to-many? There could be use-cases where an app is deployed in several different pipelines, but they share a code-base.

We are not optimizing this experience to support any CD tool: Except CodeBuild? oh, coming back to this later, it looks like the goal is to be provider agnostic. +1

It reads like you're going to be hosting a deployment management SaaS, what's this going to cost me? It appears that you're going to spin up a CB job for each pipeline stage, am I going to be paying for that?

I like the deployment process. I'd just like to make sure we keep some kind of run-order like setting so I can have a more granular control over my deployment process.

[kohidave]

This is an interesting idea! One thing that concerns me is that when you update a CodePipeline, any revision being run through the Pipeline (and in this case, since it's a self-updating Pipeline, this revision), will be stopped.

docs:

The update-pipeline command stops the pipeline. If a revision is being run through the pipeline when you run the update-pipeline command, that run is stopped. You must start the pipeline manually to run that revision through the updated pipeline.

This might be confusing if you update the Pipeline as well as some other infrastructure.

[jonny-rimek]

In order to prevent re deploying every lambda on every deploy, the cdk.out should be cached between pipelines. This way cdk "remembers" the hash of the lambda and can tell which function changed and which didn't and thus needs to be redeployed.

This is pretty important in my opinion, if you have ~30 go lambdas you would send around 200mb on every deploy, which means the size of your deployment bucket will grow rapidly.
Also, after redeploying every lambda will also code start (obviously), which means performance of you application also suffers.

[eladb]

@rhboyd

New Environments: Can we leverage some functionality in AWS Organizations to "pre-trust" the pipeline?

That's definitely something we'd like to enable at some point, but we need to look into AWS Organizations in general. I think there is no reason this won't be possible. One thing that we are thinking about is to allow extension points for cdk bootstrap so teams can customize how the bootstrap stack looks and add any additional resources that they may need for their internal use case. Added to "Roadmap".

Controlled deployment environment: I'm not sure what this is getting at. Is this talking about the target environment? (Answered Later i think. It's using a CDK-team managed build infrastructure?)

When I say "controlled deployment environment" I mean that we want to make sure that the runtime environment from which "cdk deploy" is executed, and which usually runs with administrative privileges, is well-defined and does not run arbitrary user-code.

However, I've recently realized that the current design is faulty in that assumption that user code will not run from the deployment action because at the moment we have coupled asset bundling and publishing with deployment. This means that, for example, docker build must be running on this environment, which is not only arbitrary user code but also required root on the host. So we need to go back to the drawing board to see how to mitigate this major risk.

There’s a one-to-one mapping between an app and a pipeline: One-to-many? There could be use-cases where an app is deployed in several different pipelines, but they share a code-base.

Can you provide a more concrete example? I am not sure I understand the idea.

It reads like you're going to be hosting a deployment management SaaS, what's this going to cost me? It appears that you're going to spin up a CB job for each pipeline stage, am I going to be paying for that?

In the current design, yes. Users will be paying for the codebuild jobs that run within their pipeline (unlike CFN actions). We will look into changing that.

I like the deployment process. I'd just like to make sure we keep some kind of run-order like setting so I can have a more granular control over my deployment process.

The CodePipeline construct library supports this I believe, @skinny85 is that right?

[skinny85]

I like the deployment process. I'd just like to make sure we keep some kind of run-order like setting so I can have a more granular control over my deployment process.

The CodePipeline construct library supports this I believe, @skinny85 is that right?

Yes, absolutely.

[vaneek]

Something feels a bit off here in the approach in that a pipeline is defined as a stage in a pipeline. While there are many interpretations of what constitutes a pipeline, its probably best to align with the general AWS interpretation as seen in the Codepipeline implementation. In a generic sense, a pipeline is nothing more than an application itself and should follow a serverless/ephemermal model using the same infrastructure-as-code principles or better yet, Pipeline-as-Code(PaC).

This is a bit of a chicken or egg situation, but as mentioned, there needs to be a bootstrap action to get started - this should be an option to deploy the actual pipeline stack for a CDK App (that includes any subordinate pipelines required to build assets and artifacts), be it a simple pipeline or a pipeline of pipelines, so to speak.

[jogold]

It's common to have branch or tag based deployments in CI (master branch to prod env, dev branch to dev env, etc.). How would this be handled here?

I currently rely on a stage context variable derived from a env var to suffix my stack names and adapt resources configuration.

[vaneek]

I also recall an AWS blog post on creating a branch based pipeline and destroying the pipeline when done with the branch.

[rhboyd]

@vaneek here is the reference architecture from the blog post about the pipeline

https://aws-quickstart.s3.amazonaws.com/quickstart-trek10-serverless-enterprise-cicd/doc/serverless-cicd-for-the-enterprise-on-the-aws-cloud.pdf

[jewelsjacobs]

It looks like this POC requires the manual creation of a third party role with temp credentials which you are supplying an EXTERNAL_ID for. I love the idea of passing temp credentials to the project build!

How does everyone feel about having CDK generate this role and an external id?

Referencing this issue: https://github.com/eladb/cdkcd-test/issues/1

[eladb]

Follow up in #3437

[eladb]

Sorry, accidentally closed

[eladb]

Conversation about this here: #3555

Thanks everyone for the comments. We've been discussing some of the comments and here is an initial takeaway: we can't escape decoupling "prepare" and "deploy" in order to mitigate risks related to running docker build in an environment that has administrative privileges, and in order to allow the use of the stock cloudformation actions for deployments (to address concerns related to costs and to constraint the administrative IAM role in remote accounts to the cloudformation service principal).

To that end, we will introduce a new command cdk package STACK which will take a cloud assembly as an input and will perform the following:

1. For each asset, it will build the asset (i.e. create a zip archive from a directory or build a docker image).
2. Publish all assets to S3/ECR in the destination accounts. To that end, this command will require limited "publishing" privileges in the destination account, which will be granted through the aforementioned bootstrapping process.
3. It will then take the cloudformation template for this stack and assign default values to all asset parameters which will point to the actual locations of these assets.
4. Then, it will upload the concrete template to S3 and return it's URL.

This URL can be used as a self-contained token to deploy a specific instance (in time) of the stack.

This command will allow us to separate actions of asset preparation and deployment for each stack, and use the stock cloudformation action for the actual deployment of the stack.

This approach will also allow users who prefer not to use CodePipeline to take advantage of these more granular building blocks and integrate them into their own CI/CD systems.

Would love to hear people's thoughts on this direction.

[eladb]

@jonny-rimek wrote:

In order to prevent re deploying every lambda on every deploy, the cdk.out should be cached between pipelines. This way cdk "remembers" the hash of the lambda and can tell which function changed and which didn't and thus needs to be redeployed.

According to this RFC, lambda bundles are identified based on the hash of their source (be it the contents of the directory specified in lambda.Code.asset() or the hash of the zip file. This means that these lambda functions will only get updated if their source changes.

Does that make sense?

[eladb]

@jogold wrote:

It's common to have branch or tag based deployments in CI (master branch to prod env, dev branch to dev env, etc.). How would this be handled here?
I currently rely on a stage context variable derived from a env var to suffix my stack names and adapt resources configuration.

You can still specify a context variable during build when you invoke cdk synth and then the cloud assembly will include an app synthesized with this context in mind. Makes sense?

[eladb]

@EdoBarroso wrote:

ProServe usually recommends customers to implement ALZ/CT as a foundation to their journey to the cloud. I would really love to see this process embedded in some way into ALZ/CT deployment, so we can leverage CDK adoption for IaC in our Engagements.

Added information about how to use ALZ/CT for bootstrapping.

[aws-cdk-automation]

AWS CodeBuild CI Report

• Result: FAILED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[eladb]

@jamiepmullan wrote:

At the moment, if you try and deploy a lambda with a new version via CDK and the underlying code hasn’t been modified, you get an error saying that version already exists. What will be the behaviour with this going forward? If you have 20 lambdas all with different version numbers, it'd be quite hard to keep track of; with the added complexity of adding new code versions from the pipeline process so the '$LATEST' flag wouldn't work.

Not sure I am familiar with the error you are describing. Sounds like it shouldn't be the experience. At any rate, lambda bundles are identified by their source hash. This means that they will change only if their source changes.

It would be good if you could run ‘external’ images within the pipeline as a stage i.e. TDD testing that invokes a service to verify whether the new stack is working before making version live. Are there any plans to allow this? I know at the moment in Lambda Deployments you can have pre and post hooks, but these are very geared towards having your tests in a separate lambda in the stack.

You define your pipeline. We just provide the building blocks. So you can define any pipeline you want.

The Pipeline being able to effectively 'stop' after the code build stage if it's a PR branch. Assume this will be down to how we implement the solution?

This change doesn't attempt to implement the CI/CD system, just to provide building blocks (at varying levels of abstraction) for integrating into CI/CD systems. We will offer high level stages/actions for CodePIpeline and tools that can be used in systems like GitHub Workflows, Jenkins, etc.

During the bootstrapping process cdk will create a lot of resources for you, however will you be able to provide these to the pipeline i.e. so we can manage the IAM roles, S3 buckets from the likes of a ‘service creator’ script?

The idea is that bootstrapping should be very simple. Sort of like the BIOS of your computer. Ideally those resources should be quite static. We try to make sure that the bootstrapping process itself can be done through tools like StackSets and Control Tower so you can leverage these existing services as much as possible to manage these environments.

Also didn't read anything on how old resources/states would be destroyed/deleted? What will the behaviour be for this i.e. keeping versions n-1?

Resources will be destroyed automatically as long as their stacks exist. Destruction of stacks and environments is currently a gap in this RFC. I will add that.

[eladb]

@gataka wrote:

your cdk-assets-spec would give the CDK a "CloudFormation package" flavour. In my opinion, one big missing thing when you work with the CodePipeline-CloudFormation-Actions and CDK.

In a sense, yes. The "cloud assembly" is that package.

add parameters to cdk-assets publish that you can specify a target bucket/prefix where to store the assets. This makes it easier during the CI-Build to change it. (e.g. you can use the Pipeline-ArtifactStore as a target)
let the cdk-assets publish generate a json file which can be used for the parameterOverrides property of the CloudFormationCreateReplaceChangeSetAction or append the parameters in an existing parameters.json for the templateConfiguration property. Another way to set the values in the generated templates is, to write Default values to the Asset-Parameters or remove these parameters and replace their usages with the URL (like cfn package) , this will save a lot of complexity.

You will be able to specify bucket/repo in your CDK app and the cdk-assets tool will use those. We intentionally wanted these values to resolve during synthesis and not during deployment. Can you find any specific use cases where this approach breaks down? One of the main reasons is that there is no standard way to wire cloudformation parameters in CI/CD systems and we realized that if we just avoid using parameters in the asset system, integrating our CI/CD solution into other non-CodePipeline systems would be much simpler. It will also give users much more flexibility to encode any custom unique logic for determining where assets need to be published to since it will be Just Code ™️ in their App.

Please add Nested-Stacks(the templateUrl) also to the supported Assets. Because these is the only way to create multiple stacks with the CodePipeline-CloudFormation-Actions. CDK default behaviour is not helpful, because it generates separate stacks and imports their values.

Nested stacks are going to be fully supported by this solution.

I hope these points will fit to your purpose. It would be very nice, when CDK becomes a package function.

Thanks!

[mergify]

Title does not follow the guidelines of Conventional Commits. Please adjust title before merge.

[aws-cdk-automation]

AWS CodeBuild CI Report

• Result: SUCCEEDED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[aws-cdk-automation]

AWS CodeBuild CI Report

• Result: SUCCEEDED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[jonny-rimek]

@jonny-rimek wrote:

In order to prevent re deploying every lambda on every deploy, the cdk.out should be cached between pipelines. This way cdk "remembers" the hash of the lambda and can tell which function changed and which didn't and thus needs to be redeployed.

According to this RFC, lambda bundles are identified based on the hash of their source (be it the contents of the directory specified in lambda.Code.asset() or the hash of the zip file. This means that these lambda functions will only get updated if their source changes.

Does that make sense?

yes. I reread the rfc and if I understand it correctly the information about the hash is saved inside the assets.json. That means before running cdk synth during build stage the old assets.json (or the whole cdk.out?) is downloaded so cdk compare it with the old hashes?

I get the hashes are used to check if something is changed, I just don't understand what the values are compared to as each run of code build starts in an empty container, where as if I run cdk deploy on my machine the cdk.out stays (but it's gitignored)

[NetaNir]

We should consider not always creating an ECR repo in the bootstrap process, this will allow bootstrapping regions which does not yet have ECR available. This might also be further generalized to allow the extension of assets in the future to locations other than S3 and ECR

[aws-cdk-automation]

AWS CodeBuild CI Report

• Result: FAILED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository