(aws-cloudformation) Deploy a CDK stack in a Lambda

[rantoniuk]

My use case:

• I have a pipeline stack that is defined in Typescript
• I would like this stack to be deployed dynamically via a Lambda with parameters that will be resolved during the Lambda execution

All the documentation examples that I found show how to deploy a CDK stack via a CLI cdk synth command.
What I need to achieve is exactly the same but in a Lambda, Typescript, doesn't really matter, so:

• I need to synthesize the stack out of the TypeScript objects - probably something with the use of DefaultStackSynthesizer
• then I need to pass the template to nodeJS CloudFormation client

I found this but it doesn't show how to handle the template synthesis part.

I would appreciate some info if this is the correct way to go or that there is maybe any more generic out-of-the-box way to do it.

[NGL321]

Hey @warden!

Sorry for our long delay on this. It sounds like your work may be a good candidate to use Pipelines, so you might consider using that instead of declaring your own lambdas. More to the point, I strongly recommend against using a lambda to run cdk deploy since you will need to bundle the cdk into a lambda layer and lose control over working directory.

That said, if you DO want to run deployment via lambda it will be a pretty roundabout process. The way you have described deploying above will restrict you to just a Cloudformation template (rather than the entire cdk cloud assembly), therefore you lose a large percentage of CDK functionality (no custom-constructs or assets).
As for the "right" way to do it (outside of using pipelines), you can reference this related issue, many of the suggestions are are outdated though.

You can create a new lambda and either upload the CDK CLI to a lambda layer or use a docker image lambda with the cdk prepackaged (significantly less effort). Then you insert:

import * as exec from 'child_process';

...

exec('cdk deploy');

...

or something along those lines.

The only coherent example I have been able to find is here, but even then it is not recommended.

I hope that helps!

😸 😷

[mikestopcontinues]

I think this is worth giving more consideration to. There are cases where pipelines is just a bad fit.

For instance, I run a multi-tenant app where a new stack has to be deployed when a user account is created, updated when key details change, and destroyed when the account is closed. This is in addition to pushing out bulk updates across accounts. To do this in pipelines would be incredibly inefficient, as it would have to check all user accounts for changes.

I'm not saying CDK should be redesigned lambda-first, but I think it's worth an official example of how to handle this situation. The examples out there are broken and incomplete. I've started down the path of making this work, and it's the wild west. The CDK CLI needs to be bundled in a layer. The CDK entry app needs to be bundled separately (probably with a custom cdk.json. Then the lambda func needs deploy privileges. And that's just the issues I see from the get-go.

If anything, I think the real key to this situation (and many others) is programmatic usage of CDK (filed #15851), but in the interim, there's got to be a clear way through the lambda issue.

[rantoniuk]

This is exactly the scenario/reason why I opened this issue in the first place.
We are using the ugly workaround of pipelines indeed, but I do not think this is the right approach.

What we currently have, is an integration between BitBucket Server and AWS that is not natively possible.
Each time a Pull Request is merged, new code is being uploaded to an S3 bucket that is versioned (with semver) and a new CodePipeline is created.

That last step, pipeline creation, is currently done by an artificial generic CodePipeline that creates another, component target pipeline, something like serviceX-1.1.2-pipeline. The CF template that is used to create the final pipeline is perfectly synthesizable from the CLI from the actual content of the S3 object and it looks like a perfect usecase for a Lambda to pick it up, just do some cdk synth and cdk deploy like commands via SDK.

[ishanjain28]

I am also just here to request that this is something that should be considered and made relatively simple to do.

My use case is, I want to build/test project with codepipelines(1 to 1 mirror between branches in a repo and the codepipeline instances for that repo).

Right now, Github calls the lambda function and now I want to create a codepipeline instance for that branch. I can't integrate a main/controller pipeline directly with the github repo because codepipeline is so bare bones that it just won't work.

Doing all of this in lambdas is a bit annoying/difficult to do today and Yeah, Please look into simplifying it. Thank you.

[mikestopcontinues]

My stopgap ended up to just use github actions. Since it's inside a docker image, it's pretty straightforward. And if you need extra logic, you can just use child_exec to call CDK from within JS. (I use a bulk-deploy script to quickly trigger stack changes across all my subaccounts.)

Still though, I'd really rather keep all my infra together.

[sebastian-fredriksson-bernholtz]

@NGL321 The author of #2637 mentioned that he was able to add this functionality to cdk and might open a PR if the ticket was reopened (it was not reopened).
Are you saying that his updates to cdk would no longer work? Or would the essence of his solution still work, so that there might be a fork out there that supports this, or it might be worth creating one's own fork and try to get the author's solution to work?

[fab-mindflow]

We are deploying a simple CDK app from lambda. We've successfully tried 2 options so far:

• Spawning a CDK process: cumbersome and very slow
• Using aws-cdk classes: simple and amazingly fast (especially with hotswap when that's possible depending on use case)

Here is the snippet:

  const stackArtifact = app.synth().getStackByName(stack.stackName)
  const chain = new CredentialProviderChain(CredentialProviderChain.defaultProviders)
  const sdkProvider = new SdkProvider(chain, process.env.TARGET_REGION)
  const cloudFormation = new CloudFormationDeployments({ sdkProvider })
  await cloudFormation.deployStack({ stack: stackArtifact, /* hotswap: true */ })

Hope this will become more or less the Programmatic API for CDK.

[NGL321]

Hey @rantoniuk,

I hadn't considered a use-case like that. It is definitely a valid reason to want to deploy via lambda rather than a more rigidly structured pipeline. That said, I think we are much less likely to build a high-level construct specifically for lambda deployments (especially considering the steps required), so the better approach to getting this officially contributed is with programmatic CLI access.

With that in mind I have reopened the feature-request for programmatic access to the CDK CLI. We have a long-standing RFC, but I'll have a harder time getting it attention without a related feature-request. Please go ahead and upvote and I will prod from my end.

@sebastian-fredriksson-bernholtz,

Sorry for the confusing response. It seems I wasnt quite clear about what I was referencing. The structure of some of our libraries has changed, so the code as written cant be directly transplanted. However, the principal of what the author was doing was accurate. I cant speak to why the devs decided not to pursue his implementation, but at this point, I think any contributions would be better directed at allowing programatic access to the CDK toolkit (as referenced in this RFC). Contributions to the RFC would be brilliantly helpful and if you @ me I will nudge to make sure there is a response sooner than later.

If you need an immediate resolution your best bet is to fork the repo as suggested.

[mikestopcontinues]

@NGL321 This is great news. Thanks for listening!