cdk deploy: do not use docker for dependency resolution & installation

[CoreOxide]

Describe the feature

Today, CDK deploy uses docker as it's backend to collect and install dependencies. I'm proposing a way to to that without using Docker. It will also allow building for different OS targets and Lambda runtimes.

Use Case

Not needing to run (and pay for) Docker anymore when building deploying.

Proposed Solution

This method basically implements pip's dependency resolution. It creates a flat dependency list that can be fed to pip. It's required to have an already resolved dependency list to utilize pip's --platform flag, such that dependencies can be downloaded for any target OS (given they exist for that OS).

See this blog post for more information:
https://medium.com/cyberark-engineering/dont-use-docker-to-package-python-lambdas-on-your-mac-fe64b8f98144

See this repo for working code solution:
https://github.com/CoreOxide/pip_requirements_flattener

Other Information

This code is used in production-grade deployment pipelines for some time now and has a proved record.

Acknowledgements

• This feature might incur a breaking change
• I may be able to implement this feature request

CDK version used

2.69.0

Environment details (OS name and version, etc.)

MacOS 13.3.1

[peterwoodworth]

Is this specifically regarding the lambda python module?

[CoreOxide]

Is this specifically regarding the lambda python module?

Yup. I'm not sure if containers are used for any other languages, though, an if this solution can be expanded to cover them as well.

[CoreOxide]

@peterwoodworth So do you think it will be fine to introduce this new dependency resolution & installation system to support only Python Lambdas for the first iteration?
If so, I can create a PR in CDK.

[peterwoodworth]

I think we should do an RFC for this first, as this seems to be a significant overhaul of the existing implementation

[CoreOxide]

Thanks Peter. I'll try to get to it in the coming weeks.
Should this issue be closed, then?

[kevinhq]

Any update on this? It would be great if it's implemented. Docker didn't run very well on macOS (at least from my experience, it's slower... than Linux)

[CoreOxide]

@kevinhq I want to try another approach, using AWS ClodShell instead. I hope I can get ot it soon.
If it will indeed prove the better method - I will propose it in the RFC instead.

[CoreOxide]

@kevinhq , @peterwoodworth
So I finally gov some time to work on this. Created RFC -> aws/aws-cdk-rfcs#579
Should this issue be closed now?

[mrpackethead]

@peterwoodworth

I think we should do an RFC for this first, as this seems to be a significant overhaul of the existing implementation

Except the RFC process is closed, so hows that going to work?

[evgenyka]

@CoreOxide - Thank you for submitting your RFC! Please be aware that the RFC process is currently on hold (as we announced in August 2023), and we won't be able to review it immediately. I recommend staying updated with our (near) future announcements regarding the RFC process. We will use this as a tracking issue for your RFC.