Serverless Data Lake Framework 2.5.0

This is a new release for SDLF 2.x. See the release notes for SDLF 2.0.0 to learn about all the changes in this major version.

For users of SDLF 1.x, version 1 is still available on the master branch. Development of newer versions of SDLF (2.x) happens on branch main. The workshop still contains sections for version 1 as well.

What's Changed

• new stage: sdlf-stage-lambda by @cnfait in #338
• force use of cfn-lint 0.87.7 since release 1.3.0 break cloudformation-cli by @cnfait in #341
• new stage: sdlf-stage-ecsfargate by @cnfait in #342
• new stage: emrserverless by @cnfait in #344
• new stage: sdlf-stage-glue by @cnfait in #345
• [sdlf-stage-*] generic get_transform_details in routing lambda by @cnfait in #346
• pin cfn-lint to <1 instead of 0.87.7 by @cnfait in #349
• [sdlf-pipeline] support for pipelines with no source event by @cnfait in #350
• [sdlf-stage-*] put metadata in transform field, handle decimal metadata by @cnfait in #351
• [vpc] support shared vpc subnets by @cnfait in #348
• Preview: CDK Python by @cnfait in #353
• build stage-lambda and stage-glue by default by @cnfait in #355
• [sdlf-team][cdk] lakeformation data_location_access permissions by @cnfait in #356
• [sdlf-dataset][cdk] lake formation resources by @cnfait in #357
• avoid code duplication in domain-cicd by @cnfait in #359
• [sdlf-team][cdk] lakeformation tag and tag permissions by @cnfait in #360
• [sdlf-stage-emrserverless/ecsfargate] CDK Python by @cnfait in #361
• [sdlf-foundations/team][cdk] kms removal policy by @cnfait in #362
• [sdlf-*][cdk] set removal policy on stateful resources by @cnfait in #363
• [sdlf-*][cdk] extends Construct instead of Stack, add requirements.txt by @cnfait in #364
• SeedFarmer Support by @cnfait in #365
• [sdlf-foundations] generic deployspec.yaml by @cnfait in #366
• [sdlf-team] seedfarmer support by @cnfait in #367
• [sdlf-dataset] seedfarmer support by @cnfait in #369
• pin cfn-lint to <1 in github workflow by @cnfait in #370
• disable w12 check on modulestack where exception is documented by @cnfait in #371
• remove hash from glue job cfn reference & script name by @cnfait in #372
• [sdlf-cicd] glue job deployer: support for custom templates by @cnfait in #373
• [sdlf-cicd] github support as an alternative to codecommit by @cnfait in #374
• poetry/pyproject.toml for sdlf cdk modules by @cnfait in #378
• [sdlf-monitoring] logs account policy for subscription filters by @cnfait in #379
• [sdlf-monitoring] log group for data firehose lambda by @cnfait in #380
• upgrade to seedfarmer 5.0.0 (from 4.0.4) by @cnfait in #381
• [sdlf-monitoring] firehose lambda: use logging, do not merge log events by @cnfait in #382
• [sdlf-stage-*] peh_id is under transform now by @cnfait in #383
• [sdlf-datalakeLibrary] seedfarmer support by @cnfait in #384
• skip poetry install if cfn module by @cnfait in #385
• workshop seedfarmer manifest examples by @cnfait in #386
• update version to 2.5.0-beta.1 by @cnfait in #387
• seedfarmer - make sure setuptools is available by @cnfait in #388
• delete cfn stack properly with seedfarmer destroy by @cnfait in #389
• avoid hardcoding SDLF deployment role name in LakeFormation admin list by @cnfait in #390
• use globbing for cfn-lint, set ignore_templates by @cnfait in #391
• update dependencies by @cnfait in #392
• update version to 2.5.0-beta.2 by @cnfait in #393
• avoid substitution issue on macosx by @cnfait in #394
• avoid buildspec yaml syntax issues by @cnfait in #395
• [sdlf-cicd] lambda:TagResource for event-source-mapping by @cnfait in #396
• [sdlf-cicd] lambda:TagResource for event-source-mapping in team role by @cnfait in #397
• (sdlf-stage-*) don't set default dataset by @cnfait in #398
• update version to 2.5.0 by @cnfait in #399

Full Changelog: 2.4.0...2.5.0