Skip to content

Conversation

@rix0rrr
Copy link
Contributor

@rix0rrr rix0rrr commented Apr 1, 2021

The yaml-cfn library is used as an implementation detail of
cloudformation-include, aws-cdk, decdk, and the monopackages.

Un-jsii it and bundle it into the framework packages that need it.

Does not need to be accessed over jsii and does not need to be exposed
in the monopackages.


The previous version of this PR (#13699) had to be reverted (b1ffd33), because of a bug in NPM
that causes .npmignore files in symlinked, bundled dependencies to be ignored.

This in turn caused undesired .ts files to be bundled, which caused ts-node
to exit with an error.

Introduce a new cdk-postpack tool which will strip files that NPM should have
ignored from the generated tarball, and add some verification to the pack.sh script
to make sure we don't accidentally bundle .ts files.


By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

rix0rrr added 3 commits March 30, 2021 17:29
The `yaml-cfn` library is used as an implementation detail of
`cloudformation-include`, `aws-cdk`, `decdk`, and the monopackages.

Un-jsii it and bundle it into the framework packages that need it.

Does not need to be accessed over jsii and does not need to be exposed
in the monopackages.
The `yaml-cfn` library is used as an implementation detail of
`cloudformation-include`, `aws-cdk`, `decdk`, and the monopackages.

Un-jsii it and bundle it into the framework packages that need it.

Does not need to be accessed over jsii and does not need to be exposed
in the monopackages.

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
@gitpod-io
Copy link

gitpod-io bot commented Apr 1, 2021

@rix0rrr rix0rrr self-assigned this Apr 1, 2021
@rix0rrr rix0rrr requested a review from a team April 1, 2021 12:08
@mergify mergify bot added the contribution/core This is a PR that came from AWS. label Apr 1, 2021
Copy link
Contributor

@eladb eladb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Something feels a bit off here. yaml-cfn is 60 lines of code. Does it really make sense to add all these hacks just to be able to reuse these 60 lines in two places in our code?

Did you consider the following options:

  1. Get rid of this module and copy & paste (rarely the right choice, but perhaps that's the rare case where this is a reasonable solution).
  2. Publish this module as a separate library from outside our mono-repo and use it like a normal module so we don't need all this hackery.

@rix0rrr
Copy link
Contributor Author

rix0rrr commented Apr 2, 2021

What's feeling off is the wavering tower of cards that is the infrastructure we need to build upon. I would prefer to shore up said tower of cards, rather than accept anything can just fall over at any time and just living with that.

Could we copy/paste? Yes we could. As you said, I'd rather not.

Same for new repos. Yes we could, but it feels like an unscalable solution.

Seems like being able to bundle monorepo packages would be a good capability to have in our back pocket going forward, in any case.

Copy link
Contributor

@eladb eladb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel this is the wrong direction. The fact that we need to massage the tarball after its created doesn't smell right, and the sheer amount of code required to do that feels like it's introducing unneeded risk to our packaging process (which is in itself fragile).

In this particular case, I think simply copy & pasting this single function is probably a safer and more pragmatic solution.

I'll leave it to you to decide if this is still the approach you think we should take.

Approving with "do-not-merge".

@eladb eladb added the pr/do-not-merge This PR should not be merged at this time. label Apr 4, 2021
@rix0rrr rix0rrr removed the pr/do-not-merge This PR should not be merged at this time. label Apr 6, 2021
@rix0rrr
Copy link
Contributor Author

rix0rrr commented Apr 6, 2021

I want to have the ability to bundle monorepo packages going forward.

@mergify
Copy link
Contributor

mergify bot commented Apr 6, 2021

Thank you for contributing! Your pull request will be updated from master and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

@aws-cdk-automation
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: AutoBuildProject89A8053A-LhjRyN9kxr8o
  • Commit ID: b205341
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@mergify
Copy link
Contributor

mergify bot commented Apr 6, 2021

Thank you for contributing! Your pull request will be updated from master and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

@mergify mergify bot merged commit 1cd175d into master Apr 6, 2021
@mergify mergify bot deleted the huijbers/yaml-cfn2 branch April 6, 2021 08:08
eladb pushed a commit that referenced this pull request Apr 6, 2021
This reverts commit 1cd175d.

Due to build break.
eladb pushed a commit that referenced this pull request Apr 6, 2021
rix0rrr added a commit that referenced this pull request Apr 6, 2021
`aws-cdk-lib` needs to have no dependencies (except for `constructs`),
and we don't want to support this package as part of its public API.
Therefore, it needs to be bundled into all packages that consume it.

NPM has a bug that makes it impossible to `bundledDependencies` a
package that is linked from the monorepo.

We tried to work around that bug but it blocked our build because the
`.tgz` is sometimes not available during the `postpack` phase (#13933).

Simplest way around this now is to just copy/paste the implementation
of `yaml-cfn` 3 times. Should we ever need to bundle monorepo
dependencies again in the future, we can try to revive the `postpack`
solution of #13933.
mergify bot pushed a commit that referenced this pull request Apr 7, 2021
`aws-cdk-lib` needs to have no dependencies (except for `constructs`),
and we don't want to support this package as part of its public API.
Therefore, it needs to be bundled into all packages that consume it.

NPM has a bug that makes it impossible to `bundledDependencies` a
package that is linked from the monorepo.

We tried to work around that bug but it blocked our build because the
`.tgz` is sometimes not available during the `postpack` phase (#13933).

Simplest way around this now is to just copy/paste the implementation
of `yaml-cfn` 3 times. Should we ever need to bundle monorepo
dependencies again in the future, we can try to revive the `postpack`
solution of #13933.


----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
hollanddd pushed a commit to hollanddd/aws-cdk that referenced this pull request Aug 26, 2021
The `yaml-cfn` library is used as an implementation detail of
`cloudformation-include`, `aws-cdk`, `decdk`, and the monopackages.

Un-jsii it and bundle it into the framework packages that need it.

Does not need to be accessed over jsii and does not need to be exposed
in the monopackages.

----

The previous version of this PR (aws#13699) had to be reverted (aws@b1ffd33), because of a bug in NPM
that causes `.npmignore` files in symlinked, bundled dependencies to be ignored.

This in turn caused undesired `.ts` files to be bundled, which caused `ts-node`
to exit with an error.

Introduce a new `cdk-postpack` tool which will strip files that NPM *should* have
ignored from the generated tarball, and add some verification to the `pack.sh` script
to make sure we don't accidentally bundle `.ts` files.

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
hollanddd pushed a commit to hollanddd/aws-cdk that referenced this pull request Aug 26, 2021
hollanddd pushed a commit to hollanddd/aws-cdk that referenced this pull request Aug 26, 2021
`aws-cdk-lib` needs to have no dependencies (except for `constructs`),
and we don't want to support this package as part of its public API.
Therefore, it needs to be bundled into all packages that consume it.

NPM has a bug that makes it impossible to `bundledDependencies` a
package that is linked from the monorepo.

We tried to work around that bug but it blocked our build because the
`.tgz` is sometimes not available during the `postpack` phase (aws#13933).

Simplest way around this now is to just copy/paste the implementation
of `yaml-cfn` 3 times. Should we ever need to bundle monorepo
dependencies again in the future, we can try to revive the `postpack`
solution of aws#13933.


----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

contribution/core This is a PR that came from AWS.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants