Proposed approach for build steps in deps which are not in make node

[mhdawson]

Basic principles

1. Pulling specific versions of tools to to build dependencies from npm is ok
   npm guarantees immutability, and guarantees to give you the same thing for a given version

2. Code that gets pulled into Node.js deps

• Ideally source code is built/transformed in Node.js build (make node)

• Should be able to build from canonical source

  • Project should have copy of canonical source, GitHub is not immutable, and no guarantee that is is available later
  • Project should be able to float patch on dep if needed
    • Should be able to reproduce same result
      • Need to know and have access to specific versions of tools
    • No dynamic downloads

• Couple of categories that are currently worrying in terms of these principles

  • WASM blobs
  • Minified JavaScript
  • TypeScript

Proposed approach for build steps/transforms that cannot done in make node

• Source for transform/build steps outside of make node is is location/repo under the control of the Node.js project
  • Could be:
    • deps subdirectory
    • fork of repo
    • copy of tarball
    • etc.
  • updater will store copy of canonical source in one of the above, build will be done from versioned copies under control of project.
• All build/transform steps run in Docker container
• Docker container contains all tools, etc., needed other than npm dependencies
  • We store copy of container in location under control of project
  • We store copy of dockerfile used to build container along with container
• Build step run by updater uses known container, known canonical source

Next Steps:

• Figure out how we would store containers
• Figure out how we would store tarballs
• Look at undici and outline how we would implement proposed approach above

[mhdawson]

From discussion in last session related to auditing build deps with @marco-ippolito, and earlier discussion with @richardlau as well.

[mhdawson]

@nodejs/security-wg, this has been on the agenda a few times, we assume there are no concerns/objections if we don't hear by next week and move to the next step of PR's the the basic principles into Node.js doc to get broader feedback.

[mhdawson]

Looks like we should be able to use ghcr.io to store container images. https://github.com/features/packages, https://docs.github.com/en/packages/learn-github-packages/introduction-to-github-packages

[mhdawson]

This repo shows some experimentation on createing a repo to generate wasm builder images and storing them in ghcr.io

https://github.com/mhdawson/node-wasm-build/

There is a GitHub action to build and publish. Together with the Dockerfile it tries to capture/keep all key data about what went into the build.

[mhdawson]

Next step will be to look at using the image created to build undici, llhttp instead of building the docker container on the fly as part of the builds for undici, lltttp

[marco-ippolito]

I agree with the approach, we would need to add to the source code on node a folder with the dockerfile (I assume its going to be one that covers all the dependencies but there could be more) and use it in the github action where we do the dependency update

[github-actions]

This issue has been inactive for 90 days. It will be closed in 14 days unless there is further activity or the stale label is taken off.

[mhdawson]

Sorry for not getting to this sooner. Still plan to work on it.

[mhdawson]

Spent some time on this over the last few days.

Newer repo that builds/releases container - https://github.com/mhdawson/wasm-builder

• uses release-please to manage releases
• builds multi arch docker contain to ghcr for each release that supports both x86 and arm

Fork/Branch of undici which uses the container to build, should work on arm based macs with docker, but I don't have one to try.

• https://github.com/mhdawson/undici/tree/wasm-build-experiment

Next I want to look at how the wasm is built for the module lexer to see how/if the same container can be used

[mhdawson]

Ok, I have now built the three dependencies that directly use wasm and a branch with the changes that were need for each of them.

The approach is based on what was being used in undici as it was the most consistent/reproduciable to start in my opinion.

The changes for each of them:

• undici - nodejs/undici@main...mhdawson:undici:wasm-build-experiment
• cjs-module-lexar - nodejs/cjs-module-lexer@main...mhdawson:cjs-module-lexer:wasm-build-experiment
• amaro - nodejs/amaro@main...mhdawson:amaro:wasm-build-experiment

For undici and cjs-module-lexar the WASM built seems to be the same as before.
For amaro the WASM is different, I believe because the sytem installed rust is a lower version than the latest that was being used previously.

The common container is built/published by - https://github.com/mhdawson/wasm-builder which we would move under the Node.js organization if we move forward with this approach.

The advantages include:

• persistent container so we can rebuild with exactly the same environment/versions of tools if needed at a later time
• metadata for versions of tools used captured and available (in metadata directory in the container) and captured in a consistent way across dependencies. (and earlier attempt to do this in the undici repo was broken by later updates)
• hopefully will reduce the different versions/tools that are used across the dependencies to build WASM for a given Node.js release
• deps can pin versions used based on hash of container they pull

Possible disadvantage

• Container contains superset of tools need to build any one dep. Right now thinking that is a reasonable tradeoff to only have to build a single wasm-builder container.

One thing to note is that the approach (inherited from what undici was doing) relies on the system packages in alpine for most of the components. The advantage is that you should get a set of tools that work together and they are easy to install. The disadvantage is that you cannot necessarily get the very latest version of a component only the versions which were built for the version of alpine on which the base container is installed.

[mhdawson]

PRs to make undici and amaro to use the common container have landed, next is to work on CJS lexer

[mhdawson]

PR to update cjs-module-lexer - nodejs/cjs-module-lexer#105

PR to document approach - nodejs/node#55940