Proposal to add Yarn to the existing images

[pesho]

Disclaimer: This is a proposal PR which requires discussion. Please do not merge unless consensus has been established.

This PR adds Yarn as an alternative package manager to the existing images. It's added as a separate layer and the size increase is small (up to ~3.5 MB currently; for example the 7.6.0-alpine tag increases in size from 56.2 MB to 59.7 MB). The update.sh script is updated to automatically detect the latest stable Yarn release using the GitHub API.

Adding Yarn has been by far the most upvoted request in our issue tracker (#243).

Notes:

• There is a previous PR (Docker node yarn variant #260) which also adds Yarn. The current PR is different in that it doesn't create new tags, instead adding Yarn to the existing ones. Also, the installation method is different (direct download of Webpack-bundled binary instead of npm install) and the size increase is much smaller.
• The installation method used is to simply download the webpack-bundled .js file, GnuPG-check its signature, put it in /usr/local/bin and chmod +x it. This was chosen because of the small size (~3.2 MB), in contrast to all other available methods (tarball, DEB package, NPM) which are currently much larger when unpacked (at least ~14 MB). The -legacy bundle variant is used, because the modern one does not support Node 4.x (which we do support).
• The GitHub API https://yarnpkg.com/latest-version is used by the update.sh script to automatically detect the latest release marked as stable by the Yarn team. Unstable pre-releases are not considered.
• Yarn is added as a separate layer in order to have cleaner separation from the rest of the code. In the Alpine image this comes at a small build-time efficiency cost - gnupg and curl are installed/uninstalled twice. This shouldn't be a big issue, and it helps that the installation method is mostly the same across all image variants.
• The upstream Node.js project currently bundles only NPM, not Yarn (discussion at bundle with yarn and allow which package manager during build. node#9161) so this PR can be viewed as a deviation from upstream. There is already precedent for that, as we build our own Alpine binary, because upstream does not yet provide one.
• The auto-generated Dockerfiles are not included in this PR (yet) in order to avoid merge conflicts in the future. They are a simple ./update.sh invocation away.

Related: #243 #260 yarnpkg/yarn#2721 nodejs/node#9161

[chorrell]

In general I'm pretty OK with this :)

The one thing I would change is how yarn versions are handled. Currently the update.sh script grabs the latest stable/release version via:

yarnVersion="$(curl -sSL --compressed https://api.github.com/repos/yarnpkg/yarn/releases/latest | grep '"tag_name":' | head -n 1 | cut -d '"' -f 4 | cut -d v -f 2)"

I'm more inclined to be more conservative here and pin things to SemVer major. I'm assuming that at some point when yarn hits v1 there will be breaking changes. So something like 0.x.x were we always get the latest 0.minor.patch.

[daveisfera]

Is there a use case where npm and yarn both need to be available? I personally would prefer to have a separate image for yarn that doesn't have npm in it.

[pesho]

@daveisfera a lot of package install scripts depend on npm being available in $PATH, so npm is not optional.

@chorrell I agree in general about SemVer-major pinning. We could probably use the NPM registry to resolve a SemVer selector to a specific version, with a query like https://registry.yarnpkg.com/yarn/0.20.x. There seem to be a few issues at the moment though:

• As yarn is yet to hit v1, pinning to 0.x does not work well right now, as it resolves to unstable rc versions. So we would have to pin to SemVer minor, e.g. 0.20.x, and update the pinned value often.
• Alternatively we could use latest until Yarn hits v1. But right now the NPM latest dist-tag points to version 0.18.2 which is older than 0.20.3 which is latest stable in GitHub. Not sure why the discrepancy, but in my amateur opinion the newer version is more stable than the old one.

We should probably ask the Yarn maintainers for advice as well. /cc @Daniel15

[Daniel15]

If you do want to pin the version number, the best solution right now is probably to hard-code it into your script or Docker file and manually update it for new releases. The version numbers are moving quickly so I don't know whether it's worth pinning it while we're still at 0.x releases. Maybe once 1.x comes out? We have one endpoint https://yarnpkg.com/latest-version to get the latest version number, perhaps we could add /latest-1.x-version to grab the latest 1.x version number once 1.0 comes out.

right now the NPM latest dist-tag points to version 0.18.2 which is older than 0.20.3 which is latest stable in GitHub. Not sure why the discrepancy, but in my amateur opinion the newer version is more stable than the old one.

Yeah I'm not quite sure what happened there. cc @bestander
0.20.3 is the latest version, as shown on the Yarn site and when you hit https://yarnpkg.com/latest-version (which is the 'official' way of getting the version number for the latest stable release of Yarn) .

[Starefossen]

1+ to add yarn to all images and not maintaining separate yarn variants. Also +1 to SemVer major pinning.

[pesho]

Updated PR according to comments from @Daniel15

@chorrell @Starefossen in light of the info provided by @Daniel15 I think the best course for now is to keep using the latest version recommended by the Yarn team at https://yarnpkg.com/latest-version. After Yarn v1 is released we may switch to semver-major pinning. But it wouldn't work very well before that.

[chorrell]

Looks good to me.

I'm good to merge this later today if no one objects :)

[pesho]

@Starefossen @chorrell thanks for reviewing!

I just added the auto-generated dockerfiles to the PR as well, to try and get a Travis result before merging (although it was timing out even before that).

[chorrell]

@pesho Cool.

Are the timeouts an issue with nodejs-github-bot?

[Starefossen]

@chorrell It looks like Travis is building Node.js from source? https://travis-ci.org/nodejs/docker-node/jobs/206192823 Not sure whats up with that...

[Starefossen]

That is the Alpine images building from source... looks like they take a couple of minutes to complete

[chorrell]

@Starefossen Yep. But typically the build actually succeeds in Travis but the status on the PR never gets updated and shows pending. I was wondering if nodejs-github-bot was timing out after a set time?

I guess longer term we need musl/Alpine binaries :)

[pesho]

@Starefossen Yep. But typically the build actually succeeds in Travis but the status on the PR never gets updated and shows pending

Actually that's what probably was happening. Not "timing out" as I said.

[chorrell]

Looks like we hit the same ha.pool.sks-keyservers.net timeout issue that's happening in #340 :(

[terribleplan]

This is awesome, also it broke my build on node:6 because we npm install -g yarn in our CI, which now fails. ¯\(ツ)/¯

[Daniel15]

@terribleplan - Do you get an error? I would have thought that'd just override the yarn executable.

[terribleplan]

@Daniel15

npm ERR! Refusing to delete /usr/local/bin/yarn: node_modules/npm/bin/npm-cli.js symlink target is not controlled by npm /usr/local
npm ERR! File exists: /usr/local/bin/yarn
npm ERR! Move it away, and try again.

[daveisfera]

Yes, it broke my builds as well, but once I removed the installation of yarn it worked just fine.

[Daniel15]

Ah, good catch. I think that'd be specific to installing Yarn via npm, which is discouraged in the Yarn documentation so it shouldn't be commonly used. Anyone installing Yarn via the Linux packages would still be fine, they'll just end up with two versions of Yarn on the system.

[levino]

@Daniel15 Yeah, but if you are a nvm user, you just don't do apt-get install yarn. It feels very very wrong.

[Daniel15]

but if you are a nvm user, you just don't do apt-get install yarn.

This issue is specifically about Docker. Why would you use nvm in Docker? If you want to use multiple different Node.js versions, you'd probably have multiple different Docker images rather than using nvm within a Docker image.

apt-get works fine if you're using nvm, you just need to use the --no-install-recommends flag so it uses your nvm version of Node.js rather than installing Node.js via apt-get.

[levino]

I know. But why not do it in the docker container with npm install too. It was merely a remark, instead of a call to action. I don't think that tools like yarn should ever be installed with sudo. And as such I was npm installing it in my containers too. Which worked fine, until this PR got merged, which broke the "undocumented api" of my node base image. I found the culprit and changed it, so I am fine but still I find it a little annoying.

[williamboman]

I'd like to see this incorporated in the onbuild flavor as well. Thoughts?

[pesho]

I'd like to see this incorporated in the onbuild flavor as well. Thoughts?

@williamboman the onbuild variant is heading towards deprecation (docker-library/official-images#2076)

[ljharb]

I don't think this was a good decision - yarn doesn't ship with node, so it shouldn't ship with an official node docker image. Conversely, if it is shipped with something official, it should ship with everything.

This should be reverted.

[daveisfera]

It adds 5.5 MB to the alpine images so less than 10% increase in the alpine image size (and significantly smaller for the other base images) and is extremely convenient for those using yarn (which is a sizable percentage of the community), so I think it's well worth the addition

[ljharb]

It may be convenient, but node does not ship yarn, and thus the official node docker image should not ship it.

If the value is there, then node itself should ship with yarn before this docker image does.

[daveisfera]

node also doesn't ship ls, curl, head or any of the other applications that are in /usr/bin/ in the buster-slim image. Those take up 19 MB (almost 4x the size of the entire yarn layer) and until multi-stage builds are supported in the official images we're "stuck" with those.
Keep in mind that docker is a tool to make building/deploying software easier not a way to package node and only node. I personally appreciate this convenience (along with a sizable portion of the node ecosystem) and, if you don't use yarn, then there's no impact to users of the docker images other than the extra 5.5 MB layer.
Also, npm has made significant strides, but until npm reaches feature parity with yarn, I can't switch. For example, yarn has workspaces that to my knowledge has no parallel in npm (I know Lerna is getting better but still isn't a full replacement) and without that working in a mono-repo with multiple apps is extremely painful.

[nschonni]

node also doesn't ship ls, curl, head or any of the other applications that are in /usr/bin/ in the buster-slim image.

@daveisfera those are part of the base image, not something added by the node image. In the Alpine image where they are added on top of the base image, they are removed after the build.
I think this discussion is better had on #777 rather than a merged PR

[Daniel15]

IMO there should be one variant with both npm and Yarn (for dev/builds), and one variant with neither npm nor Yarn (for production). You shouldn't need a package manager in your production Docker image. This would match what other ecosystems do - for example C# /ASP.NET Core has one Docker image with the SDK for building apps, and then a separate image with just the runtime for production. ASP.NET does this using a Docker multi-stage build: https://docs.microsoft.com/en-us/aspnet/core/host-and-deploy/docker/building-net-docker-images?view=aspnetcore-3.0
Also it's worth noting that if you use Yarn with plug-n-play, you'll end up with more free disk space than if you used npm, given npm leaves node_modules directories all over the place.

[nschonni]

@Daniel15 there is also a separate discussion for that scenario in #404