yarn is unpackaged, and breaks installing newer versions of yarn

[mikemaccana]

So this was originally thought to be a yarn issue, yarnpkg/yarn#4407, then traced back to CircleCI https://github.com/circleci/circleci-images/issues/76, and then revealed to be this image.

The node docker image includes an unpackaged version of yarn.

This means when users install a newer yarn, per yarns official instructions, it won't work - since the unpackaged /usr/local/bin/yarn is before /usr/bin/yarn. It also means things like apt-get update will skip yarn when installing security fixes (for any containers that are long lived),

Offending line is:

docker-node/8.5/Dockerfile

Line 53 in 2924f14

&& curl -fSLO --compressed "https://yarnpkg.com/downloads/$YARN_VERSION/yarn-v$YARN_VERSION.tar.gz" \

Simple fix is to follow the yarn install procedure: https://yarnpkg.com/lang/en/docs/install/#linux-tab

curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add -
echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee /etc/apt/sources.list.d/yarn.list

[SimenB]

FWIW, the official instructions doesn't really apply to these images. That said, we could probably change to following those instructions, if the issues (size wise) has been fixed (see #337 for original PR adding yarn, and some reasoning around why the current installation method is used.)

I'm a bit sceptical about the alpine version though (yarnpkg/yarn#1326 (comment)) - I'd prefer to not rely on the community updating it. @Daniel15 any possibility of the yarn team taking over updating that?

/cc @pesho

[Daniel15]

if the issues (size wise) has been fixed

Yeah, that was fixed in Yarn 0.24.x. See #401 for the previous discussion. The standalone .js file, tarball, and Debian package are all similar sizes now. It was decided to use the tarball everywhere for consistency, but perhaps that should be changed and the Debian image could use the Debian package?

@Daniel15 any possibility of the yarn team taking over updating that?

We could probably do that, although I have no idea how it's maintained or how easy it is to update. If it can be automated then we can add it to the release process - We already update npm, Homebrew and Chocolatey as part of the release process.

[SimenB]

Ah, I've completely missed #401, thanks!

Things of the top of my head that needs to be resolved:

• All of the ways of installing yarn through package managers also allows us to skip dependencies (node in our case)
• We can rely on having the same version of yarn available across all installation methods in a reasonable timeframe (to avoid different yarn versions across variants if it coincides with a node release)
• The yarn team itself updates all of the channels

Anything else? If not, I don't really see any reason to not switch, especially if it makes it easier for consumers of this image

[mikemaccana]

I don't really see any reason to not switch

Excellent :-)

allows us to skip dependencies (node in our case)

Tarballs have dependencies too, they're just unmanaged. nodesource constantly produce packaged node too, and if the images are using an unpackaged version of node, that has similar issues, eg, security updates will be missed if the OS is updated..

Also thanks for the fast reply!

[Daniel15]

also allows us to skip dependencies

apt-get --no-install-recommends will do that. nodejs is only "recommended" by the Yarn package, to handle the case where you don't install Node.js via dpkg.

The yarn team itself updates all of the channels

If you like, you could use the .deb package on Debian, and continue to use the tarball on Alpine. The Debian repo and tarball are updated as part of every release. We'll have to ask the maintainer of the Alpine package about how difficult it'd be to automate it as part of the release process.

[pesho]

@mikemaccana you can still update Yarn manually by deleting the contents of /opt/yarn and unpacking a more recent tarball there. There is no need to insist on using a different installation method which is less suitable for an official Docker image.

[mikemaccana]

@pesho why do you feel unpackaged software is better for Docker images?

[pesho]

@pesho why do you feel unpackaged software is better for Docker images?

@mikemaccana it's not "unpackaged" just because it doesn't use the distro's default package format.

The current installation method is better because it works across all supported distros (atm Debian and Alpine). There were also other reasons at the time which are no longer valid - it allowed us to pin an exact version which doesn't change when the image is rebuilt, and its size was smallest (the DEB package used to be about 15 MB).

We also install Node itself from a tarball.

[mikemaccana]

@pesho will disagree re: tarballs being considered packages on Debian, but I understand the dpkg was previously larger and that this is now fixed. I also get that a tarball is consistent across distros, but it's inconsistent with the other software in the distro. Since more people use multiple apps on Debian than move between the same app on different distros, it's better to focus on consistency there.

[davidkarlsen]

Any progress on this? I'd like to move to yarn 1.5.1

[eyalzek]

Another option would be to unpack the tarball to /opt/yarn-$YARN_VERSION and then symlink that directory to /opt/yarn, thus allowing a much easier upgrade path by just unpacking a new version into /opt and updating the directory symlink (this will keep the binary links in tact and will reduce some maintenance). I created a small PR to demonstrate this approach: #524

[mikemaccana]

Another option would be to unpack the tarball to /opt/yarn-$YARN_VERSION and then symlink that directory to /opt/yarn, thus allowing a much easier upgrade path by just unpacking a new version into /opt and updating the directory symlink

@eyalzek Why not just use the package?

[mikemaccana]

Instructions for installing yarn on Debian: https://yarnpkg.com/en/docs/install#debian-stable

[eyalzek]

@mikemaccana the first obvious problem with this approach is that it won't work on the alpine variation.

[mikemaccana]

@eyalzek The dockerfile uses Debian jessie, not Alpine. See

docker-node/8.5/Dockerfile

Line 1 in 2924f14

FROM buildpack-deps:jessie

Obviously Alpine should use Alpine packages. Instructions for Alpine are at https://yarnpkg.com/en/docs/install#alpine-stable

[eyalzek]

@mikemaccana
docker run --rm alpine:3.4 /bin/sh -c "apk update; apk policy yarn" yarn doesn't even exist on this version on alpine and on alpine:latest:

$ docker run --rm alpine:latest /bin/sh -c "apk update; apk policy yarn"
fetch http://dl-cdn.alpinelinux.org/alpine/v3.7/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.7/community/x86_64/APKINDEX.tar.gz
v3.7.0-151-gf417903f18 [http://dl-cdn.alpinelinux.org/alpine/v3.7/main]
v3.7.0-150-g3ea1ba4100 [http://dl-cdn.alpinelinux.org/alpine/v3.7/community]
OK: 9050 distinct packages available
yarn policy:
  1.3.2-r0:
    http://dl-cdn.alpinelinux.org/alpine/v3.7/community

only version 1.3.2 exists, so this solution doesn't really offer proper control of the yarn version installed. In general I do agree that using package managers and pinning the version is a good way to go, but in this case I tried to offer a simple solution with minimum code change.

[mikemaccana]

In general I do agree that using package managers and pinning the version is a good way to go, but in this case I tried to offer a simple solution with minimum code change.

Cool and understood. Sounds like it might be worth filing a bug for the Alpine instructions at https://github.com/yarnpkg/website?

[Daniel15]

only version 1.3.2 exists, so this solution doesn't really offer proper control of the yarn version installed.

The Alpine Yarn package is currently community-maintained (which is fairly common with packages for Linux distros). The Yarn Debian and CentOS packages are officially maintained by the Yarn team and are always updated as soon as new releases come out.

[SimenB]

Any chance of the yarn team taking over maintenance of the alpine package?

[Daniel15]

Any chance of the yarn team taking over maintenance of the alpine package?

It's possible... I don't know anything about how the Alpine package is built though. I was trying to use fpm for it (which is what we use for the CentOS package) but hit some issues with it: jordansissel/fpm#1227

[chorrell]

Closing since we a) update yarn pretty regularly now and b) we have docs for how to update: https://github.com/nodejs/docker-node/blob/master/docs/BestPractices.md#upgradingdowngrading-yarn

[mikemaccana]

Having docs to work around unpackaged yarn seems like too much effort to work around not installing yarn properly so 'apt-get update' works.

…

On Fri, 25 May 2018 at 2:20 pm, Christopher Horrell < ***@***.***> wrote: Closing since we a) update yarn pretty regularly now and b) we have docs for how to update: https://github.com/nodejs/docker-node/blob/master/docs/BestPractices.md#upgradingdowngrading-yarn — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#524 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAKiMr4gwVRwaRySd1Si8LxuNZHnNjpbks5t2AUIgaJpZM4PVJHW> .

[chorrell]

Sure, but until we're able to securely install Yarn (and ideally Node.js) via package managers for both Debian and alpine, this is what we're stuck with.

[mikemaccana]

# Upgrading yarn ## Debian Just use apt-get update; apt get upgrade yarn ## Alpine (Longer instructions)

…

On Fri, 25 May 2018 at 2:41 pm, Christopher Horrell < ***@***.***> wrote: Sure, but until we're able to securely install Yarn (and ideally Node.js) via package managers for both Debian and alpine, this is what we're stuck with. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#524 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAKiMpMNwR9JC7HCYpfyFcgkVy8hMpVqks5t2AoEgaJpZM4PVJHW> .