[Bug?]: supported architectures behaviour doesn't support zero installs

[laytong]

Self-service

• I'd be willing to implement a fix

Describe the bug

I have a repo using zero installs approach that requires one unplugged package (@swc-core)

its unplugged as it appears to require platform specific binaries. My CI environment is a linux env, whereas local is mac

I added the specific platforms to the supportedArchitectures field in my .yarnrc.yml

I had hoped/expected that with zero-installs the platform specific binaries would have been installed/added to the repo on local install so that when i run my ci i do not need to do a yarn install step (in keeping with the zero installs approach)

This may be the intended behaviour, but then how do I maintain zero installs in this case?

To reproduce

I don't believe this can be reproduced in sherlock due to the cross platform nature of the issue

Environment

System:
    OS: macOS 11.6.5
    CPU: (12) x64 Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
  Binaries:
    Node: 16.15.1 - /private/var/folders/bl/q3538n111qsck6ll41877f440000gp/T/xfs-d4f8e23c/node
    Yarn: 3.2.1 - /private/var/folders/bl/q3538n111qsck6ll41877f440000gp/T/xfs-d4f8e23c/yarn
    npm: 8.11.0 - ~/.nvm/versions/node/v16.15.1/bin/npm

Additional context

Here's the ci error if i do not include yarn install step

Yarnrc file

plugins:
  - path: .yarn/plugins/@yarnpkg/plugin-interactive-tools.cjs
    spec: "@yarnpkg/plugin-interactive-tools"

yarnPath: .yarn/releases/yarn-3.2.1.cjs

supportedArchitectures:
  os:
    - darwin
    - linux

[seivan]

Do you know if SWC supplies prebuilt binaries for each platform?

For instance, ESBuild does that, running install locally, gives me all the binaries (arm, x64, darwin, linux) etc.
They are part of the repo.

Unless the package supplies prebuilt binaries or cross-build install script, there really isn't much you can do.

[laytong]

So when using my local (Mac) env it installs the darwin specific prebuilt binary (which is at least better than all binaries) to the unplugged folder. Ideally when using supportedArchitectures it would grab all binaries that meet my specified platforms?

…

On Sat., Jul. 9, 2022, 10:24 Seivan, ***@***.***> wrote: Do you know if SWC supplies prebuilt binaries for each platform? For instance, ESBuild does that, running install locally, gives me all the binaries (arm, x64, darwin, linux) etc. They are part of the repo. Unless the package supplies prebuilt binaries or cross-build install script, there really isn't much you can do. — Reply to this email directly, view it on GitHub <#4615 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACJVU776YC723DQSOFAH3NDVTGYTBANCNFSM53BXCICQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[laytong]

https://swc.rs/docs/getting-started#supported-binaries On Sat., Jul. 9, 2022, 10:34 Layton Gilbraith, ***@***.***> wrote:

…

So when using my local (Mac) env it installs the darwin specific prebuilt binary (which is at least better than all binaries) to the unplugged folder. Ideally when using supportedArchitectures it would grab all binaries that meet my specified platforms? On Sat., Jul. 9, 2022, 10:24 Seivan, ***@***.***> wrote: > Do you know if SWC supplies prebuilt binaries for each platform? > > For instance, ESBuild does that, running install locally, gives me all > the binaries (arm, x64, darwin, linux) etc. > They are part of the repo. > > Unless the package supplies prebuilt binaries or cross-build install > script, there really isn't much you can do. > > — > Reply to this email directly, view it on GitHub > <#4615 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/ACJVU776YC723DQSOFAH3NDVTGYTBANCNFSM53BXCICQ> > . > You are receiving this because you authored the thread.Message ID: > ***@***.***> >

[seivan]

@laytong That's what ESBuild does for me. It's part of it's post-install script (I am guessing).
They might have the binaries, but do they actually fetch them on install?

[laytong]

For me only the darwin x64 binary is installed on yarn install

[laytong]

swc-project/swc#2898

I guess this is actually an issue with SWC rather than yarn?

[seivan]

Yep, there is stuff Yarn could do to make i easier, but yeah. Sorry.

[laytong]

Ok seems it does work,

But they need all supportedArchitectures critieria

needed to specify both cpu and libc sub options for @swc-core-linux-x64-gnu-npm to be installed

[laytong]

Can you point me to some guidance on when/why a package needs to be unplugged

i.e. in this instance could the package be modified in some way that it can link to these built binaries without needing to be unplugged?

[seivan]

As far as I understand it, something is unplugged if it has characteristics of not being something that can be shared.
Like architecture or the fact that it needs to trigger a script (post-install, etc) gives some sort of hint that this might be very unique to your installations, thus not portable.

Now for instance, CoreJS is unplugged, and it's unplugged because it has a post-install script, that you can disable in .yarnrc.yml.

[pegiadise]

Ok seems it does work,

But they need all supportedArchitectures critieria

needed to specify both cpu and libc sub options for @swc-core-linux-x64-gnu-npm to be installed

Even with yarn@4, This continues to be the case for supportedArchitectures to behave correctly (all 3 options need to be specified, in order to install binaries for all of them). Maybe this will save someone some time. Don't know if bug or feature at the time of writing.

Example working config:

supportedArchitectures:
  os:
    - current
    - darwin
    - linux
  cpu:
    - current
    - x64
    - arm64
  libc:
    - current
    - glibc
    - musl

[yuriy-yarosh]

@laytong this is still broken on yarn@4
It's still too puzzling why esbuild / swc doesn't get resolved properly from time to time.
@pegiadise yup, it still persists and it's a bug.

Downgrading to 3.7.0 helps atm.
yarn set version 3.7.0 fixes things...

[laytong]

Thanks for the update.

I've moved away from using zero installs mostly due to issues like this, so I no longer check in my yarn cache

[marceljay]

The whole thing is flaky as hell. I am getting install issues on the Github CI ubuntu runners depending on how many zip files were cached. It's completely unpredictable.

[bazo]

is it possible to restrict supported archs only to given combination of os and cpu? like:
darwin-arm64 only and linux-amd64 only?

[marceljay]

is it possible to restrict supported archs only to given combination of os and cpu? like: darwin-arm64 only and linux-amd64 only?

I don't think it is, but that would be very helpful

[clemyan]

@pegiadise

Even with yarn@4, This continues to be the case for supportedArchitectures to behave correctly (all 3 options need to be specified, in order to install binaries for all of them). Maybe this will save someone some time. Don't know if bug or feature at the time of writing.

This is the expected behavior.

Some packages specify the OS, CPU and/or the C library needed for the package to work (e.g. @swc/core-linux-x64-gnu specifies that it works on a Linux machine with glibc on an x64 CPU). The default (without specifying supportedArchitectures) is to only install a package if it works on your current machine setup. supportedArchitectures is for changing that check to use machine setups other than or in addition to your current machine setup.

Say if you are installing @swc/core on a Windows machine, you probably have no C library. So @swc/core-linux-x64-gnu (an optional dependency of @swc/core) will not be installed, because it only works on a Linux machine with glibc, not a Windows machine with no C library.

If you only specify

supportedArchitectures:
  os:
    - current
    - linux

then @swc/core-linux-x64-gnu still won't be installed, because while you say packages for Linux machines should be installed, you still have no C library, and @swc/core-linux-x64-gnu needs glibc to work.

In order for @swc/core-linux-x64-gnu to be installed, you need to configure Yarn to also install packages that need glibc

supportedArchitectures:
  os:
    - current
    - linux
  libc:
    - current
    - glibc

---

@bazo

Have you tried

supportedArchitectures:
  os:
    - darwin
    - linux
  cpu:
    - arm64