Add yarn-berry3 and yarn-berry4 to fetch and use version 2 yarn.lock files

[gador]

Problem

Currently there is no good way to use newer yarn.lock files because they rely on yarn-berry and cannot be used together with pre-existing tooling like fetch-yarn-deps.
There is an (abandoned? ) project which converts v1 files to v2 files with some caveats. This does require an internet connection and cannot be used in an isolated builder.

So one needs to convert the yarn.lock file and commit it to nixpkgs to be used for an offline cache.

There is an issue open (#254369) which discusses some ways around it. The most sophisticated way was by @szlend and this PR builds on top of it to extend to be included in our new yarnConfigHook from #318015

Implementation of a solution

A new argument has been added to fetch-yarn-deps: yarnVersion which will default to 1 which uses the original yarn.

For newer yarn.lock files, one can set the yarnVersion to either 3 or 4 which will use yarn-berry3 or yarn-berry4 respectively. We need to separate both versions, because they have some small changes in the way the lock file is created.

This also adds the corresponding tests.

How to use it

For an implementation see the included pgadmin4 derivation, which was also my main motivation for this. I had to build a rather ugly and complicated update script before and do a lot of custom work to build the frontend. This is now simplified.

In short, add yarnBerry3ConfigHook to nativeBuildInputs and

yarnOfflineCache = fetchYarnDeps {
    src = "...";
    yarnVersion = 3;
    hash = "..."
  };

to your derivation.

fixes #254369

Once we agree on the implantation, I'll add the documentation.

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
  • sandbox = relaxed
  • sandbox = true
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.
• Add documentation

---

Add a 👍 reaction to pull requests you find important.

[doronbehar]

Note also the CI failure.

[gador]

Note also the CI failure.

This is caused by not adding the yarn-berry3 and yarn-berry4 to pkgs/by-name which doesn't make sense here

Attribute pkgs.yarn-berry3 is a new top-level package using pkgs.callPackage ./pkgs/development/tools/yarn-berry { /* ... */ }.
Please define it in pkgs/by-name/ya/yarn-berry3/package.nix instead.
See pkgs/by-name/README.md for more details.
Since the second callPackage argument is not { }, the manual callPackage in pkgs/top-level/all-packages.nix is still needed.

• Attribute pkgs.yarn-berry4 is a new top-level package using pkgs.callPackage ./pkgs/development/tools/yarn-berry { /* ... */ }.
  Please define it in pkgs/by-name/ya/yarn-berry4/package.nix instead.
  See pkgs/by-name/README.md for more details.
  Since the second callPackage argument is { }, no manual callPackage in pkgs/top-level/all-packages.nix is needed anymore.

This PR introduces additional instances of discouraged patterns as listed above. Merging is discouraged but would not break the base branch.

[doronbehar]

Note also the CI failure.

This is caused by not adding the yarn-berry3 and yarn-berry4 to pkgs/by-name which doesn't make sense here

Hmm seems like a false negative to me. Let's ask the advice of @NixOS/nixpkgs-vet .

[willbush]

Think this recommendation applies here: https://github.com/NixOS/nixpkgs/tree/master/pkgs/by-name#recommendation-for-new-packages-with-multiple-versions

[gador]

Think this recommendation applies here: https://github.com/NixOS/nixpkgs/tree/master/pkgs/by-name#recommendation-for-new-packages-with-multiple-versions

Awesome, thanks for the pointer!

[doronbehar]

Looks pretty good, though I wouldn't feel comfortable to merge this because I don't know the yarn config settings in yarnBerryConfigHook but they looks pretty reasonable.

[winterqt]

This is a no-go in its current state.

[winterqt]

This will break the FODs almost instantly if Yarn changes anything in how they pull deps, and also is probably not reproducible between platforms when platform-specific dependencies are used.

[gador]

Valid point. This is partly why there are two yarn versions.

Also this could be said for all external package managers, not just yarn-berry.

I'm uncertain about your point about platform specific dependencies. I'll have to look into it.

[szlend]

Platform specific dependencies should be made reproducible by explicitly setting supportedArchitectures. Your point about Yarn breaking how they fetch deps may be valid depending on the upstream stability guarantees. Explicitly picking the version of yarn somewhat mitigates that issue, but it's no guarantee yes. This is the same path taken in pnpm.fetchDeps #290715

[doronbehar]

@winterqt what is your reply on the arguments laid out above? Your "request for changes" is blocking the merge of the PR, and this is the only review comment left really open.

[gador]

@emilazy I know this is a controversial PR, but I'd love to here your thoughts on #355053 (comment)
thanks :-)

[gador]

@emilazy I see that there is no activity from you since end of November. I'd still love to get your feedback to my reply above. In the meantime: @SuperSandro2000 could you also have a look at this? I'd like to get this merged and I believe I addressed most of the issues concerning stability and reliability in my comment above

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/how-to-implement-version-2-yarn-lock-package-management/59792/1

[yuyuyureka]

Sadly, I have to agree with winterqt.
This kind of fetcher is not sustainable and just defers the cost of an actual pure fetcher to the future.
The reason is ultimately that using FODs like this is always a hack, and the only valid use of them is to reduce evaluation cost and avoid Import-From-Derivation for a problem that we could theoretically also solve by generating many small derivations which would match the hashes specified in the lockfile.
If we don't really understand what yarn is doing under the hood to arrive at this hash, it will explode in our faces sooner or later.

[yuyuyureka]

I built a prototype to convert the npm registry tar files to zipfiles with hashes matching the ones in the lockfile: https://cyberchaos.dev/yuka/yarn-zip

I still need to perform larger scale testing, but it generally seems to produce identical outputs to what is present in my yarn berry cache.

Thanks to flokli for partially funding this work.

[flokli]

#399404 has landed, this can be closed.