fetchYarnBerryDeps: init

doc/languages-frameworks/javascript.section.md

@@ -552,7 +552,15 @@ In this example, `prePnpmInstall` will be run by both `pnpm.configHook` and by t
 
 ### Yarn {#javascript-yarn}
 
-Yarn based projects use a `yarn.lock` file instead of a `package-lock.json` to pin dependencies. Nixpkgs provides the Nix function `fetchYarnDeps` which fetches an offline cache suitable for running `yarn install` before building the project. In addition, Nixpkgs provides the hooks:
+Yarn based projects use a `yarn.lock` file instead of a `package-lock.json` to pin dependencies.
+
+To package yarn-based applications, you need to distinguish by the version pointers in the `yarn.lock` file. See the following sections.
+
+#### Yarn v1 {#javascript-yarn-v1}
+
+Yarn v1 lockfiles contain a comment `# yarn lockfile v1` at the beginning of the file.
+
+Nixpkgs provides the Nix function `fetchYarnDeps` which fetches an offline cache suitable for running `yarn install` before building the project. In addition, Nixpkgs provides the hooks:
 
 - `yarnConfigHook`: Fetches the dependencies from the offline cache and installs them into `node_modules`.
 - `yarnBuildHook`: Runs `yarn build` or a specified `yarn` command that builds the project.
@@ -602,28 +610,28 @@ stdenv.mkDerivation (finalAttrs: {
 })
 ```
 
-#### `yarnConfigHook` arguments {#javascript-yarnconfighook}
+##### `yarnConfigHook` arguments {#javascript-yarnconfighook}
 
 By default, `yarnConfigHook` relies upon the attribute `${yarnOfflineCache}` (or `${offlineCache}` if the former is not set) to find the location of the offline cache produced by `fetchYarnDeps`. To disable this phase, you can set `dontYarnInstallDeps = true` or override the `configurePhase`.
 
-#### `yarnBuildHook` arguments {#javascript-yarnbuildhook}
+##### `yarnBuildHook` arguments {#javascript-yarnbuildhook}
 
 This script by default runs `yarn --offline build`, and it relies upon the project's dependencies installed at `node_modules`. Below is a list of additional `mkDerivation` arguments read by this hook:
 
 - `yarnBuildScript`: Sets a different `yarn --offline` subcommand (defaults to `build`).
 - `yarnBuildFlags`: Single string list of additional flags to pass the above command, or a Nix list of such additional flags.
 
-#### `yarnInstallHook` arguments {#javascript-yarninstallhook}
+##### `yarnInstallHook` arguments {#javascript-yarninstallhook}
 
 To install the package `yarnInstallHook` uses both `npm` and `yarn` to cleanup project files and dependencies. To disable this phase, you can set `dontYarnInstall = true` or override the `installPhase`. Below is a list of additional `mkDerivation` arguments read by this hook:
 
 - `yarnKeepDevDeps`: Disables the removal of devDependencies from `node_modules` before installation.
 
-### yarn2nix {#javascript-yarn2nix}
+#### yarn2nix {#javascript-yarn2nix}
 
-WARNING: The `yarn2nix` functions have been deprecated in favor of the new `yarnConfigHook`, `yarnBuildHook` and `yarnInstallHook`. Documentation for them still appears here for the sake of the packages that still use them. See also a tracking issue [#324246](https://github.com/NixOS/nixpkgs/issues/324246).
+WARNING: The `yarn2nix` functions have been deprecated in favor of `yarnConfigHook`, `yarnBuildHook` and `yarnInstallHook` (for Yarn v1) and `yarn-berry_*.*` tooling (Yarn v3 and v4). Documentation for `yarn2nix` functions still appears here for the sake of the packages that still use them. See also a tracking issue [#324246](https://github.com/NixOS/nixpkgs/issues/324246).
 
-#### Preparation {#javascript-yarn2nix-preparation}
+##### Preparation {#javascript-yarn2nix-preparation}
 
 You will need at least a `yarn.lock` file. If upstream does not have one you need to generate it and reference it in your package definition.
 
@@ -638,7 +646,7 @@ If the downloaded files contain the `package.json` and `yarn.lock` files they ca
 }
 ```
 
-#### mkYarnPackage {#javascript-yarn2nix-mkYarnPackage}
+##### mkYarnPackage {#javascript-yarn2nix-mkYarnPackage}
 
 `mkYarnPackage` will by default try to generate a binary. For package only generating static assets (Svelte, Vue, React, WebPack, ...), you will need to explicitly override the build step with your instructions.
 
@@ -689,7 +697,7 @@ or if you need a writeable node_modules directory:
 }
 ```
 
-#### mkYarnModules {#javascript-yarn2nix-mkYarnModules}
+##### mkYarnModules {#javascript-yarn2nix-mkYarnModules}
 
 This will generate a derivation including the `node_modules` directory.
 If you have to build a derivation for an integrated web framework (rails, phoenix..), this is probably the easiest way.
@@ -724,7 +732,7 @@ mkYarnPackage rec {
 }
 ```
 
-#### Pitfalls {#javascript-yarn2nix-pitfalls}
+##### Pitfalls {#javascript-yarn2nix-pitfalls}
 
 - If version is missing from upstream package.json, yarn will silently install nothing. In that case, you will need to override package.json as shown in the [package.json section](#javascript-upstream-package-json)
 - Having trouble with `node-gyp`? Try adding these lines to the `yarnPreBuild` steps:
@@ -744,6 +752,116 @@ mkYarnPackage rec {
   - Exporting the headers in `npm_config_nodedir` comes from this issue: <https://github.com/nodejs/node-gyp/issues/1191#issuecomment-301243919>
 - `offlineCache` (described [above](#javascript-yarn2nix-preparation)) must be specified to avoid [Import From Derivation](#ssec-import-from-derivation) (IFD) when used inside Nixpkgs.
 
+#### Yarn Berry v3/v4 {#javascript-yarn-v3-v4}
+Yarn Berry (v3 / v4) have similar formats, they start with blocks like these:
+
+```yaml
+__metadata:
+  version: 6
+  cacheKey: 8[cX]
+```
+
+```yaml
+__metadata:
+  version: 8
+  cacheKey: 10[cX]
+```
+
+For these packages, we have some helpers exposed under the respective `yarn-berry_3` and `yarn-berry_4` packages:
+
+- `yarn-berry-fetcher`
+- `fetchYarnBerryDeps`
+- `yarnBerryConfigHook`
+
+It's recommended to ensure you're explicitly pinning the major version used, for example by capturing the `yarn-berry_Xn` argument and then re-defining it as a `yarn-berry` `let` binding.
+
+```nix
+{
+  stdenv,
+  nodejs,
+  yarn-berry_4,
+}:
+
+let
+  yarn-berry = yarn-berry_4;
+in
+
+stdenv.mkDerivation (finalAttrs: {
+  pname = "foo";
+  version = "0-unstable-1980-01-01";
+
+  src = {
+    #...
+  };
+
+  nativeBuildInputs = [
+    nodejs
+    yarn-berry.yarnBerryConfigHook
+  ];
+
+  offlineCache = yarn-berry.fetchYarnBerryDeps {
+    inherit (finalAttrs) src;
+    hash = "...";
+  };
+})
+```
+
+##### `yarn-berry_X.fetchYarnBerryDeps` {#javascript-fetchYarnBerryDeps}
+`fetchYarnBerryDeps` runs `yarn-berry-fetcher fetch` in a fixed-output-derivation. It is a custom fetcher designed to reproducibly download all files in the `yarn.lock` file, validating their hashes in the process. For git dependencies, it creates a checkout at `${offlineCache}/checkouts/<40-character-commit-hash>` (relying on the git commit hash to describe the contents of the checkout).
+
+To produce the `hash` argument for `fetchYarnBerryDeps` function call, the `yarn-berry-fetcher prefetch` command can be used:
+
+```console
+$ yarn-berry-fetcher prefetch </path/to/yarn.lock> [/path/to/missing-hashes.json]
+```
+
+This prints the hash to stdout and can be used in update scripts to recalculate the hash for a new version of `yarn.lock`.
+
+##### `yarn-berry_X.yarnBerryConfigHook` {#javascript-yarnBerryConfigHook}
+`yarnBerryConfigHook` uses the store path `offlineCache` points to, to run a `yarn install` during the build, producing a usable `node_modules` directory from the downloaded dependencies.
+
+Internally, this uses a patched version of Yarn to ensure git dependencies are re-packed and any attempted downloads fail immediately.
+
+##### Patching upstream `package.json` or `yarn.lock` files {#javascript-yarnBerry-patching}
+In case patching the upstream `package.json` or `yarn.lock` is needed, it's important to pass `finalAttrs.patches` to `fetchYarnBerryDeps` as well, so the patched variants are picked up (i.e. `inherit (finalAttrs) patches`.
+
+##### Missing hashes in the `yarn.lock` file {#javascript-yarnBerry-missing-hashes}
+Unfortunately, `yarn.lock` files do not include hashes for optional/platform-specific dependencies. This is [by design](https://github.com/yarnpkg/berry/issues/6759).
+
+To compensate for this, the `yarn-berry-fetcher missing-hashes` subcommand can be used to produce all missing hashes. These are usually stored in a `missing-hashes.json` file, which needs to be passed to both the build itself, as well as the `fetchYarnBerryDeps` helper:
+
+```nix
+{
+  stdenv,
+  nodejs,
+  yarn-berry_4,
+}:
+
+let
+  yarn-berry = yarn-berry_4;
+in
+
+stdenv.mkDerivation (finalAttrs: {
+  pname = "foo";
+  version = "0-unstable-1980-01-01";
+
+  src = {
+    #...
+  };
+
+  nativeBuildInputs = [
+    nodejs
+    yarn-berry.yarnBerryConfigHook
+  ];
+
+  missingHashes = ./missing-hashes.json;
+  offlineCache = yarn-berry.fetchYarnBerryDeps {
+    inherit (finalAttrs) src missingHashes;
+    hash = "...";
+  };
+})
+```
+
 ## Outside Nixpkgs {#javascript-outside-nixpkgs}
 
 There are some other tools available, which are written in the Nix language.

doc/redirects.json

@@ -3211,6 +3211,12 @@
   "javascript-yarn": [
     "index.html#javascript-yarn"
   ],
+  "javascript-yarn-v1": [
+    "index.html#javascript-yarn-v1"
+  ],
+  "javascript-yarn-v3-v4": [
+    "index.html#javascript-yarn-v3-v4"
+  ],
   "javascript-yarnconfighook": [
     "index.html#javascript-yarnconfighook"
   ],
@@ -3238,6 +3244,18 @@
   "javascript-yarn2nix-pitfalls": [
     "index.html#javascript-yarn2nix-pitfalls"
   ],
+  "javascript-yarnBerry-missing-hashes": [
+    "index.html#javascript-yarnBerry-missing-hashes"
+  ],
+  "javascript-yarnBerryConfigHook": [
+    "index.html#javascript-yarnBerryConfigHook"
+  ],
+  "javascript-yarnBerry-patching": [
+    "index.html#javascript-yarnBerry-patching"
+  ],
+  "javascript-fetchYarnBerryDeps": [
+    "index.html#javascript-fetchYarnBerryDeps"
+  ],
   "javascript-outside-nixpkgs": [
     "index.html#javascript-outside-nixpkgs"
   ],

pkgs/by-name/he/hedgedoc/missing-hashes.json

@@ -0,0 +1,27 @@
+{
+  "@esbuild/aix-ppc64@npm:0.25.2": "3b5ee5599a8446074bd6aad732c5f2833a4b77e8af62cfcdee7508ded661daa054c481c2fa69f5341e65cc8846a2b3f026ffca12934cb24d76df93e4800e2979",
+  "@esbuild/android-arm64@npm:0.25.2": "556d958ea6f33073669a8a41645b0e51cecb2c0788ece8827a8752e666fd178ae28b2f9749b8a968f1e9c66b09dab7933be8d8b53e391ea9eca7bddf3f53b26a",
+  "@esbuild/android-arm@npm:0.25.2": "0ce9f260216520a4d53c2736b60e8e55b8c6569b944555cb7840fbe2ff16278d4e6591aedcbc38b3f69a6d98167367f36bba1d35b41d725e4d68a67f942e2a28",
+  "@esbuild/android-x64@npm:0.25.2": "57d0c438a3bcc25db5aa89c9d9b60827d3ee6d9553234d95a16f52f8a7780778e5e7a9975374e4d37a81e600d2b8f32bd46d5633bb83bcd958ee2f5162d1359b",
+  "@esbuild/darwin-arm64@npm:0.25.2": "e148927428c0c5d69e681ee8d4b47e0bcc5116296c47c41ed44068d686a0568b23dc493dc47b4d103e7b21894fafbaf5c2d64bf2420a5d6b6b22f71bf6c2fb08",
+  "@esbuild/darwin-x64@npm:0.25.2": "2bc37a902d7828f3973d28f486bcf26da13f6f421122348fc510551a4674bc706eb8e6f692ccd3d887523c373580b3228797f2fd73d641b2e106a6d44e3f20c7",
+  "@esbuild/freebsd-arm64@npm:0.25.2": "431ddf98e7c0b7c6a7d1a689454ab449df8eaacca242442153a4f149017d6bd03e16b7fda01e9fe26316d135176d74f5de09e9a21357648c320a3211a1c862a2",
+  "@esbuild/freebsd-x64@npm:0.25.2": "21545ec11969db7ed091819b63c88a9c4490edce8a98d64718f79352a5001927d1d85e6261170562850afdb5a7a2182ccf3d936eab0db615e51188ab6f95789b",
+  "@esbuild/linux-arm64@npm:0.25.2": "332ba0533f2a2bba21dfb3c9130f9a1cef5b420da29f67116e7fbff4cfd12039d06646b4a636952b873b22129a4e5c6ea6ea9b57286949980add7bab5d6684c1",
+  "@esbuild/linux-arm@npm:0.25.2": "a89613faa8ce9f307eea4c2b4ada8cdb56e378716d5250fc163674032bfbdda8b8aacc730c6b5fa05b5e5bff04a9a11dd02934d69b2233ecf0ff58af4ef5bb45",
+  "@esbuild/linux-ia32@npm:0.25.2": "5907824945c067f092bb28807080a2617b5208583ef71b2091d57b6039df182735ba62ee0142e993260d80df3a5b7ae88c0fff28f2e4efd8254dd34de2f4ad95",
+  "@esbuild/linux-loong64@npm:0.25.2": "ed521bfa81db13fc628455fc0feec4e75150ed2248013baaf8fa7d6d278295d3a2b09196c79169b7331e50e6d3abf86ee5773ad28f8d5914bbe4034758a424c2",
+  "@esbuild/linux-mips64el@npm:0.25.2": "9961d5853e7ff048ae2cb3997136f96ca5180dcc483e88e2ae0d34ca8ec097af3cc7d7c8ac6303a3cf2582eadbe905fa97a38dd9b402f82b9e104a0070f04b79",
+  "@esbuild/linux-ppc64@npm:0.25.2": "9ee2360fa976e7c0e16b09fd551fba232e19ede65a326544d16bcae0c399e2937b853eab5649a52589e483c77dbc4ab5e082177ee70e6fd2de53c421314ea458",
+  "@esbuild/linux-riscv64@npm:0.25.2": "e6bff41e76d44a8d6f27a53b2aeb98c771f6215590d411a1b6b682aa41c96cc9ed8baa6d17412d09419b10778dc40ca75b6cfe71dc8e24f1bf5c1e7793e2c560",
+  "@esbuild/linux-s390x@npm:0.25.2": "34ca97c2506f1a1e646bc74f7cfff7bc7990052c3a2ade963c010c6bff6ae6135b9788f7972af2fb47e1a5380b26b8ca1b7a6c3f2bc9eaa7c9e7fdf227e38ab0",
+  "@esbuild/linux-x64@npm:0.25.2": "ac2c60f9dcbf9c55f66aef618b2a3218ddde92aad764aa6645b94b50c8c6242f1f97a0af956759738bca97686918dd71a0cfc82c73eadb0ab921918c9927a627",
+  "@esbuild/netbsd-arm64@npm:0.25.2": "6d820e86cbf10d305520ed7dbf4358bd4d75d333de39c100156dcc5619446916b4d5a1c1e07b838d22f6013aaf57bdd93187f2e97b1357a426e94e91b485e0e1",
+  "@esbuild/netbsd-x64@npm:0.25.2": "89153d1753254cd88f12b9e088a79398dcee11c0a1349a0886b54dd2db0ae2faf64ac0e364494f0cbd64163320cff94f02fc1fe827ff950daa0a27e50ef84a27",
+  "@esbuild/openbsd-arm64@npm:0.25.2": "71ede4e8f1086bece4e6d528b46ee3a493e0d33d749d5ac9050afd5471465badce7f36f1d04c1edddec75fde49e1229adb0f7758bcd508219a7598086b9a3c5c",
+  "@esbuild/openbsd-x64@npm:0.25.2": "95d90426c96d27980340d29c6d83f79ea2a22008041a5366c81820275ce36ba95ad0bf5e89ee199747a5921953587e7dcdc2e5a97368dec4efbcfc83ab5be4ee",
+  "@esbuild/sunos-x64@npm:0.25.2": "da59382e27897f98ee7261651eff1bcf42b1d659b078e500f664ef18137d70298f0bc0a50cf5af2311931e5b0454c6d3756a0ff9e560ab0a754ea56b0f04768c",
+  "@esbuild/win32-arm64@npm:0.25.2": "39f9a30d00a788ef168d7186f366397f7c471f1db3328efc0b9ff0861475f8a4484723b60ab6ab04c3bb27ebcdc8632e4883c28b5fa93b0606dd9f85f7904043",
+  "@esbuild/win32-ia32@npm:0.25.2": "74a7726d21a347faa1debdcd540e77fa5af9f96d5e0d625f2752d24a1616ac0fdea6582a0499935d9d964ee4c7a57ad8f30c6fe355089af2095bd0e41b70ea49",
+  "@esbuild/win32-x64@npm:0.25.2": "1e3f44cc53acaff9d45e2cded9811c750b7eb946f7b76d1fec66b862a89da800f9291269607d7cbb8d7e6068504571904d93a0813692e74fbae1fc321fe46440"
+}

pkgs/by-name/he/hedgedoc/package.nix

@@ -4,11 +4,11 @@
   fetchFromGitHub,
   gitMinimal,
   cacert,
-  yarn,
   makeBinaryWrapper,
   nodejs,
   python3,
   nixosTests,
+  yarn-berry_4,
   writableTmpDirAsHomeHook,
 }:
 
@@ -21,58 +21,32 @@ let
     tag = version;
     hash = "sha256-hXcPcGj+efvRVt3cHQc9KttE0/DOD9Bul6f3cY4ofgs=";
   };
-
-  # we cannot use fetchYarnDeps because that doesn't support yarn 2/berry lockfiles
-  offlineCache = stdenv.mkDerivation {
-    name = "hedgedoc-${version}-offline-cache";
-    inherit src;
-
-    nativeBuildInputs = [
-      cacert # needed for git
-      gitMinimal # needed to download git dependencies
-      nodejs # needed for npm to download git dependencies
-      yarn
-      writableTmpDirAsHomeHook
-    ];
-
-    buildPhase = ''
-      yarn config set enableTelemetry 0
-      yarn config set cacheFolder $out
-      yarn config set --json supportedArchitectures.os '[ "linux" ]'
-      yarn config set --json supportedArchitectures.cpu '["arm", "arm64", "ia32", "x64"]'
-      yarn
-    '';
-
-    outputHashMode = "recursive";
-    outputHash = "sha256-KTUj1O2AA1qTQOqTbGBPLHAgiG5sG832Na8qLvEccmc=";
-  };
+  missingHashes = ./missing-hashes.json;
 
 in
 stdenv.mkDerivation {
   pname = "hedgedoc";
-  inherit version src;
+  inherit version src missingHashes;
+
+  offlineCache = yarn-berry_4.fetchYarnBerryDeps {
+    inherit src missingHashes;
+    hash = "sha256-V7ptquAohv0t5oA+3iTvlQOZoEtY5xWyhSoJP8jwYI8=";
+  };
 
   nativeBuildInputs = [
     makeBinaryWrapper
     (python3.withPackages (ps: with ps; [ setuptools ])) # required to build sqlite3 bindings
-    yarn
-    writableTmpDirAsHomeHook # A writable home directory is required for yarn
+    yarn-berry_4
+    yarn-berry_4.yarnBerryConfigHook
   ];
 
   buildInputs = [
     nodejs # for shebangs
   ];
 
-  dontConfigure = true;
-
   buildPhase = ''
     runHook preBuild
 
-    yarn config set enableTelemetry 0
-    yarn config set cacheFolder ${offlineCache}
-    export npm_config_nodedir=${nodejs} # prevent node-gyp from downloading headers
-
-    yarn --immutable-cache
     yarn run build
 
     # Delete scripts that are not useful for NixOS
@@ -102,7 +76,6 @@ stdenv.mkDerivation {
   '';
 
   passthru = {
-    inherit offlineCache;
     tests = { inherit (nixosTests) hedgedoc; };
   };