nodePackages: Switch to 12.x and 14.x, continue #85764

[calbrecht]

Motivation for this change

To continue with PR #85764 and implement suggested changes on top of that.

To be consistent with future updates of Node.js, i propose (and implement within this PR) on top of the suggested changes in mentionend PR, that we stick to the official released Node.js versions LTS and Current, now at 12.x and 14.x.

Let us reflect that in the change of the nodePackages attribute in the top-level/all-packages.nix package set by renaming the version suffixed package set to nodePackages_latest for the latest official Current release, and nodePackages for the latest official LTS release.

Things done

To ensure all node-packages build successfully, i wrote a shell.nix for testing.

shell.nix in nixpkgs before PR (hacky, but you get the point)

{ pkgs ? import ./default.nix {} }:

let
  node10Pkgs = (import ./pkgs/development/node-packages/default-v10.nix {
    inherit pkgs;
    inherit (pkgs) stdenv;
    nodejs = pkgs.nodejs;
  });
  node12Pkgs = (import ./pkgs/development/node-packages/default-v12.nix {
    inherit pkgs;
    inherit (pkgs) stdenv;
    nodejs = pkgs.nodejs-12_x;
  });
  node13Pkgs = (import ./pkgs/development/node-packages/default-v13.nix {
    inherit pkgs;
    inherit (pkgs) stdenv;
    nodejs = pkgs.nodejs-13_x;
  });
in
pkgs.mkShell {

  buildInputs = with pkgs.lib;
  (attrValues (
    filterAttrs (name: _: 1 > length (
      intersectLists [ name ] [
        # broken packages 10_x
        "coc-imselect"
        "fast-cli-1.x"
        "git-ssb"
        "pulp"
        "insect"
        "node-red"
        "node-inspector"
        "stf"
        "ssb-server"
        # not applicable 10_x
        "ios-deploy"
      ])) node10Pkgs
  )) ++
  (attrValues (
    filterAttrs (name: _: 1 > length (
      intersectLists [ name ] [
        # broken packages 12_x
        # not applicable 12_x
      ])) node12Pkgs
  )) ++
  (attrValues (
    filterAttrs (name: _: 1 > length (
      intersectLists [ name ] [
        # broken packages 13_x
        # not applicable 13_x
      ])) node13Pkgs
  ));

  shellHook = ''
    ${pkgs.lib.concatMapStrings (name: ''
      echo -n 10: ${name}:\ ; echo -en $(which ${name} 2>/dev/null)"\n" ;
    '') (pkgs.lib.attrNames node10Pkgs) }
    ${pkgs.lib.concatMapStrings (name: ''
      echo -n 12: ${name}:\ ; echo -en $(which ${name} 2>/dev/null)"\n" ;
    '') (pkgs.lib.attrNames node12Pkgs) }
    ${pkgs.lib.concatMapStrings (name: ''
      echo -n 13: ${name}:\ ; echo -en $(which ${name} 2>/dev/null)"\n" ;
    '') (pkgs.lib.attrNames node13Pkgs) }
  '';
}

With the help of this shell.nix i noticed there are some packages already broken in the old node-packages. Namely coc-imselect fast-cli-1.x git-ssb pulp insect node-red node-inspector stf ssb-server and one package that claims to be incompatible with linux, namely ios-deploy.

So i thought about what to do if node-packages break with new releases and in different versions, and i came to the conclusion the easiest way is to mark them as broken in the meta attribute depending on the nodejs.version in use. Similar method is applicable to platform support with the meta.platforms attribute. So we are able to write in node-packages/default.nix e.g.

let
  since = (version: pkgs.lib.versionAtLeast nodejs.version version);
in nodePackages // {
  # ....
  "fast-cli-1.x" = nodePackages."fast-cli-1.x".override {
    meta.broken = since "10";
  };
}

To ensure that is working i came up with this shell.nix

shell.nix in nixpkgs after PR

{ pkgs ? import ./default.nix {} }:

let
  isBroken = pkg: pkgs.lib.hasAttrByPath ["meta" "broken"] pkg && pkg.meta.broken;
  isAvailable = pkg: pkgs.lib.hasAttrByPath ["meta" "available"] pkg && pkg.meta.available;
  toInputs = nodePkgs: pkgs.lib.mapAttrsToList (_: pkg: pkg) nodePkgs;
  filterPkgs = nodePkgs: pkgs.lib.filterAttrs (_: pkg: ! isBroken pkg && isAvailable pkg) nodePkgs;
  toFilteredInputs = nodePkgs: toInputs (filterPkgs nodePkgs);

  node12Pkgs = pkgs.lib.filterAttrs (_: v: pkgs.lib.isDerivation v) pkgs.nodePackages;
  node14Pkgs = pkgs.lib.filterAttrs (_: v: pkgs.lib.isDerivation v) pkgs.nodePackages_latest;
in
pkgs.mkShell {

  buildInputs = (toFilteredInputs node12Pkgs) ++ (toFilteredInputs node14Pkgs);

  shellHook = ''
    ${pkgs.lib.concatMapStrings (name: ''
      echo -n 12: ${name}:\ ; echo -en $(which ${name} 2>/dev/null)"\n" ;
    '') (pkgs.lib.attrNames node12Pkgs) }
    ${pkgs.lib.concatMapStrings (name: ''
      echo -n 14: ${name}:\ ; echo -en $(which ${name} 2>/dev/null)"\n" ;
    '') (pkgs.lib.attrNames node14Pkgs) }
  '';
}

🎉

• Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
• Built on platform(s)
  • NixOS
  • macOS
  • other Linux distributions
• Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
• Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
• Tested execution of all binary files (usually in ./result/bin/)
• Determined the impact on package closure size (by running nix path-info -S before and after)
• Ensured that relevant documentation is up to date
• Fits CONTRIBUTING.md.

cc @Mic92 @marsam @prusnak

The broken node-packages need to be fixed, but i do not consider the fix being part of this PR.

[Mic92]

Looks good to me, if those missing packages gets re-added.

[calbrecht]

Opened #89205 to track/remove/fix broken packages.

[prusnak]

Love it! 👍

Since we are regenerating node-packages.nix, can we also change hashes from sha1 to sha256? This would fix most of the problems from #77238

[Mic92]

Those are coming from npm and are not computed locally. Hence we have little control over that. However they might be an indicator for old packages, which we might want to remove.

[calbrecht]

@prusnak, yes @Mic92 is right, that would be something that has to be tackled in node2nix with, as there is already some abstraction over generating/converting sha sums.

Also i did the following to check the status:

tutnix ~/ws/nixpkgs-upstream (git) 6149df8fd095 U continue-pull-85764
% rg -o -c 'sha1' pkgs/development/node-packages/node-packages.nix
2342
tutnix ~/ws/nixpkgs-upstream (git) 6149df8fd095 U continue-pull-85764
% rg -o -c 'sha256' pkgs/development/node-packages/node-packages.nix
18
tutnix ~/ws/nixpkgs-upstream (git) 6149df8fd095 U continue-pull-85764
% rg -o -c 'sha512' pkgs/development/node-packages/node-packages.nix
3399

[Mic92]

Also the node ecosystem has bigger problem than sha1 collisions, i.e. malware.

[calbrecht]

Also the node ecosystem has bigger problem than sha1 collisions, i.e. malware.

Indeed, but i guess that holds true for any language specific package repository e.g. python

[Mic92]

True also there we have more control/supervision since we handcraft all python modules.

[calbrecht]

Thank you @prusnak for taking care of that. I totally missed that.