[BUG] gzip archives (from git) have inconsistent checksums across Linux distributions

[ThiefMaster]

What / Why

This is very similar to #62 and possibly npm/cli#2846 but since it's happening reliably on the same machine (ie no architectural differences) in separate docker containers and comments are more likely to be overlooked, I preferred to open a new issue.

I can reliably reproduce this bug using ubuntu and archlinux Docker images using these two Dockerfiles (just for the sake of reproducing it easily) - simply run builds using docker build -f Dockerfile.XXX . and check the different lock files:

How

FROM ubuntu:focal
RUN set -ex && \
        apt update && \
        apt install -y curl git && \
        curl -fsSL https://deb.nodesource.com/setup_15.x | bash - && \
        apt install -y nodejs
RUN npm i -g [email protected]
RUN mkdir /test
WORKDIR /test
RUN npm i --verbose 'github:jqplot/jqplot#d96a669fbb729f4f51e2214688e54320411219af'
RUN cat package-lock.json

{
  "name": "test",
  "lockfileVersion": 2,
  "requires": true,
  "packages": {
    "": {
      "dependencies": {
        "jqplot": "github:jqplot/jqplot#d96a669fbb729f4f51e2214688e54320411219af"
      }
    },
    "node_modules/jqplot": {
      "version": "1.0.9",
      "resolved": "git+ssh://[email protected]/jqplot/jqplot.git#d96a669fbb729f4f51e2214688e54320411219af",
      "integrity": "sha512-X/WC4DGdoiLof0cK/nTywyNzBwTNsEwH7Ky6ndwn5SUgsNmZDNnfugNhpfMX1y3Jh+GG6O9UxSMaFH/3pcffHQ==",
      "license": "(MIT AND GPL-2.0)"
    }
  },
  "dependencies": {
    "jqplot": {
      "version": "git+ssh://[email protected]/jqplot/jqplot.git#d96a669fbb729f4f51e2214688e54320411219af",
      "integrity": "sha512-X/WC4DGdoiLof0cK/nTywyNzBwTNsEwH7Ky6ndwn5SUgsNmZDNnfugNhpfMX1y3Jh+GG6O9UxSMaFH/3pcffHQ==",
      "from": "jqplot@github:jqplot/jqplot#d96a669fbb729f4f51e2214688e54320411219af"
    }
  }
}

---

FROM archlinux/base:latest
RUN set -ex && \
        pacman -Sy --noconfirm && \
        pacman -S --noconfirm nodejs npm git && \
        pacman -Syu --noconfirm
RUN npm i -g [email protected]
RUN mkdir /test
WORKDIR /test
RUN npm i --verbose 'github:jqplot/jqplot#d96a669fbb729f4f51e2214688e54320411219af'
RUN cat package-lock.json

{
  "name": "test",
  "lockfileVersion": 2,
  "requires": true,
  "packages": {
    "": {
      "dependencies": {
        "jqplot": "github:jqplot/jqplot#d96a669fbb729f4f51e2214688e54320411219af"
      }
    },
    "node_modules/jqplot": {
      "version": "1.0.9",
      "resolved": "git+ssh://[email protected]/jqplot/jqplot.git#d96a669fbb729f4f51e2214688e54320411219af",
      "integrity": "sha512-hMjKgDiIZ2RWZOe0wOUk9V1kWwyuvpNoqIoDT1hJ/1RmzKnYIfKM1BUPdJAo4gXr/LgmEF6GxGPZ1uXn7cfVBw==",
      "license": "(MIT AND GPL-2.0)"
    }
  },
  "dependencies": {
    "jqplot": {
      "version": "git+ssh://[email protected]/jqplot/jqplot.git#d96a669fbb729f4f51e2214688e54320411219af",
      "integrity": "sha512-hMjKgDiIZ2RWZOe0wOUk9V1kWwyuvpNoqIoDT1hJ/1RmzKnYIfKM1BUPdJAo4gXr/LgmEF6GxGPZ1uXn7cfVBw==",
      "from": "jqplot@github:jqplot/jqplot#d96a669fbb729f4f51e2214688e54320411219af"
    }
  }
}

When running npm pack in node_modules/jqplot manually I also get the different integrity hashes, but when I gunzip those files the .tar files have the same checksum so it's clearly related to gzip producing different output.

On my Gentoo system I get the same hash as on archlinux; I just used arch in the dockerfile because it's faster than compiling nodejs manually on a Gentoo image ;)

[luxaritas]

I think we're running into this as well - we rely on github repo-based packages, and having installs fail due to integrity issues based on who's installing is rather problematic.

[ahuth]

I think we're running into this, too.

The package-lock.json is generated on linux dev environments. But pushes to Heroku fail for the integrity checksum mismatch.

[akselikap]

I just ran into this issue and exactly the same variant as @ahuth. It seems like there hasn't been any updates to this as I'm using the newest version of npm (7.22.0) and it's still occurring.

[nevcos]

Same happening between Windows and Linux machines, very frustrating:

npm pack "git+ssh://[email protected]/jhiesey/idb-kv-store.git#109ccad165fd6470e12fd66025da9e4743a46043"
# integrity:     sha512-qpMpLBvXY0w8J[...]J8drdjr3X9/2w==   # Windows 10 with NPM 7.21.1
# integrity:     sha512-4IWjabwqciNun[...]aXD5qGvf1Yrnw==   # Windows 10 with NPM 6.14.15
# integrity:     sha512-qpMpLBvXY0w8J[...]J8drdjr3X9/2w==   # Ubuntu WSL with NPM 7.21.1
# integrity:     sha512-pu/VZ6qKntixi[...]iPaYSOpTaEgrA==   # Ubuntu WSL with NPM 6.14.15

[jameskerr]

I have a package that contains a prepare script to build a go binary when installed. When I run npm pack <my git dependency> I get a different integrity value each time I run it on the same machine. With npm 6 it was stable.

Npm 6 -- stable

$ npm pack "git+https://github.com/brimdata/zed.git#0b6233ba43b8ae48585af9e1f4e7b93c97195411" |& grep integrity
npm notice integrity:     sha512-DwGGh2q5ta6An[...]F71cfQHzLWyKQ==

$ npm pack "git+https://github.com/brimdata/zed.git#0b6233ba43b8ae48585af9e1f4e7b93c97195411" |& grep integrity
npm notice integrity:     sha512-DwGGh2q5ta6An[...]F71cfQHzLWyKQ==

$ npm pack "git+https://github.com/brimdata/zed.git#0b6233ba43b8ae48585af9e1f4e7b93c97195411" |& grep integrity
npm notice integrity:     sha512-DwGGh2q5ta6An[...]F71cfQHzLWyKQ==

$ npm -v
6.14.15

Npm 7 -- different every time

$ npm pack "git+https://github.com/brimdata/zed.git#0b6233ba43b8ae48585af9e1f4e7b93c97195411" |& grep integrity
npm notice integrity:     sha512-/i5QW944neOZe[...]PlmRMHSvk0hZA==

$ npm pack "git+https://github.com/brimdata/zed.git#0b6233ba43b8ae48585af9e1f4e7b93c97195411" |& grep integrity
npm notice integrity:     sha512-kh664omwauvXp[...]UDd6RbkQKYW9A==

$ npm pack "git+https://github.com/brimdata/zed.git#0b6233ba43b8ae48585af9e1f4e7b93c97195411" |& grep integrity
npm notice integrity:     sha512-iZW1VW06yVaQG[...]eytlUvcwkkR+g==

$ npm -v
7.24.2

Note on reproducing: This is the public Zed repo so you can use the commands above, but you'll need go and make installed for it to succeed.

[wraithgar]

npm/cli#2846 (comment)

[ahuth]

Can confirm this is fixed for me.