Skip to content

Commit

Permalink
Update package lock
Browse files Browse the repository at this point in the history
  • Loading branch information
@jaltekruse
jaltekruse committed Feb 23, 2022
1 parent f0779e1 commit 1c8ea11
Showing 1 changed file with 443 additions and 563 deletions.
Loading

15 comments on commit 1c8ea11

@maggie44
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this lock file valid? There are errors installing since this push. Only installs (npm install) if I delete this lock file first (node:16.13.2).

@jaltekruse
Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Maggie0002 I do believe this is valid, I am partway through updating all of the dependencies on a branch, as a number of them are out of date and as a part of that I think I installed a newer node version on my dev machine.

@maggie44
Copy link

@maggie44 maggie44 commented on 1c8ea11 Mar 16, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Maggie0002 I do believe this is valid, I am partway through updating all of the dependencies on a branch, as a number of them are out of date and as a part of that I think I installed a newer node version on my dev machine.

I tried with a couple of different Node versions. Which one are you using/is supported?

Have you tried deleting the node_modules folder and doing an npm install again? Or cloning the repo again to a clean folder and doing an npm install? I'm surprised it can't be reproduced. I suspect one of these steps will reproduce it, and if the node_modules folder was deleted, and the package-lock.json file was deleted, followed by an npm install, the new lock file would be slightly different to the one committed here.

@jaltekruse
Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I realized it likely is an npm version issue not node. I have node 16.3.1 and npm 8.3.0 installed and I just deleted my node_modules folder and re-installed on this commit and it was able to build for me. It does report a number of errors and warnings because of the out of date dependencies, would you possibly have npm configured to fail harder on these types of issues?

Happy to help get this resolved, you can schedule time with me using this link https://calendly.com/freemath

@maggie44
Copy link

@maggie44 maggie44 commented on 1c8ea11 Mar 16, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

npm ERR! code EINTEGRITY
npm ERR! sha512-gMCv51aQtW+VsAEXNwTXlMBPdAIe8w9VWSm2yHj2aFgOOCCBA0mQOd/KoNCgU9e3704msf7PGQA6nVuIOHWxAQ== integrity checksum failed when using sha512: wanted sha512-gMCv51aQtW+VsAEXNwTXlMBPdAIe8w9VWSm2yHj2aFgOOCCBA0mQOd/KoNCgU9e3704msf7PGQA6nVuIOHWxAQ== but got sha512-3iBj4jo/3kqU+tkN+ytCK6kh94KuF+H3A6jtrV3VQMFplLyJ+hegnFDRW5ogHVaiE1Y+QTiehNP7MMvm0P6Pbw==. (368183 bytes)

https://stackoverflow.com/a/51201661/16019434

Could your npm be building from cache? npm cache clean --force

I tried with npm 8.3.0, same issue. And if I delete package-lock.json and run the install, it installs no problem.

@maggie44
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Before and after deleting package-lock.json:

    "node_modules/tui-image-editor": {
      "version": "3.11.0",
      "resolved": "git+ssh://[email protected]/jaltekruse/tui.image-editor.git#3b55a120ff4336e1a3b5b5c51c21c5a01d5e623a",
      "integrity": "sha512-gMCv51aQtW+VsAEXNwTXlMBPdAIe8w9VWSm2yHj2aFgOOCCBA0mQOd/KoNCgU9e3704msf7PGQA6nVuIOHWxAQ==",
      "license": "MIT",
      "dependencies": {
        "core-js-pure": "^3.6.4",
        "fabric": "4.3.1",
        "tui-code-snippet": "^1.5.0",
        "tui-color-picker": "^2.2.6"
      }
    },

    "node_modules/tui-image-editor": {
      "version": "3.11.0",
      "resolved": "git+ssh://[email protected]/jaltekruse/tui.image-editor.git#3b55a120ff4336e1a3b5b5c51c21c5a01d5e623a",
      "integrity": "sha512-3iBj4jo/3kqU+tkN+ytCK6kh94KuF+H3A6jtrV3VQMFplLyJ+hegnFDRW5ogHVaiE1Y+QTiehNP7MMvm0P6Pbw==",
      "license": "MIT",
      "dependencies": {
        "core-js-pure": "^3.6.4",
        "fabric": "4.3.1",
        "tui-code-snippet": "^1.5.0",
        "tui-color-picker": "^2.2.6"
      }
    },

@jaltekruse
Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm, I do depend on a custom version of the tui-image-editor that I added a few new configs to and fixed some bugs in, but my machine isn't complaining about this, or generating a package-lock with different hashes if I delete it, and the node_moules folder and force clear the cache.

I just tried re-run the builds of the two repos that feed into this (there is the core editor and a react wrapper) and updated the dependency to point to a new version. Can you try to build the current tip of master I just pushed and see if that works?

@maggie44
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still an error.

npm ERR! code EINTEGRITY
npm ERR! sha512-8qiBd/Rd85goROwoijpESKoFQ2SS3t3+e2mVcERlrQfSGRsg0K2GzGsTBY5sxMwTAPbYgxXoyOsJ/7jeOi8+BQ== integrity checksum failed when using sha512: wanted sha512-8qiBd/Rd85goROwoijpESKoFQ2SS3t3+e2mVcERlrQfSGRsg0K2GzGsTBY5sxMwTAPbYgxXoyOsJ/7jeOi8+BQ== but got sha512-8Y/6Aez4RzkyBDBnPu7Dk22t75WgldA2waRXVZp4RL1KoTKZrQqtB+Jocsi3Ce7U7oY5uyRWZdp97TrSK6SlgA==. (368183 bytes)

Very weird. You could try replicating it in a Docker container. This pattern creates the error and is as close as we can probably get to running the same environment:

docker run -it node:16.13.2 sh

apt-get update && apt-get install \
    git \
    build-essential \
    libcairo2-dev \
    libpango1.0-dev \
    libjpeg-dev \
    libgif-dev \
    librsvg2-dev -y

git clone https://github.com/jaltekruse/Free-Math.git

cd Free-Math

npm install

@jaltekruse
Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @Maggie0002, sorry about the slow response on this. I finally got back to testing it out.

It appears that different machines can produce different hashes for git-based dependencies, so the npm team moved to completely remove integrity checksums for them. Apparently these checksums were based on gzipped archives, which are not guaranteed to be binary identical for the same inputs across different CPU architectures. There is still some cryptographic integrity defense as the dependency is pinned to a git commit and that relies on the entire previous history of the repo, as discussed in the later parts of this issue on npm.

npm/cli#2846

I have pushed a commit that removes these integrity values from the package-lock.json.

Thanks for reporting this and helping spread Free Math with the learners block!

@maggie44
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @Maggie0002, sorry about the slow response on this. I finally got back to testing it out.

It appears that different machines can produce different hashes for git-based dependencies, so the npm team moved to completely remove integrity checksums for them. Apparently these checksums were based on gzipped archives, which are not guaranteed to be binary identical for the same inputs across different CPU architectures. There is still some cryptographic integrity defense as the dependency is pinned to a git commit and that relies on the entire previous history of the repo, as discussed in the later parts of this issue on npm.

npm/cli#2846

I have pushed a commit that removes these integrity values from the package-lock.json.

Thanks for reporting this and helping spread Free Math with the learners block!

Interesting issue, good to know.

Still not quite there unfortunately. Just tired those docker steps I outlined above and got the following:

npm ERR! code 1
npm ERR! git dep preparation failed
npm ERR! command /usr/local/bin/node /usr/local/lib/node_modules/npm/bin/npm-cli.js install --force --cache=/root/.npm --prefer-offline=false --prefer-online=false --offline=false --no-progress --no-save --no-audit --include=dev --include=peer --include=optional --no-package-lock-only --no-dry-run
npm ERR! npm WARN using --force Recommended protections disabled.
npm ERR! npm WARN tarball tarball data for tui-image-editor@git+ssh://[email protected]/jaltekruse/tui.image-editor.git#6bdc2614ed4d014a357518e096e55fa8a4ee77f1 (sha512-8qiBd/Rd85goROwoijpESKoFQ2SS3t3+e2mVcERlrQfSGRsg0K2GzGsTBY5sxMwTAPbYgxXoyOsJ/7jeOi8+BQ==) seems to be corrupted. Trying again.
npm ERR! npm WARN tarball tarball data for tui-image-editor@git+ssh://[email protected]/jaltekruse/tui.image-editor.git#6bdc2614ed4d014a357518e096e55fa8a4ee77f1 (sha512-8qiBd/Rd85goROwoijpESKoFQ2SS3t3+e2mVcERlrQfSGRsg0K2GzGsTBY5sxMwTAPbYgxXoyOsJ/7jeOi8+BQ==) seems to be corrupted. Trying again.
npm ERR! npm ERR! code EINTEGRITY
npm ERR! npm ERR! sha512-8qiBd/Rd85goROwoijpESKoFQ2SS3t3+e2mVcERlrQfSGRsg0K2GzGsTBY5sxMwTAPbYgxXoyOsJ/7jeOi8+BQ== integrity checksum failed when using sha512: wanted sha512-8qiBd/Rd85goROwoijpESKoFQ2SS3t3+e2mVcERlrQfSGRsg0K2GzGsTBY5sxMwTAPbYgxXoyOsJ/7jeOi8+BQ== but got sha512-8Y/6Aez4RzkyBDBnPu7Dk22t75WgldA2waRXVZp4RL1KoTKZrQqtB+Jocsi3Ce7U7oY5uyRWZdp97TrSK6SlgA==. (368183 bytes)
npm ERR! 
npm ERR! npm ERR! A complete log of this run can be found in:
npm ERR! npm ERR!     /root/.npm/_logs/2022-03-24T14_49_30_441Z-debug.log

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2022-03-24T14_49_32_987Z-debug.log

I am using an M1 like it notes in that issue post, although I am getting the same issue on a GitHub Linux runner too.

@maggie44
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interestingly, now when I delete package-lock.json it fails too, whereas before that used to let it go through. After deleting package-lock.json and running npm install:

npm ERR! code 1
npm ERR! git dep preparation failed
npm ERR! command /usr/local/bin/node /usr/local/lib/node_modules/npm/bin/npm-cli.js install --force --cache=/root/.npm --prefer-offline=false --prefer-online=false --offline=false --no-progress --no-save --no-audit --include=dev --include=peer --include=optional --no-package-lock-only --no-dry-run
npm ERR! npm WARN using --force Recommended protections disabled.
npm ERR! npm WARN tarball tarball data for tui-image-editor@git+ssh://[email protected]/jaltekruse/tui.image-editor.git#6bdc2614ed4d014a357518e096e55fa8a4ee77f1 (sha512-8qiBd/Rd85goROwoijpESKoFQ2SS3t3+e2mVcERlrQfSGRsg0K2GzGsTBY5sxMwTAPbYgxXoyOsJ/7jeOi8+BQ==) seems to be corrupted. Trying again.
npm ERR! npm WARN tarball tarball data for tui-image-editor@git+ssh://[email protected]/jaltekruse/tui.image-editor.git#6bdc2614ed4d014a357518e096e55fa8a4ee77f1 (sha512-8qiBd/Rd85goROwoijpESKoFQ2SS3t3+e2mVcERlrQfSGRsg0K2GzGsTBY5sxMwTAPbYgxXoyOsJ/7jeOi8+BQ==) seems to be corrupted. Trying again.
npm ERR! npm ERR! code EINTEGRITY
npm ERR! npm ERR! sha512-8qiBd/Rd85goROwoijpESKoFQ2SS3t3+e2mVcERlrQfSGRsg0K2GzGsTBY5sxMwTAPbYgxXoyOsJ/7jeOi8+BQ== integrity checksum failed when using sha512: wanted sha512-8qiBd/Rd85goROwoijpESKoFQ2SS3t3+e2mVcERlrQfSGRsg0K2GzGsTBY5sxMwTAPbYgxXoyOsJ/7jeOi8+BQ== but got sha512-8Y/6Aez4RzkyBDBnPu7Dk22t75WgldA2waRXVZp4RL1KoTKZrQqtB+Jocsi3Ce7U7oY5uyRWZdp97TrSK6SlgA==. (368183 bytes)
npm ERR! 
npm ERR! npm ERR! A complete log of this run can be found in:
npm ERR! npm ERR!     /root/.npm/_logs/2022-03-24T14_55_28_403Z-debug.log

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2022-03-24T14_56_02_051Z-debug.log

@jaltekruse
Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @Maggie0002 , sorry about the slow follow up. I haven't been able to reproduce this myself, but I'm hopeful I fixed it with the latest push. I removed all references to integrity SHAs of the package-lock.json files in the dependencies that are being pulled in. Can you try to build the tip of master?

Out of curiosity, are you by any chance using an M1 mac? I did just try the build on an intel mac and it worked fine for me, my regular machine is just a linux latop. The arm architecture of the M1 macs seemed to be the primary culprit for the bug manifesting, but it could be anything that would make gzip produce different compression results for the same input, so I guess that could be stuff other than CPU architecture maybe?

@maggie44
Copy link

@maggie44 maggie44 commented on 1c8ea11 Apr 14, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @Maggie0002 , sorry about the slow follow up. I haven't been able to reproduce this myself, but I'm hopeful I fixed it with the latest push. I removed all references to integrity SHAs of the package-lock.json files in the dependencies that are being pulled in. Can you try to build the tip of master?

Out of curiosity, are you by any chance using an M1 mac? I did just try the build on an intel mac and it worked fine for me, my regular machine is just a linux latop. The arm architecture of the M1 macs seemed to be the primary culprit for the bug manifesting, but it could be anything that would make gzip produce different compression results for the same input, so I guess that could be stuff other than CPU architecture maybe?

Yes it is M1, but I am also seeing the error on a GitHub runner which isn’t M1. Both are in Docker though. Will test later today.

@maggie44
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You were right about the architecture thing, I had forgotten that the GitHub runner was building multi-arch docker images and it was armv7 tripping it up. It was still generating errors though. As was my M1 Mac.

Good news and bad news. Updating NPM seemed to resolve the issue (npm install -g [email protected]). Good news that it is resolved, bad news in that perhaps this would have resolved it all along? I am trying to test again by pulling the original commit and trying npm 8.7.0 but for some reason my system isn't letting me do any apt update or apt install, I guess the stars are just not aligned right now. Will try and test at some point. For now, all working, thanks for taking the time.

@jaltekruse
Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @Maggie0002 ,

Glad to hear that this resolved the issue!

Actually based on the discussion of the NPM issue, it isn't surprising that upgrading NPM would make the original commit that caused you problems work.

The "fix" to the problem that they merged into a recent NPM version was just to ignore these integrity values, as they weren't stable across CPU architectures. The "fixes" I was doing on this end were just removing those integrity values manually from the package lock files so earlier NPM versions wouldn't fail on ARM (both in Free Math and the two repos I have the customized dependencies in).

Happy to help, thank you again for spreading Free Math. I would love to hear more about your work, if you have a few minutes to spare and would be up for swapping notes on building open source edtech stuff you can schedule time with me at the link I had shared: https://calendly.com/freemath

Please sign in to comment.