Skip to content
This repository has been archived by the owner on Aug 11, 2022. It is now read-only.

npm 5.0.1 can't install a specific git dependency #16898

Closed
1 of 13 tasks
mistydemeo opened this issue Jun 2, 2017 · 48 comments
Closed
1 of 13 tasks

npm 5.0.1 can't install a specific git dependency #16898

mistydemeo opened this issue Jun 2, 2017 · 48 comments

Comments

@mistydemeo
Copy link

I'm opening this issue because:

  • npm is crashing.
  • npm is producing an incorrect install.
  • npm is doing something I don't understand.
  • Other (see below for feature requests):

What's going wrong?

When I try to install a package which uses a particular git repository at a specific commit, npm tries to git clone with the wrong branch and the clone fails. The specific spec is https://github.com/github/hubot-slack.git#87f012a8112d52261e1572e05c545b5ae59ababd (or its shorthand github/hubot-slack#87f012a8112d52261e1572e05c545b5ae59ababd). Instead of trying to clone at that commit, npm runs the following command:

/usr/local/bin/git clone --depth=1 -q -b 11/head https://github.com/github/hubot-slack.git

e.g., it's trying to clone branch 11/head. I'm not sure where it's getting that from. npm 3 does fetch this repository/reference correctly.

Sorry if this is reported in the wrong project. :) I stepped a module or two down the dependency tree but couldn't quite place what code is responsible for doing the clone and preparing its arguments.

How can the CLI team reproduce the problem?

The following package.json, with a single dependency, reproduces the problem:

{
  "name": "repro",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "",
  "license": "ISC",
  "dependencies": {
    "hubot-slack": "https://github.com/github/hubot-slack.git#87f012a8112d52261e1572e05c545b5ae59ababd"
  }
}

The debug log is here: https://gist.github.com/283f0f4484d3a8dd7b0cf31acda81b40

supporting information:

  • npm -v prints: 5.0.1
  • node -v prints: v8.0.0
  • npm config get registry prints: https://registry.npmjs.org/
  • Windows, OS X/macOS, or Linux?: macOS 10.12
  • Network issues:
    • Geographic location where npm was run: Vancouver, Canada
    • I use a proxy to connect to the npm registry.
    • I use a proxy to connect to the web.
    • I use a proxy when downloading Git repos.
    • I access the npm registry via a VPN
    • I don't use a proxy, but have limited or unreliable internet access.
  • Container:
    • I develop using Vagrant on Windows.
    • I develop using Vagrant on OS X or Linux.
    • I develop / deploy using Docker.
    • I deploy to a PaaS (Triton, Heroku).
@zkat
Copy link
Contributor

zkat commented Jun 2, 2017

bah humbug. This one requires some explanation... this is all in pacote itself.

So! There's some cleverness that npm5 added in order to maximize how often it does shallow clones rather than full clones. I researched all the various restrictions on different kinds of clone to see what could be done, and one thing that stood out was that if we have an actual named ref, then we can (probably?) do a git clone of that ref after getting a remote rev list and seeing if the hash that the user requested is in that list.

More succintly, the algo is something like:

  1. User requests user/repo#deadbeef.
  2. pacote does an ls-remote on user/repo and collects all the refs
  3. it looks for deadbeef in the list of refs, and picks the ref that refers to the hash, if it finds a match
  4. If it finds a named ref, it will then try to do a shallow clone of the repo targeting that ref as a branch -- which means the clone only needs to fetch the one commit and its objects
  5. if the clone succeeds, it packs it up in a tarball and the rest of the install continues.

I'm not sure off the top of my head why you're getting 11/head specifically, or why that doesn't work. I think there's two things that can both be done pacote-side to fix this without throwing away the shallow clone trick:

  1. if a shallow clone fails, pacote should just fall back and retry the full clone without using the ref, and switch to the commit by hash before packing

  2. we should figure out why 11/head in particular was picked as The Chosen Ref, and then why it didn't actually work.

Both of these changes would happen pacote-side, but I'll keep this issue open to track it. I think the bulk of the changes needed to do this could be done in pacote's git fetcher, with possibly some tweaks to pacote's git utility depending on the findings.

If anyone wants to pick this patch up, I'll help however I can and accept a PR that fixes this over there.

@mistydemeo thanks for this report! I'm really glad we found a corner case with my Cunning Plan™. I clearly did not think it all the way through 😂

@mistydemeo
Copy link
Author

Looks like the output of git ls-remote is being used to get the ref:

$ git ls-remote https://github.com/github/hubot-slack.git
a23e19405f33da0de68c123f7dad2ef01fd572fd	HEAD
0f20def5919a7fede5ec5b250ac22c0568d80505	refs/heads/adapter-send-attachments
ed214feb5e64ae815761c08ba1bca68911edf7a2	refs/heads/always-use-chat-postMessage
ea562aeadab3f9f58e8db6ee5e86a4a41509db6a	refs/heads/better-formatting-removal
fa5085a506e80359b6429963988973731230c77b	refs/heads/bhuga-patch-1
9c670bc6349c4cb3fe5392d8d6bfa8b1454b4719	refs/heads/custom-response
e2e197c3c1b67f0929efa675127ac24c89a0a646	refs/heads/fallback-on-attachments
e37132d9975076ad61330e6f728a5fc08137bedc	refs/heads/last-message-timestamp
a23e19405f33da0de68c123f7dad2ef01fd572fd	refs/heads/master
4fba9145c414cfe37b0ea60ef020b4a3bfb23af6	refs/heads/no_attachment_archive_links
421c44ebec4de6f588490fa540267c9d9acfd316	refs/heads/not-an-attachment
9674b7875ff1601bde13b436c7ab47aa0ee60972	refs/heads/persist-more-slack-user-data
3d80813c6927f3b1ca841c1cd2a8e549e3e7982b	refs/heads/private-messages
ecf9db9fb3bb1b905fe83d0a86e9fc7165afd625	refs/heads/remember-less-slack-info
ad634a5b19ba454375c3b491a2048e2c7a45a637	refs/heads/restore_rawText
b2617e001ec0082e72ff27ff3779dcd90f3a9d2a	refs/heads/send-multiline-as-attachments
f6a7c715ab2273cb3495c5de3d659d8152fada24	refs/heads/travis-fix
8a8cd0904d76f17495f856c73dab5edd85765808	refs/heads/v2
df7d2f7fc4148de782da56a82e1a4b1229466ebc	refs/pull/1/head
cf658d8a68e04d5eb3287cfe402891881bbf31b9	refs/pull/10/head
87f012a8112d52261e1572e05c545b5ae59ababd	refs/pull/11/head
b1bbd94d9cadf61500c892d1b84e0748c6083b6a	refs/pull/3/head
e2e197c3c1b67f0929efa675127ac24c89a0a646	refs/pull/4/head
e2e197c3c1b67f0929efa675127ac24c89a0a646	refs/pull/5/head
fa5085a506e80359b6429963988973731230c77b	refs/pull/6/head
504fa95f091f216755f4d0a1faf2497b45686987	refs/pull/7/head
ee90948bf52c22ea98c52b996d96474ee6e18561	refs/pull/8/head
4fba9145c414cfe37b0ea60ef020b4a3bfb23af6	refs/pull/9/head
df3e9cc20fb54ae28125268add6cbdac48aec858	refs/pull/9/merge
bbc23aa87580f38c2093a24ec52669dcb7645a81	refs/tags/1.5.0
e70a4a258fa6108e4dd5086483f5d72373b15966	refs/tags/1.5.1
a0d0a43336676368d7d1bffdb9b5d815bda8aeb3	refs/tags/1.5.1^{}
c502bfb0429484209c31bce10771adcc6900b326	refs/tags/1.5.2
dc09cc183f535a5749c1f9afe19be9634ee3ed8a	refs/tags/1.5.2^{}
1c4ab632b72e4f58ee474678c3baf43801567bed	refs/tags/1.5.3
16741d0fc2e8fddee780629f03c875959e8aea10	refs/tags/1.5.3^{}
ed623d540b5b39659c487808798c24fe70d6c3da	refs/tags/2.0.0
2febcf550065e4a26d871fbe994f12baf9cbffe3	refs/tags/2.0.0^{}
926a87de8a5221dea977e062a77cb32a2a6a4deb	refs/tags/2.0.1
9ce5ddcd93b4571fae989a677329246766c07230	refs/tags/2.0.1^{}
e60e034f10f726f882b4b94fa0de6571094f140b	refs/tags/2.0.2
fca11022dd0ba3d09b0ff31719b9d2aae8704d8b	refs/tags/2.0.2^{}
e63a298234af5c164c600b248a655d86b5544da5	refs/tags/2.0.3
f985af87bd42b2c96c184b6f4c642db4ce1a9cf1	refs/tags/2.0.3^{}
cb035f781f93717115de4f2dbeb279c8c64c84b2	refs/tags/2.0.4
c07240c231326ba2b612b99ab38d4a224bec8646	refs/tags/2.0.4^{}
1f865e2381a8de7abf11d507c882b73bcc3c8b8a	refs/tags/2.1.0
a7d6b5c7cfd1ea6aa3b4b048a116a0796ba43310	refs/tags/2.1.0^{}
64d1566d91c6fc37ce52bc92179ebb67aaadc523	refs/tags/2.1.1
091b598152239ac9567129121be9ff617e7a8e14	refs/tags/2.1.1^{}
9a7cadd87cd7bfb90c685ccacec8f777a6840b97	refs/tags/2.1.2
335e694207d77fe16d7bc21079998645bd30804f	refs/tags/2.1.2^{}
db7837977db5d547bc608d6cb2506350db2446be	refs/tags/2.1.3
8a5d7dcc273e5c62ff929143e945b16f2b26894b	refs/tags/2.1.3^{}
010905c9512d34939845b3f056bd9cee9b0c6b6a	refs/tags/2.2.0
c90ba26076b9ac4321c9baf9d239cafe7945e223	refs/tags/2.2.0^{}
f43deba2b53a73c01aaf1a046dc49ccf12291261	refs/tags/3.0.0
dd265c8ab8895e24b136515635eec714602d5036	refs/tags/3.0.0^{}
a1bf28281b3aa7135fb4c0473efaa6e9787b488e	refs/tags/3.1.0
6afbc1ba1213006e75975c3bfeb0eb9c9d8993e8	refs/tags/3.1.0^{}
aeb96c85f705188b7f7288244fad1e899b8770dd	refs/tags/3.2.0
d8c23424f96ce59e3f2e89ee0daf204ddf4dc4fe	refs/tags/3.2.0^{}
ba9fffb9e572ff890a190bf28adeb0e196eeef1a	refs/tags/3.2.1
f3a288b44b7df5eb688c84a1843b9e10c0c6efeb	refs/tags/3.2.1^{}
a2890d12104cf45cbd9487f9a4c7c8ba47921fb1	refs/tags/3.3.0
af9001bc14c2af358aaec301272b2ec95cb8d61c	refs/tags/3.3.0^{}

This line is what's transforming refs/pull/11/head into 11/head. The chosen ref here is neither a tag nor a branch, and may not be clonable.

@zkat
Copy link
Contributor

zkat commented Jun 2, 2017

I bet the fix is just to change that command back to ls-remote -h -t *.

@mistydemeo
Copy link
Author

Yeah, I think so:

$ git ls-remote -h -t https://github.com/github/hubot-slack.git '*'
0f20def5919a7fede5ec5b250ac22c0568d80505	refs/heads/adapter-send-attachments
ed214feb5e64ae815761c08ba1bca68911edf7a2	refs/heads/always-use-chat-postMessage
ea562aeadab3f9f58e8db6ee5e86a4a41509db6a	refs/heads/better-formatting-removal
fa5085a506e80359b6429963988973731230c77b	refs/heads/bhuga-patch-1
9c670bc6349c4cb3fe5392d8d6bfa8b1454b4719	refs/heads/custom-response
e2e197c3c1b67f0929efa675127ac24c89a0a646	refs/heads/fallback-on-attachments
e37132d9975076ad61330e6f728a5fc08137bedc	refs/heads/last-message-timestamp
a23e19405f33da0de68c123f7dad2ef01fd572fd	refs/heads/master
4fba9145c414cfe37b0ea60ef020b4a3bfb23af6	refs/heads/no_attachment_archive_links
421c44ebec4de6f588490fa540267c9d9acfd316	refs/heads/not-an-attachment
9674b7875ff1601bde13b436c7ab47aa0ee60972	refs/heads/persist-more-slack-user-data
3d80813c6927f3b1ca841c1cd2a8e549e3e7982b	refs/heads/private-messages
ecf9db9fb3bb1b905fe83d0a86e9fc7165afd625	refs/heads/remember-less-slack-info
ad634a5b19ba454375c3b491a2048e2c7a45a637	refs/heads/restore_rawText
b2617e001ec0082e72ff27ff3779dcd90f3a9d2a	refs/heads/send-multiline-as-attachments
f6a7c715ab2273cb3495c5de3d659d8152fada24	refs/heads/travis-fix
8a8cd0904d76f17495f856c73dab5edd85765808	refs/heads/v2
bbc23aa87580f38c2093a24ec52669dcb7645a81	refs/tags/1.5.0
e70a4a258fa6108e4dd5086483f5d72373b15966	refs/tags/1.5.1
a0d0a43336676368d7d1bffdb9b5d815bda8aeb3	refs/tags/1.5.1^{}
c502bfb0429484209c31bce10771adcc6900b326	refs/tags/1.5.2
dc09cc183f535a5749c1f9afe19be9634ee3ed8a	refs/tags/1.5.2^{}
1c4ab632b72e4f58ee474678c3baf43801567bed	refs/tags/1.5.3
16741d0fc2e8fddee780629f03c875959e8aea10	refs/tags/1.5.3^{}
ed623d540b5b39659c487808798c24fe70d6c3da	refs/tags/2.0.0
2febcf550065e4a26d871fbe994f12baf9cbffe3	refs/tags/2.0.0^{}
926a87de8a5221dea977e062a77cb32a2a6a4deb	refs/tags/2.0.1
9ce5ddcd93b4571fae989a677329246766c07230	refs/tags/2.0.1^{}
e60e034f10f726f882b4b94fa0de6571094f140b	refs/tags/2.0.2
fca11022dd0ba3d09b0ff31719b9d2aae8704d8b	refs/tags/2.0.2^{}
e63a298234af5c164c600b248a655d86b5544da5	refs/tags/2.0.3
f985af87bd42b2c96c184b6f4c642db4ce1a9cf1	refs/tags/2.0.3^{}
cb035f781f93717115de4f2dbeb279c8c64c84b2	refs/tags/2.0.4
c07240c231326ba2b612b99ab38d4a224bec8646	refs/tags/2.0.4^{}
1f865e2381a8de7abf11d507c882b73bcc3c8b8a	refs/tags/2.1.0
a7d6b5c7cfd1ea6aa3b4b048a116a0796ba43310	refs/tags/2.1.0^{}
64d1566d91c6fc37ce52bc92179ebb67aaadc523	refs/tags/2.1.1
091b598152239ac9567129121be9ff617e7a8e14	refs/tags/2.1.1^{}
9a7cadd87cd7bfb90c685ccacec8f777a6840b97	refs/tags/2.1.2
335e694207d77fe16d7bc21079998645bd30804f	refs/tags/2.1.2^{}
db7837977db5d547bc608d6cb2506350db2446be	refs/tags/2.1.3
8a5d7dcc273e5c62ff929143e945b16f2b26894b	refs/tags/2.1.3^{}
010905c9512d34939845b3f056bd9cee9b0c6b6a	refs/tags/2.2.0
c90ba26076b9ac4321c9baf9d239cafe7945e223	refs/tags/2.2.0^{}
f43deba2b53a73c01aaf1a046dc49ccf12291261	refs/tags/3.0.0
dd265c8ab8895e24b136515635eec714602d5036	refs/tags/3.0.0^{}
a1bf28281b3aa7135fb4c0473efaa6e9787b488e	refs/tags/3.1.0
6afbc1ba1213006e75975c3bfeb0eb9c9d8993e8	refs/tags/3.1.0^{}
aeb96c85f705188b7f7288244fad1e899b8770dd	refs/tags/3.2.0
d8c23424f96ce59e3f2e89ee0daf204ddf4dc4fe	refs/tags/3.2.0^{}
ba9fffb9e572ff890a190bf28adeb0e196eeef1a	refs/tags/3.2.1
f3a288b44b7df5eb688c84a1843b9e10c0c6efeb	refs/tags/3.2.1^{}
a2890d12104cf45cbd9487f9a4c7c8ba47921fb1	refs/tags/3.3.0
af9001bc14c2af358aaec301272b2ec95cb8d61c	refs/tags/3.3.0^{}

@zkat
Copy link
Contributor

zkat commented Jun 2, 2017

@mistydemeo would you like to do a PR against pacote, or should I? I definitely lean more towards "MOAR CONTRIBZ" if I can <3

@mistydemeo
Copy link
Author

I'll do a PR!

@ljharb
Copy link
Contributor

ljharb commented Jun 2, 2017

Please note that github PR refs are actually quite cloneable (origin/pr/123) and it'll be important to retain the ability to npm install a pull request by ref :-)

mistydemeo added a commit to mistydemeo/pacote that referenced this issue Jun 2, 2017
GitHub repositories can contain other types of refs, such as pull
requests, which can be fetched via `git-fetch` but not cloned via
`git clone -b`.

Fixes npm/npm#16898.
mistydemeo added a commit to mistydemeo/pacote that referenced this issue Jun 2, 2017
GitHub repositories can contain other types of refs, such as pull
requests, which can be fetched via `git-fetch` but not cloned via
`git clone -b`.

Fixes npm/npm#16898.
@zkat
Copy link
Contributor

zkat commented Jun 2, 2017

@ljharb you should still be able to clone by direct ref -- it just won't hit the fast-path for cloning. Maybe it's worth going back to the no-args version, and doing manual filtering? That's possible, too. Still gotta fix whatever's making HEAD fail like that, tho

@ljharb
Copy link
Contributor

ljharb commented Jun 2, 2017

Cool, fast vs slow path is an implementation detail; as long as this fix doesn't break the functionality, I'm happy :-)

zkat pushed a commit to zkat/pacote that referenced this issue Jun 5, 2017
GitHub repositories can contain other types of refs, such as pull
requests, which can be fetched via `git-fetch` but not cloned via
`git clone -b`.

Fixes: npm/npm#16898
@shubhamarora
Copy link

I getting the following error. (upgraded to 5.0.3 after reading linked threads).

npm ERR! code 128
npm ERR! Command failed: /usr/bin/git clone --depth=1 -q -b master https://github.com/<REPO>.git /root/.npm/_cacache/tmp/git-clone-734abb0f
npm ERR! fatal: could not create leading directories of '/root/.npm/_cacache/tmp/git-clone-734abb0f': Permission denied

Ran chown and chmod 777 for /root/.npm/_cacache/tmp/ but still facing the same error.
@zkat @mistydemeo

@zkat
Copy link
Contributor

zkat commented Jun 6, 2017

@shubhamarora try running it as root or sudo if you're gonna be writing to /root, or just change your prefix

@shubhamarora
Copy link

@zkat I am already running it as root. :|

@zkat
Copy link
Contributor

zkat commented Jun 7, 2017

@shubhamarora ooh interesting. I think I get it. But I'm confused why it worked in older npm versions. I'm lowering permissions for the git process if you have SUDO_UID/SUDO_GID. I'll have to look closer at what npm was doing before -- it was probably working by virtue of creating all the directories earlier. Or not dropping perms the way I thought it was. Cheers, and thanks 👍

@mc-zone
Copy link

mc-zone commented Jun 7, 2017

Maybe same problem as @shubhamarora I‘ve just met (after upgraded to 5.0.3):

0 info it worked if it ends with ok
1 verbose cli [ '/usr/local/Cellar/node/6.7.0/bin/node',
1 verbose cli   '/usr/local/bin/npm',
1 verbose cli   'install' ]
2 info using [email protected]
3 info using [email protected]

......

2348 silly fetchPackageMetaData error for PRIVATE_PROJ@git+http://PRIVATE_PROJ_URL.git#be967bf1bf0d9f6052290cfdfd095519ff1d04c9 Command failed: /usr/bin/git clone -q http://PRIVATE_PROJ_URL.git /Users/mc-zone/.npm/_cacache/tmp/git-clone-6df1e4ff
2348 silly fetchPackageMetaData /Users/mc-zone/.npm/_cacache/tmp/git-clone-6df1e4ff/.git: Permission denied
2348 verbose stack
2348 verbose stack     at ChildProcess.exithandler (child_process.js:206:12)
2348 verbose stack     at emitTwo (events.js:106:13)
2348 verbose stack     at ChildProcess.emit (events.js:191:7)
2348 verbose stack     at maybeClose (internal/child_process.js:877:16)
2348 verbose stack     at Socket.<anonymous> (internal/child_process.js:334:11)
2348 verbose stack     at emitOne (events.js:96:13)
2348 verbose stack     at Socket.emit (events.js:188:7)
2348 verbose stack     at Pipe._handle.close [as _onclose] (net.js:493:12)

Looks like it can't clone our repos into _cacache/tmp/xxx :

fetchPackageMetaData /Users/mc-zone/.npm/_cacache/tmp/git-clone-6df1e4ff/.git: Permission denied

I've also ran it with sudo, and chmod -R 777 for ~/.npm, still not work.

But work fine after back to [email protected].

SYS: Mac OS X 10.11.6

cc @zkat .

@shubhamarora
Copy link

@zkat maybe you can check version 3.10.3 (one I am running currently). Can you help me understand this statement? - "... it was probably working by virtue of creating all the directories earlier. Or not dropping perms the way I thought it was.". And I can help in fixing this if you can guide me?

@zkat
Copy link
Contributor

zkat commented Jun 8, 2017

@shubhamarora There's a new canary, [email protected], which includes some more git fixes. Mind giving it a whirl and seeing if things work well again for you? (with npm i -g [email protected] && npmc install)

@ferrants
Copy link

ferrants commented Jun 8, 2017

Hitting this issue as well.

@zkat , I tried that:

$ npm i -g [email protected] && npmc install
/var/lib/jenkins/.nvm/versions/node/v4.4.6/bin/npmc -> /var/lib/jenkins/.nvm/versions/node/v4.4.6/lib/node_modules/npmc/bin/npm-cli.js
+ [email protected]
added 350 packages in 51.847s
npm notice CANARY npmc is experimental software. If you find an issue, please file it in the main npm repository, and call out that you were using npmc.
npm ERR! code 128
npm ERR! Command failed: /usr/bin/git clone --depth=1 -q -b master http://ikt.pm2.io/ikt.git /var/lib/jenkins/.npm/_cacache/tmp/git-clone-cff92e9b
npm ERR! warning: templates not found /tmp/pacote-git-template-tmp/git-clone-6791ef75
npm ERR! error: RPC failed; curl 18 transfer closed with outstanding read data remaining
npm ERR! fatal: The remote end hung up unexpectedly
npm ERR! 

npm ERR! A complete log of this run can be found in:
npm ERR!     /var/lib/jenkins/.npm/_logs/2017-06-08T17_07_36_769Z-debug.log

$ tail -n 200 /var/lib/jenkins/.npm/_logs/2017-06-08T17_07_36_769Z-debug.log
0 info it worked if it ends with ok
1 verbose cli [ '/var/lib/jenkins/.nvm/versions/node/v4.4.6/bin/node',
1 verbose cli   '/var/lib/jenkins/.nvm/versions/node/v4.4.6/bin/npmc',
1 verbose cli   'install' ]
2 info using [email protected]
3 info using [email protected]
4 notice CANARY npmc is experimental software. If you find an issue, please file it in the main npm repository, and call out that you were using npmc.
5 verbose npm-session 069993fda593d76f
6 silly install runPreinstallTopLevelLifecycles
7 silly preinstall [email protected]
8 info lifecycle [email protected]~preinstall: [email protected]
9 silly lifecycle [email protected]~preinstall: no script for preinstall, continuing
10 silly install loadCurrentTree
11 silly install readLocalPackageData
12 silly install loadIdealTree
13 silly install cloneCurrentTreeToIdealTree
14 silly install loadShrinkwrap
15 silly fetchPackageMetaData error for ikt@git+http://ikt.pm2.io/ikt.git#3325a3e39a502418dc2e2e4bf21529cbbde96228 Command failed: /usr/bin/git clone --depth=1 -q -b master http://ikt.pm2.io/ikt.git /var/lib/jenkins/.npm/_cacache/tmp/git-clone-cff92e9b
15 silly fetchPackageMetaData warning: templates not found /tmp/pacote-git-template-tmp/git-clone-6791ef75
15 silly fetchPackageMetaData error: RPC failed; curl 18 transfer closed with outstanding read data remaining
15 silly fetchPackageMetaData fatal: The remote end hung up unexpectedly
16 verbose stack Error: Command failed: /usr/bin/git clone --depth=1 -q -b master http://ikt.pm2.io/ikt.git /var/lib/jenkins/.npm/_cacache/tmp/git-clone-cff92e9b
16 verbose stack warning: templates not found /tmp/pacote-git-template-tmp/git-clone-6791ef75
16 verbose stack error: RPC failed; curl 18 transfer closed with outstanding read data remaining
16 verbose stack fatal: The remote end hung up unexpectedly
16 verbose stack
16 verbose stack     at ChildProcess.exithandler (child_process.js:213:12)
16 verbose stack     at emitTwo (events.js:87:13)
16 verbose stack     at ChildProcess.emit (events.js:172:7)
16 verbose stack     at maybeClose (internal/child_process.js:827:16)
16 verbose stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:211:5)
17 verbose cwd /var/lib/jenkins/traverse-api
18 verbose Linux 4.4.23-31.54.amzn1.x86_64
19 verbose argv "/var/lib/jenkins/.nvm/versions/node/v4.4.6/bin/node" "/var/lib/jenkins/.nvm/versions/node/v4.4.6/bin/npmc" "install"
20 verbose node v4.4.6
21 verbose npm  v5.0.3-canary.10
22 error code 128
23 error Command failed: /usr/bin/git clone --depth=1 -q -b master http://ikt.pm2.io/ikt.git /var/lib/jenkins/.npm/_cacache/tmp/git-clone-cff92e9b
23 error warning: templates not found /tmp/pacote-git-template-tmp/git-clone-6791ef75
23 error error: RPC failed; curl 18 transfer closed with outstanding read data remaining
23 error fatal: The remote end hung up unexpectedly
24 verbose exit [ 1, true ]

Still failing on pacote

@zkat
Copy link
Contributor

zkat commented Jun 8, 2017

npm ERR! error: RPC failed; curl 18 transfer closed with outstanding read data remaining
npm ERR! fatal: The remote end hung up unexpectedly

What

@ferrants
Copy link

ferrants commented Jun 8, 2017

Apologies, maybe this isn't the same issue. At first glance, it seemed to be.

@ferrants
Copy link

ferrants commented Jun 9, 2017

@zkat, i'm still running into an issue. what do you think of this series of commands to test:

$ npm install                       # on my older npm version with a npm-shrinkwrap.json to get all the modules in place
$ npmc shrinkwrap --dev             # shrinkwrap with new npmc to have the npm-shrinkwrap.json have all the same versions I currently use
$ mv node_modules/ node_modulesbk/  # moved aside so I can reference later
$ npm install                       # see if my older npm version worked with new shrinkwrap, it finished successfully
$ rm -rf node_modules/              # removed so I can try with npmc
$ npmc install                      # attempt with npmc, failed

Versions:

$ npm --version
2.15.5

$ npmc --version
5.0.3-canary.12

The npmc install failed like this:

$ npmc install
npm notice CANARY npmc is experimental software. If you find an issue, please file it in the main npm repository, and call out that you were using npmc.
events.js:141......] \ extract:async: sill extract [email protected]
      throw er; // Unhandled 'error' event
      ^

Error: sha1-kwAtRjBP2LQvvhw56auEqJ8U4n8= integrity checksum failed when using sha1: wanted sha1-kwAtRjBP2LQvvhw56auEqJ8U4n8= but got sha1-gVsA/I03qh/watPfLgV08+bad48=. (15470 bytes)
    at Transform.<anonymous> (/var/lib/jenkins/.nvm/versions/node/v4.4.6/lib/node_modules/npmc/node_modules/ssri/index.js:275:19)
    at emitNone (events.js:72:20)
    at Transform.emit (events.js:166:7)
    at endReadableNT (_stream_readable.js:913:12)
    at nextTickCallbackWith2Args (node.js:442:9)
    at process._tickCallback (node.js:356:17)
npm ERR! code EINTEGRITY
npm ERR! sha1-kwAtRjBP2LQvvhw56auEqJ8U4n8= integrity checksum failed when using sha1: wanted sha1-kwAtRjBP2LQvvhw56auEqJ8U4n8= but got sha1-gVsA/I03qh/watPfLgV08+bad48=. (15470 bytes)

npm ERR! A complete log of this run can be found in:
npm ERR!     /var/lib/jenkins/.npm/_logs/2017-06-09T17_59_45_368Z-debug.log

$ tail -n 20 /var/lib/jenkins/.npm/_logs/2017-06-09T17_59_45_368Z-debug.log
22958 silly extract [email protected]
22959 silly extract [email protected]
22960 silly extract [email protected]
22961 verbose unlock done using /var/lib/jenkins/.npm/_locks/staging-ca95f10f2868fd3f.lock for /var/lib/jenkins/tmp/traverse-api/node_modules/.staging
22962 verbose type OperationalError
22963 verbose stack Error: sha1-kwAtRjBP2LQvvhw56auEqJ8U4n8= integrity checksum failed when using sha1: wanted sha1-kwAtRjBP2LQvvhw56auEqJ8U4n8= but got sha1-gVsA/I03qh/watPfLgV08+bad48=. (15470 bytes)
22963 verbose stack     at Transform.<anonymous> (/var/lib/jenkins/.nvm/versions/node/v4.4.6/lib/node_modules/npmc/node_modules/ssri/index.js:275:19)
22963 verbose stack     at emitNone (events.js:72:20)
22963 verbose stack     at Transform.emit (events.js:166:7)
22963 verbose stack     at endReadableNT (_stream_readable.js:913:12)
22963 verbose stack     at nextTickCallbackWith2Args (node.js:442:9)
22963 verbose stack     at process._tickCallback (node.js:356:17)
22964 verbose cwd /var/lib/jenkins/tmp/traverse-api
22965 verbose Linux 4.4.23-31.54.amzn1.x86_64
22966 verbose argv "/var/lib/jenkins/.nvm/versions/node/v4.4.6/bin/node" "/var/lib/jenkins/.nvm/versions/node/v4.4.6/bin/npmc" "install"
22967 verbose node v4.4.6
22968 verbose npm  v5.0.3-canary.12
22969 error code EINTEGRITY
22970 error sha1-kwAtRjBP2LQvvhw56auEqJ8U4n8= integrity checksum failed when using sha1: wanted sha1-kwAtRjBP2LQvvhw56auEqJ8U4n8= but got sha1-gVsA/I03qh/watPfLgV08+bad48=. (15470 bytes)
22971 verbose exit [ 1, true ]

I would have expected that if the new shrinkwrap worked with the old npm version, it should work with the newest one, the one it was created with.

@ferrants
Copy link

ferrants commented Jun 9, 2017

@zkat , though, note that the shallow clone on unsupporting hosts issue does seem to have gone away

@zkat
Copy link
Contributor

zkat commented Jun 9, 2017

@ferrants I can't guarantee that legacy shrinkwraps are going to work with the new package-lock code. The best thing I can recommend is that you regenerate it.

@ferrants
Copy link

ferrants commented Jun 9, 2017

@zkat , I regenerated it with npmc, the new version. That is why I was alarmed when I attempted to npmc install with the same version that I generated the shrinkwrap file with.

@zkat
Copy link
Contributor

zkat commented Jun 10, 2017

@ferrants that does include not having either an npm-shrinkwrap.json or a node_modules generated by a legacy npm

@ferrants
Copy link

@zkat hm, I understand your point. how then should I ensure that I keep the same exact versions? if I npm install and then npmc install, will that update my node_modules directory to the npmc version while keeping the same dependency versions?

@dschinkel
Copy link

same issue, checksum errors #17152

@robwalch
Copy link

robwalch commented Jun 21, 2017

We're having this problem with node 8.1.0 and npm 5.0.3.

package.json has a dependency pointing to a pr ref

  "dependencies": {
    "jwplayer": "git+ssh://[email protected]/jwplayer/jwplayer.git#refs/pull/2124/head",

Call update on this package npm update jwplayer and we get this error

npm ERR! code 1
npm ERR! Command failed: /usr/bin/git checkout refs/pull/227/head
npm ERR! error: pathspec 'refs/pull/2124/head' did not match any file(s) known to git.
npm ERR! 

This kind of update works with npm 3 and 4. On npm 5 we can only pull from #refs/heads/master, but not the pull refspec defined in the current repo +refs/heads/*:refs/remotes/origin/* +refs/pull/*:refs/remotes/origin/pr/*.

I tried origin/pr/* too and no luck (that did not work in npm 3 or 4 for us either).

@amelnikoff
Copy link

@shubhamarora There's a new canary, [email protected], which includes some more git fixes. Mind giving it a whirl and seeing if things work well again for you? (with npm i -g [email protected] && npmc install)

@zkat for me that solved problem with 'Permission denied' on git clone.

@yanivel
Copy link

yanivel commented Jul 16, 2017

Does not work with [email protected]

Does not work with [email protected]

Does not work with [email protected] ( + node 8.1.4).

Works with [email protected].

@dhrn
Copy link

dhrn commented Jul 17, 2017

Getting this error :

0 info it worked if it ends with ok
1 verbose cli [ '/usr/bin/nodejs',
1 verbose cli   '/usr/bin/npm',
1 verbose cli   'install',
1 verbose cli   '--save',
1 verbose cli   'angular/material2-builds',
1 verbose cli   'angular/cdk-builds' ]
2 info using [email protected]
3 info using [email protected]
4 verbose npm-session 08e6c58d6f8090a3
5 silly install loadCurrentTree
6 silly install readLocalPackageData
7 silly fetchPackageMetaData error for github:angular/cdk-builds Command failed: /usr/bin/git clone --depth=1 -q -b master git://github.com/angular/cdk-builds.git /home/dhrn/.npm/_cacache/tmp/git-clone-8f0a06bd
7 silly fetchPackageMetaData /home/dhrn/.npm/_cacache/tmp/git-clone-8f0a06bd/.git: Permission denied
8 silly fetchPackageMetaData error for github:angular/material2-builds Command failed: /usr/bin/git clone --depth=1 -q -b master git://github.com/angular/material2-builds.git /home/dhrn/.npm/_cacache/tmp/git-clone-09eb2504
8 silly fetchPackageMetaData /home/dhrn/.npm/_cacache/tmp/git-clone-09eb2504/.git: Permission denied
9 verbose stack Error: Command failed: /usr/bin/git clone --depth=1 -q -b master git://github.com/angular/cdk-builds.git /home/dhrn/.npm/_cacache/tmp/git-clone-8f0a06bd
9 verbose stack /home/dhrn/.npm/_cacache/tmp/git-clone-8f0a06bd/.git: Permission denied
9 verbose stack
9 verbose stack     at ChildProcess.exithandler (child_process.js:270:12)
9 verbose stack     at emitTwo (events.js:125:13)
9 verbose stack     at ChildProcess.emit (events.js:213:7)
9 verbose stack     at maybeClose (internal/child_process.js:897:16)
9 verbose stack     at Socket.stream.socket.on (internal/child_process.js:340:11)
9 verbose stack     at emitOne (events.js:115:13)
9 verbose stack     at Socket.emit (events.js:210:7)
9 verbose stack     at Pipe._handle.close [as _onclose] (net.js:549:12)
10 verbose cwd /home/dhrn/Angular/firstmodule
11 verbose Linux 4.8.0-58-generic
12 verbose argv "/usr/bin/nodejs" "/usr/bin/npm" "install" "--save" "angular/material2-builds" "angular/cdk-builds"
13 verbose node v8.1.3
14 verbose npm  v5.3.0
15 error code 1
16 error Command failed: /usr/bin/git clone --depth=1 -q -b master git://github.com/angular/cdk-builds.git /home/dhrn/.npm/_cacache/tmp/git-clone-8f0a06bd
16 error /home/dhrn/.npm/_cacache/tmp/git-clone-8f0a06bd/.git: Permission denied
17 verbose exit [ 1, true ]

npm: 5.3.0
node: 8.1.3

@devony
Copy link

devony commented Jul 26, 2017

I also want to share my log--note this is in a github.com hosted repo, just using a hash. I'm running as root.
Fails on npm 5.2.0 and 5.3.0

root@jenkins-slave-04:/home/ubuntu# cat /root/.npm/_logs/2017-07-26T01_33_34_991Z-debug.log
0 info it worked if it ends with ok
1 verbose cli [ '/usr/local/bin/node',
1 verbose cli   '/usr/local/bin/npm',
1 verbose cli   'install',
1 verbose cli   '--global',
1 verbose cli   'git://github.com/blendlabs/npm-pkgr.git#088a8f436181b67aef7bb290185b82446626855b' ]
2 info using [email protected]
3 info using [email protected]
4 verbose npm-session a51b461de3c9ee7a
5 silly install loadCurrentTree
6 silly install readGlobalPackageData
7 silly fetchPackageMetaData error for git://github.com/blendlabs/npm-pkgr.git#088a8f436181b67aef7bb290185b82446626855b Command failed: /usr/bin/git clone --depth=1 -q -b master git://github.com/blendlabs/npm-pkgr.git /root/.npm/_cacache/tmp/git-clone-16d6e31f
7 silly fetchPackageMetaData fatal: could not create leading directories of '/root/.npm/_cacache/tmp/git-clone-16d6e31f': Permission denied
8 verbose stack Error: Command failed: /usr/bin/git clone --depth=1 -q -b master git://github.com/blendlabs/npm-pkgr.git /root/.npm/_cacache/tmp/git-clone-16d6e31f
8 verbose stack fatal: could not create leading directories of '/root/.npm/_cacache/tmp/git-clone-16d6e31f': Permission denied
8 verbose stack
8 verbose stack     at ChildProcess.exithandler (child_process.js:204:12)
8 verbose stack     at emitTwo (events.js:106:13)
8 verbose stack     at ChildProcess.emit (events.js:191:7)
8 verbose stack     at maybeClose (internal/child_process.js:891:16)
8 verbose stack     at Socket.<anonymous> (internal/child_process.js:342:11)
8 verbose stack     at emitOne (events.js:96:13)
8 verbose stack     at Socket.emit (events.js:188:7)
8 verbose stack     at Pipe._handle.close [as _onclose] (net.js:497:12)
9 verbose cwd /home/ansible
10 verbose Linux 3.13.0-96-generic
11 verbose argv "/usr/local/bin/node" "/usr/local/bin/npm" "install" "--global" "git://github.com/blendlabs/npm-pkgr.git#088a8f436181b67aef7bb290185b82446626855b"
12 verbose node v6.11.1
13 verbose npm  v5.2.0
14 error code 128
15 error Command failed: /usr/bin/git clone --depth=1 -q -b master git://github.com/blendlabs/npm-pkgr.git /root/.npm/_cacache/tmp/git-clone-16d6e31f
15 error fatal: could not create leading directories of '/root/.npm/_cacache/tmp/git-clone-16d6e31f': Permission denied
16 verbose exit [ 1, true ]

@shawn-mcginty
Copy link

shawn-mcginty commented Aug 1, 2017

Having this same exact problem as above with a github.com repo on ubuntu, works fine on my mac.

node v8.2.1
npm 5.3.0

@AstRonin
Copy link

AstRonin commented Aug 4, 2017

I got error like above

$ node -v
v6.10.3
$ npm -v
5.3.0

Unhandled rejection Error: Command failed: /usr/bin/git clone --depth=1 -q -b master git://github.com/andyinabox/FileSaver.js.git /home/ubuntu/.npm/_cacache/tmp/git-clone-37fbc4be
/home/ubuntu/.npm/_cacache/tmp/git-clone-37fbc4be/.git: Permission denied

    at ChildProcess.exithandler (child_process.js:204:12)
    at emitTwo (events.js:106:13)
    at ChildProcess.emit (events.js:191:7)
    at maybeClose (internal/child_process.js:886:16)
    at Socket.<anonymous> (internal/child_process.js:342:11)
    at emitOne (events.js:96:13)
    at Socket.emit (events.js:188:7)
    at Pipe._handle.close [as _onclose] (net.js:497:12)

tmp with 777 permissions

@eusonlito
Copy link

Same using Ubuntu, as regular user and as root:

root# npm update -g                        
                      
npm ERR! Error while executing:
npm ERR! /usr/bin/git ls-remote -h -t ssh://[email protected]/gulpjs/gulp-cli.git
npm ERR! 
npm ERR! fatal: failed to stat '.': Permission denied
npm ERR! 
npm ERR! exited with error code: 128

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2017-08-11T18_04_22_438Z-debug.log

root# tail -27 /root/.npm/_logs/2017-08-11T18_03_19_804Z-debug.log

173 silly install loadCurrentTree
174 silly install readGlobalPackageData
175 silly fetchPackageMetaData error for github:gulpjs/gulp-cli Error while executing:
175 silly fetchPackageMetaData /usr/bin/git ls-remote -h -t ssh://[email protected]/gulpjs/gulp-cli.git
175 silly fetchPackageMetaData
175 silly fetchPackageMetaData fatal: failed to stat '.': Permission denied
175 silly fetchPackageMetaData
175 silly fetchPackageMetaData exited with error code: 128
176 http fetch GET 200 https://registry.npmjs.org/node-inspector 43ms (from cache)
177 silly pacote version manifest for [email protected] fetched in 58ms
178 verbose stack Error: exited with error code: 128
178 verbose stack     at ChildProcess.onexit (/usr/local/lib/node_modules/npm/node_modules/mississippi/node_modules/end-of-stream/index.js:39:36)
178 verbose stack     at emitTwo (events.js:125:13)
178 verbose stack     at ChildProcess.emit (events.js:213:7)
178 verbose stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:200:12)
179 verbose cwd /root
180 verbose Linux 4.4.0-89-generic
181 verbose argv "/usr/local/bin/node" "/usr/local/bin/npm" "update" "-g"
182 verbose node v8.2.1
183 verbose npm  v5.3.0
184 error Error while executing:
184 error /usr/bin/git ls-remote -h -t ssh://[email protected]/gulpjs/gulp-cli.git
184 error
184 error fatal: failed to stat '.': Permission denied
184 error
184 error exited with error code: 128
185 verbose exit [ 1, true ]

root# node -v

v8.2.1

root# npm -v

5.3.0

@OshotOkill
Copy link

OshotOkill commented Aug 12, 2017

Having this problem while installing webpack-cli globally.

npm ERR! code 1
npm ERR! Command failed: /usr/local/bin/git clone --depth=1 -q -b Feature-getArgument git://github.com/ev1stensberg/generator.git /Users/norn/.npm/_cacache/tmp/git-clone-3e3ab4ce
npm ERR! /Users/norn/.npm/_cacache/tmp/git-clone-3e3ab4ce/.git: Permission denied
npm ERR! 

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/norn/.npm/_logs/2017-08-12T03_12_53_734Z-debug.log

MacOS 10.12.6
Node 8.3.0
npm 5.3.0

@ravikp7
Copy link

ravikp7 commented Aug 19, 2017

Having similar issue while installing a module that relies on git repo

ravi@ravi-Lenovo-G50-45:~/node-beagle-boot$ sudo npm install
npm ERR! code 1
npm ERR! Command failed: /usr/bin/git clone --depth=1 -q -b 1.3.0 git://github.com/tessel/node-usb.git /home/ravi/.npm/_cacache/tmp/git-clone-7de5e8bf
npm ERR! /home/ravi/.npm/_cacache/tmp/git-clone-7de5e8bf/.git: Permission denied
npm ERR!

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/ravi/.npm/_logs/2017-08-19T06_29_43_900Z-debug.log

Node 8.4.0
npm 5.3.0

@GiedriusM
Copy link

It's still relevant.

$ sudo npm install -g
npm ERR! code 128
npm ERR! Command failed: /usr/bin/git clone --depth=1 -q -b master https://github.com/mcollina/node-coap.git /root/.npm/_cacache/tmp/git-clone-cd8d9f52
npm ERR! fatal: could not create leading directories of '/root/.npm/_cacache/tmp/git-clone-cd8d9f52': Permission denied
npm ERR!

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2017-09-18T08_20_56_607Z-debug.log

$ npm -v
5.4.2

Workaround - npm i -g [email protected] && npmc install and then sudo npm install -g

@bellalMohamed
Copy link

@yanivel Downgrading for npm 3.10.10 worked for me thanks :D

@mattmacadams
Copy link

Running this resolved this issue for me:
cd /root/
sudo npm i -g update

@dhrn
Copy link

dhrn commented Oct 10, 2017

sudo rm -rf node_modules/
npm install
This issue resolved for me

@zsolt-szabo
Copy link

zsolt-szabo commented Oct 13, 2017

EDIT: With a work around that seems to push though the permissions issue
location=`pwd`;sudo su - -c "cd ${location};npm install npm cache clean --unsafe-perm"

Interestingly I have noticed that this error happens when I run
sudo npm install npm cache clean --unsafe-perm

Command failed: /usr/bin/git clone --depth=1 -q -b master git://github.com/sleeplessinc/ds.git /root/.npm/_cacache/tmp/git-clone-79e49909
npm ERR! /root/.npm/_cacache/tmp/git-clone-79e49909/.git: Permission denied

but if I do
sudo su -
then switch to the directory and run the command again it works.

@omtinez
Copy link

omtinez commented Oct 14, 2017

Thanks for the workaround @zsolt-szabo. It appears that the following combination is still broken:

  1. Running npm install with sudo
  2. Passing --unsafe-perm flag
  3. Package hosted in Git repo

Switching to root user and then running npm install [...] bypasses the problem, but ideally this should be fixed by npm.

@charginghawk
Copy link

I've been encountering this issue for a few months at seemingly random. The heart of the issue was here:

zkat/pacote#105

Basically, I use root to run my build, and afaict due to npm's uid/guid switching it creates a tmp directory using root but doesn't change permissions so then it can't clone into it. I'd run into the issue, run npm install as non-root, and that'd set the cache so the problem would disappear. It wouldn't show up until I added a new git dependency and tried to build.

Anyway, the fix was to not use root to run the build.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests