Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] npm v7 handles resolved urls in shrinkwrap files differently than v6. #3783

Open
1 task done
everett1992 opened this issue Sep 22, 2021 · 2 comments
Open
1 task done

[BUG] npm v7 handles resolved urls in shrinkwrap files differently than v6. #3783

everett1992 opened this issue Sep 22, 2021 · 2 comments
Labels
Bug thing that needs fixing Release 8.x work is associated with a specific npm 8 release

Comments

@everett1992
Copy link

everett1992 commented Sep 22, 2021 •
edited
Loading

Is there an existing issue for this?

  • I have searched the existing issues

Current Behavior

Some packages in the npm registry use a npm-shrinkwrap with resolved urls to other registries. For example [email protected]

$ curl -sS "$(npm info [email protected] dist.tarball)" | tar xzO package/npm-shrinkwrap.json | rg resolved | head -n 1
      "resolved": "https://registry.yarnpkg.com/@jsii/check-node/-/check-node-1.33.0.tgz#55d75cbef1c84e2012c67ab8d6de63f773be4a9b",

With npm v6 the resolved url appears to be ignored and npm uses the configured registry instead. With v7 npm makes requests to registry.yarnpkg.com. We run a private registry and network isolate our builds so requests to yarnpkg are failing.

Expected Behavior

I can understand how this is a feature, but I'm not sure it was intended. In general I think the behavior of resolved urls in lock and shrinkwraps needs to be better documented and configurable.

I'll take a pass at documentation based on this discussion

Steps To Reproduce

With npm v6 I notice that their resolved url is ignored and packages are fetched from the configured registry.

$ npm -v
6.14.13
# Without package-lock
$ rm node_modules ~/.npm/_cacache -rf
$ npm install [email protected] --verbose |& rg 'yarnpkg'
# No match

But with v7 the shrinkwrap resolved url is respected.

$ npm -v
7.21.0
$ rm node_modules ~/.npm/_cacache -rf
$ npm install [email protected] --verbose |& rg 'yarnpkg'
npm http fetch GET 200 https://registry.yarnpkg.com/y18n/-/y18n-5.0.8.tgz#7f4934d0f7ca8c56f95314939ddcd2dd91ce1d55 762ms (cache miss)
...

Environment

  • OS: Ubuntu
  • Node: 16.8.0
  • npm: 7.21.0
@everett1992 everett1992 added Bug thing that needs fixing Needs Triage needs review for next steps Release 7.x work is associated with a specific npm 7 release labels Sep 22, 2021
everett1992 pushed a commit to everett1992/aws-cdk that referenced this issue Sep 22, 2021
fix: use registry.npmjs.com to fix shinkwrap resolves
b5f0ac2 
Use the npm registry instead of yarns mirror.
npm treats registry.npmjs.org as a special value that means 'the current
configured package' in package-lock and npm-shrinkwrap.  if we use
registry.yarnpkg.com in our shrinkwrap then users with a custom registry
will be forced to registry.yarnpkg.com.
npm/cli#3783

I updated yarn.lock with sed instead of running `yarn install` and
checking in the generated lockfile because there were other changes in
the file. `yarn install` in a clean package also showed a diff.

```
sed 's|https://registry.yarnpkg.com|https://registry.npmjs.org|' yarn.lock -i
```
@everett1992 everett1992 mentioned this issue Sep 22, 2021
Merged
everett1992 pushed a commit to everett1992/documentation that referenced this issue Sep 22, 2021
Document special meaning of registry.npmjs.com
ece764e 
This behavior has been present in npm for a while, but I haven't found
where it's documented. This is my attempt at documenting the behavior
based on my understanding of it. I think a SME should contribute to this
so the documentation is correct.

npm/feedback#544
https://github.com/npm/arborist/blob/478871bf0a44a8ec516b9057585b8707e60b0349/lib/arborist/reify.js#L687-L693
npm/cli#3783
@everett1992 everett1992 mentioned this issue Sep 22, 2021
Closed
everett1992 pushed a commit to everett1992/cli that referenced this issue Sep 22, 2021
docs: document special meaning of registry.npmjs.com
ca029d4 
This behavior has been present in npm for a while, but I haven't found
where it's documented. This is my attempt at documenting the behavior
based on my understanding of it. I think a SME should contribute to this
so the documentation is correct.

npm/feedback#544
npm#3783
https://github.com/npm/arborist/blob/478871bf0a44a8ec516b9057585b8707e60b0349/lib/arborist/reify.js#L687-L693
@everett1992 everett1992 mentioned this issue Sep 22, 2021
Merged
everett1992 pushed a commit to everett1992/cli that referenced this issue Sep 22, 2021
docs: document special meaning of registry.npmjs.com
e19a2f5 
This behavior has been present in npm for a while, but I haven't found
where it's documented. This is my attempt at documenting the behavior
based on my understanding of it. I think a SME should contribute to this
so the documentation is correct.

npm/feedback#544
npm#3783
https://github.com/npm/arborist/blob/478871bf0a44a8ec516b9057585b8707e60b0349/lib/arborist/reify.js#L687-L693
wraithgar pushed a commit to everett1992/cli that referenced this issue Sep 23, 2021
@wraithgar
docs: document special meaning of registry.npmjs.com
dae5ce3 
This behavior has been present in npm for a while, but I haven't found
where it's documented. This is my attempt at documenting the behavior
based on my understanding of it. I think a SME should contribute to this
so the documentation is correct.

npm/feedback#544
npm#3783
https://github.com/npm/arborist/blob/478871bf0a44a8ec516b9057585b8707e60b0349/lib/arborist/reify.js#L687-L693

PR-URL: npm#3784
Credit: @everett1992
Close: npm#3784
Reviewed-by: @wraithgar
@johndiiorio
Copy link

I am also seeing that this option changed between v6 and v7. Is there any way in v7, for example through a config setting, to override the resolved urls to instead use a different registry?

everett1992 pushed a commit to everett1992/aws-cdk that referenced this issue Sep 29, 2021
fix: use registry.npmjs.com to fix shinkwrap resolves
84492a7 
Use the npm registry instead of yarns mirror.
npm treats registry.npmjs.org as a special value that means 'the current
configured package' in package-lock and npm-shrinkwrap.  if we use
registry.yarnpkg.com in our shrinkwrap then users with a custom registry
will be forced to registry.yarnpkg.com.
npm/cli#3783

I updated yarn.lock with sed.

```
sed 's|https://registry.yarnpkg.com|https://registry.npmjs.org|' yarn.lock -i
```
everett1992 pushed a commit to everett1992/aws-cdk that referenced this issue Oct 11, 2021
fix: use registry.npmjs.com to fix shinkwrap resolves
77901ae 
Use the npm registry instead of yarns mirror.
npm treats registry.npmjs.org as a special value that means 'the current
configured package' in package-lock and npm-shrinkwrap.  if we use
registry.yarnpkg.com in our shrinkwrap then users with a custom registry
will be forced to registry.yarnpkg.com.
npm/cli#3783

I updated yarn.lock with sed.

```
sed 's|https://registry.yarnpkg.com|https://registry.npmjs.org|' yarn.lock -i
```
everett1992 pushed a commit to everett1992/aws-cdk that referenced this issue Oct 11, 2021
fix: use registry.npmjs.com to fix shinkwrap resolves
0e7b804 
Use the npm registry instead of yarns mirror.
npm treats registry.npmjs.org as a special value that means 'the current
configured package' in package-lock and npm-shrinkwrap.  if we use
registry.yarnpkg.com in our shrinkwrap then users with a custom registry
will be forced to registry.yarnpkg.com.
npm/cli#3783

I updated yarn.lock with sed.

```
sed 's|https://registry.yarnpkg.com|https://registry.npmjs.org|' yarn.lock -i
```
mergify bot pushed a commit to aws/aws-cdk that referenced this issue Oct 13, 2021
@everett1992
fix: use registry.npmjs.com to fix shinkwrap resolves (#16607)
8f91531 
npm treats registry.npmjs.org as a special value that means 'the current configured package' in package-lock and npm-shrinkwrap. npm will request aws-cdk's dependencies from yarnpkg instead of from the installers configured registry because aws-cdk's shrinkwrap uses yarnpkg. This behavior seems new to [npm v7]. It causes issues for us because we run our builds with a isolated network and a private registry.

[npm v7]: npm/cli#3783

This commit changes the registry from yarnpkg to npmjs. I updated yarn.lock with sed.

```
sed 's|https://registry.yarnpkg.com|https://registry.npmjs.org|' yarn.lock -i
```
Alternatively we could modify the yarn-cling tool to replace the registry. [registry.yarnpkg.com is a cname for registry.npmjs.org](https://yarnpkg.com/getting-started/qa#why-registryyarnpkgcom-does-facebook-track-us) so changing to registry.npmjs.org shouldn't affect available packages or performance.

```
dig registry.yarnpkg.com | rg CNAME
registry.yarnpkg.com.	300	IN	CNAME	yarn.npmjs.org
```

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
TikiTDO pushed a commit to TikiTDO/aws-cdk that referenced this issue Feb 21, 2022
@everett1992 @TikiTDO
fix: use registry.npmjs.com to fix shinkwrap resolves (aws#16607)
4c766eb 
npm treats registry.npmjs.org as a special value that means 'the current configured package' in package-lock and npm-shrinkwrap. npm will request aws-cdk's dependencies from yarnpkg instead of from the installers configured registry because aws-cdk's shrinkwrap uses yarnpkg. This behavior seems new to [npm v7]. It causes issues for us because we run our builds with a isolated network and a private registry.

[npm v7]: npm/cli#3783

This commit changes the registry from yarnpkg to npmjs. I updated yarn.lock with sed.

```
sed 's|https://registry.yarnpkg.com|https://registry.npmjs.org|' yarn.lock -i
```
Alternatively we could modify the yarn-cling tool to replace the registry. [registry.yarnpkg.com is a cname for registry.npmjs.org](https://yarnpkg.com/getting-started/qa#why-registryyarnpkgcom-does-facebook-track-us) so changing to registry.npmjs.org shouldn't affect available packages or performance.

```
dig registry.yarnpkg.com | rg CNAME
registry.yarnpkg.com.	300	IN	CNAME	yarn.npmjs.org
```

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
@everett1992
Copy link
Author

I've written an RFC and pull request implementing option to ignore resolved values in lockfiles that would fix this bug.

npm/rfcs#486
#4264

@wraithgar wraithgar added Release 8.x work is associated with a specific npm 8 release and removed Needs Triage needs review for next steps Release 7.x work is associated with a specific npm 7 release labels Mar 17, 2022
@fritzy fritzy removed their assignment Feb 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Bug thing that needs fixing Release 8.x work is associated with a specific npm 8 release
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@wraithgar @fritzy @everett1992 @johndiiorio