[Replace #2507] Support VSTS and other custom registries with unusual feeds

[lumaxis]

This is a continuation of #2507 where the progress has recently stalled. Fixes #2505.

Quick Summary

Visual Studio Team Services (VSTS) unfortunately produces package feed URLs that look like https://account.pkgs.visualstudio.com/_packaging/feed-name/npm/registry but the URLs to fetch the actual tars then point to https://account.pkgs.visualstudio.com/_packaging/a917cf10-dfe4-4781-b0d1-105df772da02/npm/registry/@scope/package-name/-/package-name-0.0.1.tgz.

This Pull Request aims to fix this by picking up a suggestion from #2507 by @bestander and introduces a new .npmrc option called custom-host-suffix. That options allows to skip a check in the form of requestPath.startsWith(registryPath) if the request's URL contains the value of custom-host-suffix.

Tests

I tried to add tests as best as I could but had trouble coming up with a good way of integration testing this change as I couldn't find an easy way to set up a package feed on VSTS that is publicly accessible. Is there a precedence for using shared credentials in the test suite? Maybe that would be an option.
I'll appreciate any advice for better tests 👍

[lumaxis]

Would be great to get some feedback on this! 🙂

[bestander]

Great job, thanks for patience, @lumaxis.

[bestander]

@lumaxis, now that we have this feature in Yarn it would be great for people to know about it.
Would you write a post https://github.com/yarnpkg/website/tree/master/_posts how it can be used?

[xnnkmd]

Has this feature been released in a public downloadable yarn ? I so, which version?

[bestander]

Not yet, will be released in 0.26

[lumaxis]

@bestander I think I should be able to put something together within the week 🙂 Anything special I should look out for?

[bestander]

Cool!
Nothing special, just spread the word about the cool feature :)

[lumaxis]

@bestander There you go 🙂

[bestander]

@lumaxis, this change might have caused #3765.
We have a tentative quick fix baking here #3774.

Apparently we don't have tests covering neither private repos nor vsts repo and I am concerned that change in one might break the other.
Can you chime in the review of #3774?
And let's think more about setting up e2e tests, I think the best way would be to create an account with read access to one private package and store the credential on the CI.
Would you help us out with that?

[lumaxis]

@bestander Oh dear, that sounds terrible! Very sorry to maybe have caused that 😞
I was kind of expecting that private repos were covered by either unit or integration tests already, so didn't check that explicitly ...
I'll take a look at #3774 and also check if and how we could set up an E2E test.

[bestander]

Thanks, Lukas. Very much appreciated, it could be not your change after all but it seems to touch related code. It is our (mine) laziness to blame though, we should have set up e2e tests for private packages long ago.

…

On Fri, Jun 30, 2017 at 12:48 PM Lukas Spieß ***@***.***> wrote: @bestander <https://github.com/bestander> Oh dear, that sounds terrible! Very sorry to maybe have caused that 😞 I was kind of expecting that private repos were covered by either unit or integration tests already, so didn't check that explicitly ... I'll take a look at #3774 <#3774> and also check if and how we could set up an E2E test. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3231 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACBdWMZ5l3Wt5F52iy4MtAKPHVHE8i7dks5sJVDxgaJpZM4NFNVM> .

[Mardoxx]

We had an issue where we would get 401 with VSTS npm feed, yielded the following error for yarn install:

[1/4] Resolving packages...
verbose 1.275 Performing "GET" request to "https://companyname.pkgs.visualstudio.com/_packaging/FeedName/npm/body-parser".
verbose 1.465 Request "https://companyname.pkgs.visualstudio.com/_packaging/FeedName/npm/body-parser" finished with status code 401.
verbose 1.466 Error: Couldn't find package "body-parser" on the "npm" registry.
    at C:\Program Files (x86)\Yarn\lib\cli.js:48169:15
    at Generator.next (<anonymous>)
    at step (C:\Program Files (x86)\Yarn\lib\cli.js:92:30)
    at C:\Program Files (x86)\Yarn\lib\cli.js:103:13
    at <anonymous>
    at process._tickCallback (internal/process/next_tick.js:169:7)
error Couldn't find package "body-parser" on the "npm" registry.
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.

.npmrc was like this

registry=https://companyname.pkgs.visualstudio.com/_packaging/FeedName/npm/registry
always-auth=true

setting the URL to https://companyname.pkgs.visualstudio.com/_packaging/FeedName/npm/registry/ i.e. with the trailing slash made it work fine.

VSTS have updated the URL generated in the instructions so it now produces them with the trailing slash. ~~~So this should not be a problem in the future, but this tripped us up this morning. Not sure why this would be an issue. Worth noting that it works fine with, and without, trailing slash with npm install.~~~

---

Just realised what the actual problem was, will open a separate issue #4413