feat(git): add support for ::subdir in #committish

[sspiff]

More to do (docs and test, at least), but thought I'd pause for feedback:

What / Why

For git URLs, allow the specification of a subdirectory within the given repository via the hash portion of the URL using the syntax: #[COMMITTISH][::SUBDIR]. If SUBDIR is specified, the result object will have a new key gitSubdir with the value /SUBDIR, which can then be consumed by e.g. pacote (see References below for corresponding pacote changes). The result object's gitCommittish and gitRange keys are unchanged from current behavior.

This feature will help npm provide improved support for "monorepos" or any git repository in which the package.json is not at the repository root.

References

• Related to Allow subdirectories within git repos in npm install  npm#2974, [FEATURE] Allow adding Git branch dependencies in a subdirectory cli#528
• Corresponding pacote support (latest): sspiff/pacote@8bd5461
• Corresponding pacote support (v9): sspiff/pacote@5cb8739

[mikemimik]

Hey @sspiff just wanted to let you know that we're taking a look at this :D

[sspiff]

Thanks for the update!

Note that I'm not wedded to the ::SUBDIR syntax; it was suggested previously by the community and it seemed the path of least resistance in implementation (albeit at the expense of overloading #COMMITTISH). I did investigate using query parameters as well as just extending the path, but these routes would (I think) also require changes to hosted-git-info and it wasn't clear to me that they would fit the intended scope of that module.

[mikemimik]

Added the Agenda label. We're going to chat about this in our Open RFC call tomorrow. I'll link the issue when it gets created today.

[sspiff]

Are these open RFC meetings something that I can/should attend? I noticed in the meeting minutes that this PR was not discussed. Just wondering if I need to join in order to have it covered? In any case, apologies for not understanding the process.

[prmichaelsen]

There's strong community support for such a feature. I know many from this thread would appreciate it: npm/npm#2974

[isaacs]

summoning @coreyfarrell to this issue.

This is an easy one to accept. We just have to write down a spec over on https://github.com/npm/rfcs so that it's documented, and then get it implemented in this repo.

If :: is a valid character in a branch name, then that won't work. Another option I've heard suggested is # for the committish and another # for the path. (So, if you want to specify a path on the default branch, it'd be git://repo##some/path.)

EDIT: scratch that, :: is better.

$ git checkout -b 'hash##hash'
Switched to a new branch 'hash##hash'
$ git checkout -b 'paamayim::nekudotayim'
fatal: 'paamayim::nekudotayim' is not a valid branch name.

[coreyfarrell]

Colons are not valid branch names (single or double colon would work). I think a single colon makes sense as it reminds me of git show master:path/to/file.js which displays path/to/file.js from the master branch. So:
git://github.com/istanbuljs/istanbuljs#commitish:packages/istanbul-lib-instrument

Or for the default branch:
git://github.com/istanbuljs/istanbuljs#:packages/istanbul-lib-instrument

Use of # needs to be mandatory when specifying a monorepo directory to avoid ambiguity between the path to the repository and the path within the repository and to allow use of the standard new URL to parse the string (just have to split url.hash on the first ':').

[isaacs]

I think a single colon makes sense as it reminds me of git show master:path/to/file.js which displays path/to/file.js from the master branch.

This is an amazing git trick that I just learned about now. Thanks!

Also, sold, let's do a single colon!

[coreyfarrell]

Discussed this further with @isaacs on Slack yesterday, single colon will not work due to git://github.com/user/repo#semver:v1.x syntax. The idea we came to is to support multiple name value pairs after the # and specifically support path:packages/monorepo-dir for subdir selection. Some examples of valid GIT install URIs would be:

git://github.com/user/repo#master::path:subdir
git://github.com/user/repo#path:subdir::master
git://github.com/user/repo#path:subdir
git://github.com/user/repo#semver:v1.x
git://github.com/user/repo#semver:v1.x::path:subdir
git://github.com/user/repo#path:subdir::future:ignored

So the algorithm would be to take new URL(str).hash.slice(1), split on ::. Then for each :: separated item:

• If the item has no : then it is a commit-ish
• If the item has : then split into name and value.
  1. If the name is semver then do semver lookup of ref or tag
  2. If the name is path then use the value as the subdir to install from.
  3. If the name is unknown then warn that the name-value pair is being ignored.

This loop should error if duplicate values are found. So specifying a commit-ish and semver:range would be an error, or specifying path twice. I lean towards NOT having a hard error if unknown values are found twice, assume some future key might support an array of values. So I think having ignore:path1::ignore:path2 would not error but would cause 2 warnings?

One comment @isaacs didn't respond to my idea of warning about items of unknown name but he did agree that a hard error for unknown names would be unnecessary. Just wanted to make sure I don't claim endorsement of the warning on his behalf.

[sspiff]

I'm happy to continue work on this, but I'm not sure what the next steps should be. Should a spec be submitted to https://github.com/npm/rfcs before refining any implementation?

[prmichaelsen]

@isaacs @coreyfarrell I echo sspiff's comment. Should we submit an rfc?

[kerryj89]

This would be amazing. Any further talks outside of here about implementing this?

[wottpal]

Is merging this feature still planned? This would be so nice for monorepos (e.g. Gatsby.js) ❤️.

[tilgovi]

Might be nice to align with Yarn v2 on this: https://yarnpkg.com/features/protocols#git

[tilgovi]

https://yarnpkg.com/features/protocols#git

No arbitrary paths, but you can express a workspace to depend on.

[wraithgar]

This landed in #91