Explicitly add stream to cache on cache add

[mjsir911]

Pacote doesn't do this automatically anymore

Fixes npm/pacote#73 & Fixes #2160

[isaacs]

This is a little bit complicated.

Pacote will tee things into the cache if it's a remote fetch. If it's local, you're right, since the rewrite it just says "oh, you want a tarball? This is a tarball. Here you go!" and saves the extra work of copying the file from one place on disk to another place on disk.

So, two changes and one question. The changes:

1. It should only explicitly pipe into cacache if it's a file: or git specifier. If it's a registry, remote, or alias spec, don't bother. (This can be determined by using npm-package-arg to parse the specifier. const npa = require('npm-package-arg'); spec = npa(spec); if (spec.type === 'remote' || spec.registry) { just return the stream } else { pipe to cacache }
2. Cacache's folder is not just npm.config.get('cache'). We tack a _cacache path portion onto it. So you'll have to do something like the const cachePath = path.join(this.npm.cache, '_cacache') that appears elsewhere in the file, and pass that to cacache instead.

The question I have is, why do you need to add a file to the cache which is already present on disk? I'm curious about your use case.

The answer may well be that we should have an option to pacote to say "no, really, pipe this into the cache even if you don't think you have to", because the CLI should ideally remain somewhat blissfully unaware of exactly how various types of packages get handled. That is pacote's job.

[mjsir911]

Thank you for the response @isaacs!

The question I have is, why do you need to add a file to the cache which is already present on disk? I'm curious about your use case.

My use case is *absolutely* not intended nor supported within npm, but I would like to install packages without connecting to the internet. Specifically, gentoo's package manager pre-downloads all sources and sandboxes internet access during build-time, so using npm becomes a bit complicated. There are some workarounds floating around on the internet but adding all the dependency tarballs to a cache and then doing `npm ci` with an existing `package-lock.json` file gets me 99% of the way there. Unfortunately, `npm cache add` only accepts one package to cache at a time so a bash loop with 1000 packages takes a bit of time :(. I have a local patch to allow it to accept multiple packages at once but that breaks existing behaviour.

Pacote _will_ tee things into the cache _if_ it's a remote fetch. If it's local, you're right, since the rewrite it just says "oh, you want a tarball? This is a tarball. Here you go!" and saves the extra work of copying the file from one place on disk to another place on disk. So, two changes and one question. The changes: 1. It should only explicitly pipe into `cacache` if it's a `file:` or `git` specifier. If it's a registry, remote, or alias spec, don't bother. (This can be determined by using `npm-package-arg` to parse the specifier. `const npa = require('npm-package-arg'); spec = npa(spec); if (spec.type === 'remote' || spec.registry) { just return the stream } else { pipe to cacache }` 2. Cacache's folder is not just `npm.config.get('cache')`. We tack a `_cacache` path portion onto it. So you'll have to do something like the `const cachePath = path.join(this.npm.cache, '_cacache')` that appears elsewhere in the file, and pass that to cacache instead.

Thank you for the clarifications & direction, will get on implementing these fixes (Not so sure about point 2, if that were true that would be breaking things for me but everything is working as expected, will double check this)

The answer may well be that we should have an option to pacote to say "no, really, pipe this into the cache even if you don't think you have to", because the CLI should ideally remain somewhat blissfully unaware of exactly how various types of packages get handled. That is pacote's job.

Not that is has any bearing on the patchset, but I disagree. Currently, pacote doesn't have any responsibilities caching the handled packages, and is implicitly relying on `make-fetch-happen` to cache downloaded tarballs. pacote < v10 implicitly did have these responsibilities, but I assume was removed for a good reason or maybe the use case just wasn't realized. Having pacote manually add to cache when it expects a package not to already have been added to the cache is just another recipe for broken assumptions somewhere down the line. My uneducated recommendation would be, *if* you want pacote to handle caching the source-independent tarball, tell `fetch` not to cache it on download so the caching is still only handled & done in one place: `pacote` I'm still partial to explicitly saying `cacache.put` in npm-cli's `cache add`, but I understand your thought process and my opinions aren't very strong :-)

…

-- Marco Sirabella

[mjsir911]

With the new changes I agree that this logic probably shouldn't be in npm-cli, feel free to close & I'll take a look at hacking onto pacote