Speculative idea - Cache metadata for sdists?

[pfmoore]

What's the problem this feature will solve?
In this discussion, @uranusjr said:

One of the most resource-consuming (both for development and runtime) part of dependency resolution is the need to download a package matching the host environment, extract it, and potentially build it to get dependencies.

Describe the solution you'd like
Thinking about this, pip could cache metadata for sdists, much like the wheel cache. When we prepare a sdist to get metadata, we could save that metadata to a local cache, keyed by project name and version. Then, future runs could use the cache rather than downloading and building the sdist again.

Alternative Solutions
The wheel cache provides similar data, so we could check the wheel cache when looking for sdist metadata. However:

1. We'd have to make sure we picked the right wheel. Wheels for other platforms/architectures could be in the cache, and (much as we'd prefer it if this wasn't the case) metadata isn't currently guaranteed to be consistent across platforms.
2. Not all sdists get built to wheels. Some will potentially get prepared, the metadata checked, and then discarded. So caching sdist metadata directly will help more cases.

Additional context
As @uranusjr pointed out, the biggest benefits will be to platforms like musl which don't have wheels. However, a common musl platform is Docker alpine images, and as Docker containers are throwaway, a cache won't help much here. So we should make sure the benefits will be achieved in practice before investing time into this solution.

[sbidoul]

We'd have to make sure we picked the right wheel. Wheels for other platforms/architectures could be in the cache

Does the wheel cache adequately cope with that via supported pep 425 tags?

Could it be an alternative approach to leverage the wheel cache, and extend it to support "partial" wheels containing metadata only? These would basically be wheels containing the outcome of PEP 517 prepare_metadata_for_build_wheel only. For legacy non PEP 517 sdists, it might be possible to easily generate such partial wheels with the outcome of generate_metadata.

[pfmoore]

Maybe...? Those certainly sound like good things to look at when we look at how to implement this.

[pradyunsg]

I'm on board in principle. :)

Here's a significantly outdated, and poorly worded variant of the same idea: https://gist.github.com/pradyunsg/5cf4a35b81f08b6432f280aba6f511eb#stage-2