Expand documentation for importlib.metadata custom providers to include caveats regarding backports

[jaraco]

Documentation

As importlib_metadata provides a backport of importlib.metadata and asserts primacy when imported, for consumers of the API, it provides a consistent interface, except when a custom provider serves objects as imported from importlib.metadata (not using the backport). For example:

• project A implements and installs a DistributionFinder to sys.meta_path providing importlib.metadata.Distribution instances.
• project B calls importlib_metadata.distributions() to return all distributions, including that of project A.
• A user installs project A and project B.
• When project B queries for the distributions, expecting it to return Iterable[importlib_metadata.Distribution] objects, their expectation is violated when the DistributionFinder installed by project A emits importlib.metadata.Distribution objects.

python/importlib_metadata#486 captures this concern. python/importlib_metadata#505 promises to correct this concern by wrapping and transforming any objects from custom providers referencing importlib.metadata to instead use objects from importlib_metadata.

However, for importlib.metadata.Distribution objects, it's not possible to replace safely the objects of an arbitrary custom provider, so the proposed implementation emits a warning when such an object is encountered.

Therefore, we'd like to add some documentation to the custom provider docs to advise custom providers on what to do to avoid their users encountering the warning. In particular:

• Custom providers should emit importlib_metadata.Distribution objects when importlib_metadata is present, or
• Custom providers should register their objects with importlib_metadata to bypass the warning.

Linked PRs

• gh-123976: Refresh docs around custom providers. #123977

[jaraco]

As I write that recommendation, I'm already uncomfortable with the recommendation. It's unclear what "when importlib_metadata is present" means. Does it mean the module is available for import? Does it mean it's been imported previously (e.g. in sys.modules)?

I think it has to mean it's available for import, because the provider can't tell if it will later be imported by a consumer. But that also means the provider might activate importlib_metadata in an environment that would otherwise have never had it activated. Perhaps that's not a huge concern - if it's installed, it's likely to be activated.

That also means that users of importlib.metadata will be getting importlib_metadata.Distribution objects, violating the type spec. But that's the case already today, so perhaps that's not so bad.

Also, "register with importlib_metadata to bypass the warning" sounds messy at best. There probably aren't a lot of providers out there, but even so, having to make new releases that are required to support their custom provider does not sound like a lot of fun for anyone.

[jaraco]

All of this has me re-thinking the strategy.

Maybe instead of having two distinct class trees (importlib.metadata.Distribution and importlib_metadata.Distribution and subclasses), there should be a protocol that's shared by both versions of the library, explicitly modeling what's currently implicit - that there's an interface expected by Distribution objects (also PackagePath and Message objects), and those protocols are what the two libraries are coded against.

[hugovk]

Triage: #123977 has been merged, ready to close this?

[jaraco]

Thanks, but no. The PR only did some minor refreshing of the docs. The core concern of this issue remains.