Storing "installed extras" metadata in installations

[pradyunsg]

Filing an issue for discussion on https://mail.python.org/archives/list/[email protected]/thread/MDZ35APWKARAYH6I5K7K7SJM2NJ2I5P7/#KQGEPFD22X3KCYDVOPIBARKXFYZY4U2R

Relavant: @njsmith had an older proposal that covers much more ground and is more general with some additional complexity - https://mail.python.org/pipermail/distutils-sig/2015-October/027364.html

[njsmith]

Well, one could argue that that proposal will actually reduce complexity, because if extras are just packages with funny names then we can throw away most of the code for handling them :-).

Of course it gets more complicated when you include transition costs.

[pradyunsg]

We discussed this at the packaging summit.

@techalchemy and I will be writing a PEP for this, with (likely lots of) help from @ncoghlan and @njsmith.

[brainwane]

Notes from the packaging minisummit at PyCon last month (thanks to @bskinn and @crwilcox for notes I am transferring here):

Overview

When you install a package with extra, e.g. requests with security. Pip will install these extras but nothing is saved in the metadata that those packages were installed. A few options for implementation:

• (simpler but less powerful) Add an item into the .dist-info directory
• (more complex but more powerful) Add a new "package requests[security] with its own .dist-info directory

njs has a proposal; PEP draft from 2015

Discussion

• Paraphrase NJS: extras are weird → don’t fit the resolver model. Who requested the extras? What should resolver do about them?
• Possibility of having multiple wheels which are compiled in a different manner
• Possibility of having multiple entry points depending on extras (or, entry_points only on the extras?)
• Setting this up only for extras allows “more trusty(?)” interactions with provides, etc. → a package controls its own extras, but can’t control aspects that would be delegated to external packages (notetaker said: not sure he captured this accurately)
• Conflicts and Provides (but probably not Obsoletes?) metadata entries
• Constraints (numpy[mkl] + scipy[mkl])
• Virtual packages to describe features
  • Implementing as virtual packages instead of plain extras would solve the extras problem while also giving a much more powerful tool
  • Not really feasible to embed the versions of features that are provided into a (PEP440?) versioned package name -- can’t easily say “provides v1 and v3, but not v2 or v4”
• Namespaces for virtual packages
  • There is the problem of the PyPI namespace -- may need to provide sub-namespaces?
• Need default vs. non-default
  • soft requirements
  • concrete vs. virtual
• Can some of this provide accelerated and non-accelerated variants of packages?
  • Either warn if a non-accelerated version is installed; or, require e.g. --force in order to install non-accelerated version (avoid users/customers observing degraded performance without being made aware upon install of why that degradation should be expected)
• pipenv has ‘this sort of functionality’ (indicating versions of extras?) (notetaker said he wasn't sure what this was referring to and asked more knowledgable people to fix notes)
• Conda has a sat solver
• Specific implementation aspects
  • Consider using a default, as Debian does (default as fallback if no virtuals?) (notetaker wasn't sure he caught this correctly)
  • Soft requirements possibly helpful for some cases
  • Consider something like a --use flag if desire a virtual(?) to be applied for all packages

Action Items

• Write a solver
• Draft a PEP for installation database (Dan, Nathaniel, Pradyun)

[techalchemy]

Thanks for carrying this forward, I've had several offline conversations about this with @njsmith, @pradyunsg, @ncoghlan, and @jjhelmus (conda) that I will try to document in the near future, but I think it essentially boiled down to:

• Conda has an implementation of something like a metapackage, Jonathan had some insight about the constraints they have run into on that front
• Lots of people want to stay away from the debian model of having "provides" metadata
• The easiest approach seemed to be treating package+extras as a separate first class (but possibly child) artifact which functions as a metapackage (i.e. one that installs the base package + the extras, not quite sure what this means for the metadata though)
  • This avoids having to decide which packages are allowed to "provide" which other packages
  • But it would require some backend parsing to verify that the uploader owns the package without the extras maybe? Not too sure about how the security works on that side

I think that captures my current understanding of the situation, feel free to jump in and correct whatever I've forgotten / left out

[nlhkabu]

As this ticket is blocked by the development of the dependency resolver (#988), I thought I would mention here that the team is looking for help from the community to move forward on that subject.

We need to better understand the circumstances under which the new resolver fails, so are asking for pip users with complex dependencies to:

1. Try the new resolver (use version 20.1, run --unstable-feature=resolver)
2. Break it :P
3. File an issue

You can find more information and more detailed instructions here

[webknjaz]

As this ticket is blocked by the development of the dependency resolver

FTR the dependency resolver has been implemented. So I suppose this is now unblocked.

[CGamesPlay]

I just wanted to link to pypa/pip#12879, which explains a use case where knowing the extras at installation time is necessary to alter the compilation of native extensions (if an extra dependency is enabled, the native code wants to link against it). Worth considering if this ever gets to the PIP stage.