Upgrade Message on Linux (and talking to Linux Vendors)

[pradyunsg]

It seems pip's current upgrade message instructions on Linux doesn't really match what the OS package managers would want users to do -- running pip install --upgrade pip on a Debian/Fedora system can break it (especially when they might end up doing sudo pip or something like that).

It would be nice if we could ask the upstream vendors to modify their version of pip to either not show the message or (better) show the message when the OS vendors the newer version, with appropriate instructions.

---

For easier future reference; as on 16th July 2018, the following are the people we'd want to ping based on Distro:

• Debian: @doko42 @warsaw (neither are "maintainers")
• Fedora: @torsava @stratakis @hroncok
• Arch Linux: @felixonmars (maintainer), @eli-schwartz (community contributor)

Additionally, we have Conda folks too: @kalefranz @jakirkham @msarahan

[pradyunsg]

I guess that @ncoghlan would be a good person to ask for inputs here.

Doing some research, @warsaw for Debian and @stratakis for Fedora might be the right people to ping here about this.

[benoit-pierre]

You could suppress the message when pip is unvendored, as that's a pretty good way to tell pip is installed from a distribution package manager.

[pfmoore]

While it's not a reason for not doing this, the problem is that we can put such a change in the next version of pip, but the problem is precisely for the cases where the OS version of pip is old, so the fix won't help for a long time.

So while I agree with the change, I think explaining to distro vendors the need for them to customise the selfcheck would be beneficial as well.

[ncoghlan]

At a pip implementation level, I'd suggest checking INSTALLER in pip's own dist-info directory to decide whether or not to run the version check, as that would also add an incentive for distro packagers to ensure that they modify the INSTALLER file appropriately (rather than leaving it as pip, which is what happens when the results of pip-based installation into the build root are copied without modification into the resulting package).

At a documentation level, I also agree with Paul that it would be good to recommend that redistributors patch that message appropriately for their particular system.

[stratakis]

Thanks for creating the issue and notifying us! We recently got bitten by this in Fedora and I guess it was only reported now due to the incompatibilities between pip 9 and pip 10. I'm pretty sure many more people have actually updated pip using that method. We are tracking the issue now downstream and we'll change the message accordingly.

I am also thinking if it would be possible to entirely disable this behaviour (using pip to upgrade pip) while not inside a venv.

[hroncok]

I wasn't aware that this is tracked here, thanks @stratakis for pointing me here. Several things I consider important:

• The massage explicitly mentions pip install --upgrade pip -> that would attempt to upgrade the Python 2 pip on Fedora*. The message is like this even when pip3 or python3 -m pip was used to run pip.
• The pip install --upgrade pip command does not work on Fedora* - it will fail with Permission error. Users then immediately try with sudo and that can be nasty.
• Even python3 -m pip install --user --upgrade pip isn't particularly nice: executing just pip3 after the upgrade still uses /usr/bin/pip3 and that's comes from the old version. (That's probably something we can solve with PATH order.)

However, the message is completely fine in venv. If we are changing it, let's make sure we are not changing it there. I wonder if INSTALLER in pip's own dist-info differs in venv. If it does, @ncoghlan's solution is perfect.

* outside of venv

[hroncok]

I'm working on a patch for Fedora. Will test it and provide a PR.

[hroncok]

Fedora PR first: https://src.fedoraproject.org/rpms/python-pip/pull-request/5

Note that I went with the easy option: I just disabled the warning when INSTALLER doesn't say pip. Allowing different messages based on different INSTALLERs would also require different checking for the newer version (when a new version of pip is released, it takes a while before it is packaged for distros). I felt like the entire thing would need to be redesigned if we would like to allow that. So my patch only went with the simple option: not installed by pip? no warning.

[ncoghlan]

Silencing the warning for non-pip installs in general seems reasonable to me, since redistributors also tend to have their own ways of encouraging upgrades-by-default (e.g. Fedora Workstation points out available updates on start-up, and then has release-monitoring.org+the-new-hotness to nudge package maintainers to make those updates available to their users).

[hroncok]

pip PR: #5368

[pradyunsg]

Oh, while we're here, who all should I ping when there's something related to pip and how it interacts with Linux Distros?

[pradyunsg]

Maybe @doko42 (if not @warsaw) is the right person to ping for Debian? 1

[hroncok]

For Fedora that would be @torsava @stratakis and me on GitHub. Don't know if you can ping a group with @fedora-python.

[pradyunsg]

Thanks for the info @hroncok!

I don't think you can send notifications by mentioning a group.

[mbargull]

Oh, while we're here, who all should I ping when there's something related to pip and how it interacts with Linux Distros?

Even though there is no "Conda Linux", I'd like to bring up some Conda folks like @kalefranz and @jakirkham, if you don't mind (in reference to assessments like in #5060 (comment)).

[pradyunsg]

Sure thing, thanks @mbargull. :)

[doko42]

On 10.05.2018 15:11, Pradyun Gedam wrote: Thanks for the info @hroncok! I don't think you can send notifications by mentioning a group.

I'm not really maintaining python-pip. But if you want to meet up here at PyCon, both Barry and myself are at the venue, including the sprints. And please, never touch/remove system installed eggs with pip. That's really a no-go. Never. Ever. Thanks, Matthias

[pradyunsg]

Updated the title to be more on-topic to how this discussion went. :P

[eli-schwartz]

The Arch Linux maintainer for pip would be @felixonmars, though I would not be averse to being pinged myself as well.

(I don't have access to the [extra] repository where we package pip, but I do do some python packaging in our [community] repository, and am in general pretty interested, so I could help figure issues out even if @felixonmars makes the final calls etc.)

[pradyunsg]

So, right now we have a change in, so that pip won't print an "upgrade me!" message if the "INSTALLER" of pip is not pip -- #5368.

[pradyunsg]

Another thing we might want to do is what @benoit-pierre suggested -- suppressing the message if pip is unvendored.

[eli-schwartz]

Distros might or might not choose to unvendor pip. I guess a more reliable check is whether the INSTALLER metadata is there. Is there a situation where pip is devendored by the distro but installed using itself?

I cannot think of a reason to install pip using pip for distribution packaging, personally -- it works quite well with python setup.py build && python setup.py install --root="$DESTDIR", in which case there will be no INSTALLER. This is what we do in Arch Linux, certainly.
It's also consistent with the general state of packaging python modules for distribution packages for which we use setuptools.

Using pip to install pip would lead to a bootstrap problem, how do you get pip in order to use pip? We currently have no bootstrap issues (I think) other than setuptools, which we use to install everything else (but which depends on packaging and appdirs.)

We try to keep bootstrapping problems to a minimum, because they require additional work to stage temporary bootstrap-only packages that let the build proceed.

[pradyunsg]

#5603 documents the changes discussed here, in the vendoring/debundling documentation.

[nehaljwani]

Why not go one step further and also disable upgrades for packages which were not installed via pip? For example, this is very much possible:

$ rpm -qf /usr/lib64/python2.7/site-packages/nacl
python2-pynacl-1.2.0-2.fc28.x86_64

$ pip freeze
cffi==1.11.5
ply==3.9
pycparser==2.14
PyNaCl==1.2.0
six==1.11.0

$ sudo pip install --upgrade pynacl
...
Successfully installed pycparser-2.18 pynacl-1.2.1

[pradyunsg]

@nehaljwani That's something that we can discuss in a new feature request issue. :)

[pradyunsg]

Thanks for filing #5605; if anyone from this thread wants to chip in over there, they're welcome to. :)

[pradyunsg]

I'm going to go ahead and close this -- this issue still has the current reference for @pypa/pip-committers, on who to ping if there's an issue regarding Linux vendors.

[ncoghlan]

New issue to start discussing a general purpose _platformhints module to let Python level installers issue queries to the underlying platform: #6468

[pradyunsg]

I'm going to go ahead and close this

I'm slow at clicking buttons. :)