Define all old pip entrypoints

[chrahunt]

What's the problem this feature will solve?

Tons of users are experiencing problems related to #5599, as evidenced by the traffic that we're getting to related issues (past 2 weeks):

Link	Views	Unique Visitors
#5447	11,368	9,005
#5221	3,252	2,733
#5240	2,398	2,068
#5599	2,247	1,857

This probably leads to a lot of negative feelings not just for pip, but the Python packaging ecosystem in general.

Describe the solution you'd like

Move the current entrypoint from pip._internal.main:main to pip._internal.cli.main:main (or some other location).

Create a function for every known past entrypoint:

• pip._internal.main:main
• pip._internal:main
• pip:main

At the very least the functions should invoke the correct current pip entrypoint. This alone would prevent a lot of needless suffering and should be easy to implement and test.

Later we could also consider printing a warning, but I'd rather handle that as a separate issue since this is useful on its own.

Alternative Solutions

• do nothing

Additional context

• pip install --user --upgrade #7493
• pip is not working #7403
• Help? pip won't work #7396
• Why I can't install pip #7369
• what problems ?Help me!! #7368
• I cant #7366
• TypeError: 'module' object is not callable #7264
• pip does not work after upgrade to v19.3 #7223
• Moving pip main function to own module broke dependent packages #7220
• 19.3 doesn't work after upgrading from 19.2.3 #7209
• Trying to execute module as a function #7206
• pip 19.3 breaking on CircleCI sys.exit(main()) TypeError: 'module' object is not callable #7205
• pip install --upgrade pip breaks pip (new version 19.2.3) #6927
• "cannot import name 'main'" on SO
• "TypeError: 'module' object is not callable" pip on SO
• more

[pfmoore]

I agree with this as a stopgap measure, but I'm a strong -1 on retaining such "legacy" entry points indefinitely (unless we change our official position on "no supported API").

Personally, I would rather include a warning right now, saying something along the lines of "you are calling the wrong internal entry point for pip (possibly as a result of using an old wrapper script)". We can tidy up the message, and/or find a better long-term solution, later, but I feel that we should have something from the start.

The long-term solution will be hard to identify, as most of the time the failing wrappers appear to be distro-supplied, or the result of users with limited command line knowledge misunderstanding concepts like PATH. Both of which are reasonable issues, but out of our control. Maybe we should finally switch to making python -m pip the only supported way of invoking pip, and stop shipping the wrappers? It's hard to argue that the wrapper is for "convenience" if it's causing this level of problems for so many users. We could then have the warning suggest that users need to invoke pip that way.

[pfmoore]

Alternative solution: #3164.

This could be handled as a phased approach:

1. Document that python -m pip is the supported way to run pip.
2. Add a warning to the wrappers, saying that people should switch to python -m pip.
3. Make that warning a hard error.
4. Remove the wrappers altogether.

It would obviously need a long deprecation period, but I'm fine with that.

In the short term, any issues with wrappers failing can be handled by simply advising the user to switch to python -m pip. That's a policy that could be adopted now.

[uranusjr]

I’m a little skeptical wrt deprecating the pip entrypoint. The main source of the problem here is entrypoints from Linux distributors. I kind of wonder whether they would honour pip removing entrypoints, or go on to generate their own anyway. We wouldnt solve much of the problem if they do (and maybe creating more since there would then be two different entrypoint flavours in the market).

[pfmoore]

The main source of the problem here is entrypoints from Linux distributors.

Note: You mean "wrapper" rather than "entrypoint" here, but your point is valid.

I've been very careful as a Windows user to avoid mentioning distros, but as you raised it, I think it's relevant. Many of the issues we see with the main() API appear to come from how distributions ship wrappers as part of their system repackaging of pip. Without some sort of assistance or support from the distros, I don't think there's any way we can cleanly solve this issue (short of taking the view that the distros are an irresistable force and we simply have to make the main() API supported at least to the level that they require, regardless of our wishes).

To be fair, though, pip's policy has always been to leave the system pip to the distro (although user installs are a sticking point here). But many users don't follow that advice as well as we'd like - not least because doing so (as far as I can tell) needs a reasonable understanding of how PATH works, which not all of our users have.

If we deprecated the pip wrapper, we could at least say to anyone using a wrapper "either reproduce the issue using python -m pip or discuss with your distro support channel". Hardly ideal, but at least a clear separation of responsibilities...

[pradyunsg]

I agree with making this a stop-gap solution (I'd assumed that was the case, since that's basically how I was personally viewing this).

It would obviously need a long deprecation period, but I'm fine with that.

IMO, we should consider reaching out to distros, to get them to fix their wrappers -- in their current distro releases as bug fixes, or in the next "major" version. That would help inform the length of this deprecation period.

[pfmoore]

IMO, we should consider reaching out to distros

+1. Does anyone have a good means of doing this (and of tracking it, we should monitor responses and progress on this, and link from here)?

[chrahunt]

Is this actually a problem with distro wrappers? I assumed that the distro wrappers were consistent with the system-managed pip and the issue in most cases was either:

• A user-scheme install, but with
  • ~/.local/bin later on PATH
  • ~/.local/bin not on PATH
  • ~/.local/bin earlier on PATH, but the shell has hashed pip
• A global install into the same site-packages as the system-managed pip, but the "scripts" dir used is after the one used by the system-managed pip in PATH
• A global install into a different site-packages and "scripts" dir as the system-managed pip, but which has sys.path precedence to the system-managed pip, but the system-managed pup's script dir is ahead on PATH
• Equivalent on Windows

None of which I think are distro issues.

[pfmoore]

None of which I think are distro issues.

Maybe you're right. But the whole point of having this issue at all is that we don't want to categorise these problems as user issues (after all, we're putting a lot of effort into working round the fact that some users don't understand how PATH works). And yet this clearly isn't a pip issue, as the wrapper pip installs works with the version of pip that pip installs - it's not our fault if the user executes a different wrapper. So there's not really anyone left apart from the distros 🙁

The point of involving the distros is (as far as I can see) that they are better able to help users with general "command line UX" issues. Maybe they won't want to, and maybe they are stricter in their view that knowing how to use the command line is the user's problem. But at least we can ask.

Maybe this is something we should be referring to the funded work on pip's UX that I believe is under consideration. I've no idea where that is up to right now, but I'd assume that the UX specialists who would be involved in that would have good insight into this sort of thing.

[oldium]

Big thanks to @chrahunt that I will not need to face this issue again and again on multiple systems and installations. 👍🎉

[oldium]

For others wanting to try this locally before official release – this does the trick python -m pip install --user --upgrade git+https://github.com/pypa/pip.git.

The warning is very confusing to me, because I actually just updated the pip package. I did nothing wrong, still I see the warning. Pip is basically warning that it uses pip... 😕

Anyway, good move towards having this definitely fixed. 👍

[chrahunt]

Hi @oldium, can you explain what you mean about the warning being confusing? What is your current understanding of the situation given what was output by pip? We can always improve it.

[oldium]

Hi, I will illustrate it on an example:

#> pip --version
WARNING: pip is being invoked by an old script wrapper. This will fail in a future version of pip.
Please see https://github.com/pypa/pip/issues/5599 for advice on fixing the underlying issue.
To avoid this problem you can invoke Python with '-m pip' instead of running pip directly.
pip 20.0.dev0 from C:\Users\oldium\AppData\Roaming\Python\Python38\site-packages\pip (python 3.8)

I have latest python 3.8.1, pip during installation of some package wrote to me that there is a newer version. Ok, I updated pip (I updated to latest master), but now pip complains with much bigger warning that there is something wrong. Also the requested version output is unreadable because of this.

I find it really useful to have pip shortcut, because it is harder to remember the sequence python -m pip, I had to search Google almost every time, because I use pip very rarely. I see the warning confusing, because it only informs me that pip is calling some internal API. I can imagine that public API during development prints warnings, but I see no reason to bother user about something that he cannot influence - in the release version and internal API usage. This does not make sense to me.

[pradyunsg]

@oldium What's the output of the following commands for you?

> where pip
> type "<copy-paste the output from the last command here>"

[oldium]

#> where.exe pip
C:\Python38\Scripts\pip.exe
C:\Users\oldium\AppData\Roaming\Python\Python38\Scripts\pip.exe

Note: I updated pip with the --user flag, because I did not want to modify the installed version (and therefore use console with administrator rights).

pip gets executed from C:\Python38\Scripts, because it is found in the system's PATH environment variable, which is prepared by Python installer (I used Chocolatey to install it). User's C:\Users\oldium\AppData\Roaming\Python\Python38\Scripts is not on PATH by default, I had to add it manually. I logically did this on the user's PATH environment variable, but that one is taken after the system's one, so even when I have the updated pip.exe binary in AppData folder, it does not get executed. So I have a classic mismatch of old pip executable and new pip python site-package.

Annoying is that when I use admin rights (not everybody has them) and add my user's AppData Python38 folder to the beginning of the system PATH environment variable, this breaks with next release of Python39, because I would have to manually update the system PATH again (and remember that I have to do it). And I expect that once I will not be ready, the pip API breakage strikes again (as it did several times in past).

That was Windows. Ubuntu 18.04 adds ~/.local/bin to the beginning of the PATH, if the bin folder exists, so it might take the updated pip binary. The .local/bin directory does not exist initially, so if you simply update pip and try to use it immediately from the same shell, you will face the same issue actually...

During past several years I have not found a good solution to this problem, neither on Windows and neither on Linux. The simplest would be if pip had a stable internal API entry point. And I prefer to use pip shorthand, because remembering python -m pip is harder (I remember it now, but that is a result of multiple frustrations in past spending searching Google for the reason of pip breakage and the right syntax to use the module directly).

Why is pip executable simply not trying to call python -m pip anyway?