GUIDE: How to install PyTorch via Pipenv (and how to add other 3rd Party Repositories). #4961

Arcitec · 2022-02-18T23:56:08Z

Issue description

Try this Pipfile on 64-bit Linux. It fails to find the specified version:

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[[source]]
url = "https://download.pytorch.org/whl/cu113/torch_stable.html"
verify_ssl = true
name = "pytorch-cu113"

[packages]
torch = {version = "==1.10.2+cu113", index = "pytorch-cu113"}

[dev-packages]

[requires]
python_version = "3.9"

Edit: The problem has been found. Torch's repo (the URL I used) is a non-standardized repo that only pip understands. Pipenv only reads the PEP 503-standardized repos. See solution in my followup comments below.

Arcitec · 2022-02-19T00:41:44Z

edit: Solved, see below. It's Torch's fault, not Pipenv's.

Arcitec · 2022-02-19T03:45:55Z

Alright I've figured it out. Pipenv only supports standardized PEP 503 repositories. Torch, by default, gives us the URL for a non-standard "pip" bastard repo. I will post a followup comment with the full answer/solution for people.

Arcitec · 2022-02-19T03:46:01Z

Solved: This is how to install Torch CORRECTLY in Pipenv. Most people do this incorrectly even in various "solutions" I've seen from other people... I've researched to the bottom of this problem and found out the real, intended solution that everyone should use.

This technique is the correct one and it's documented here:

https://pipenv.pypa.io/en/latest/advanced/#specifying-package-indexes

Basically, you add a new "source repository" to your Pipfile. And you should always use verify_ssl = true for extra security when possible.

Anyway, there's a shortcut to adding a custom repository without editing the pipfile. The --extra-index-url shortcut says "Add this PEP 503-standard repo URL":

Version that uses CUDA Toolkit 11.3, with GPU acceleration (this toolkit is required by NVIDIA Ampere (RTX 30x0) GPUs and newer):

pipenv install --extra-index-url https://download.pytorch.org/whl/cu113/ "torch==1.10.1+cu113"

Version that uses CUDA Toolkit 10.2, with GPU acceleration:

pipenv install --extra-index-url https://download.pytorch.org/whl/ "torch==1.10.1+cu102"

Version that uses CPU instead:

pipenv install --extra-index-url https://download.pytorch.org/whl/ "torch==1.10.1+cpu"

Now you may notice something: There's no ".html" in my repo filenames.

That's because the repos with a HTML filename (such as https://download.pytorch.org/whl/torch_stable.html) are NON-STANDARD repos that ONLY pip understands.

You CANNOT use those URLs in Pipenv.

Pipenv uses PEP 503 standardized repos. Torch publishes those at the URLs WITHOUT ".html" filenames. So remove the filename to get the proper repo URLs for Pipenv usage.

This fact is documented here, when Torch's team decided to finally publish PEP 503 repos: pytorch/pytorch#25639 (comment)

You may notice something else: I always specify a version with `+cpu` or `+cu113` or whatever.

This is NECESSARY, because otherwise Pipenv doesn't know which version to install and won't get GPU acceleration. The "architecture" (GPU vs CPU vs different CUDA toolkits) is baked into the version number. It's literally what Torch decided to do, and the ONLY correct way to download them is to specify the exact version and architecture (such as +cu113) you want.

If you want to figure out the URL to the correct repos/versions in the future, do as follows:

Go to pytorch.org and select "Stable, Pip, Python, CUDA 11.3" (or whatever is the latest CUDA you may be using), and then the command textbox will reveal the repo URL for the latest CUDA toolkit. You MUST then remove the HTML filename because that's the pip-specific repo.

Visit the resulting URL, such as https://download.pytorch.org/whl/cu113/, and navigate into the desired library folder. Then search for the version you wanted, and be sure to specify its FULL version specifier, such as 1.10.1+cu113.

Oh and sometimes, the Torch team forgets to upload the latest version to the PEP 503 repo, such as right now, where 1.10.2 is available but only in the non-standard pip-style repo. That's why you should always visit the PEP 503 repo to see the latest version they've uploaded into that repo. (I would love if they completely stop providing the old-school pip-repo at all, since pip supports PEP 503 too, so they should focus on the standards-compliant repo...)

See my examples above for perfect installation instructions for the current versions as of this writing.

One final note: ALWAYS RESPECT THE TORCH PYTHON VERSIONS.

The packages will have names such as "cp37", "cp38", "cp39". As of this writing, the highest version they have created is for Python 3.9. There's no Python 3.10 version of Torch.

Therefore, it helps to install Pyenv and specify an exact Python version in your Pipenv via pipenv install --python=3.9 to ensure that you have the latest version that Torch supports and not anything "too new/unsupported". :)

Good luck.

Arcitec · 2022-02-19T04:15:53Z

I've gone on a posting spree in various incorrect topics (pretty much all of them were wrong), and linking people here so that random interwebs cybersurfers can find the correct way to install PyTorch and other PEP 503 repos in Pipenv!

Hopefully this will stop the flow of misinformation and "wrong solutions" that's currently out there! 😄

Welcome, everyone!

DanielPerezJensen · 2022-03-29T07:40:04Z

Thanks for this guide, but when I try to do: pipenv install --extra-index-url https://download.pytorch.org/whl/cu113/ "torch==1.10.1+cu113"

I get an AttributeError: 'list' object has no attribute 'startswith'.

The full error is:

Traceback (most recent call last):
  File "/usr/local/bin/pipenv", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.8/dist-packages/pipenv/vendor/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/pipenv/vendor/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.8/dist-packages/pipenv/vendor/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.8/dist-packages/pipenv/vendor/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.8/dist-packages/pipenv/vendor/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/pipenv/vendor/click/decorators.py", line 84, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/pipenv/vendor/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/pipenv/cli/command.py", line 194, in install
    do_install(
  File "/usr/local/lib/python3.8/dist-packages/pipenv/core.py", line 2114, in do_install
    index, verify_ssl=index.startswith("https:")
AttributeError: 'list' object has no attribute 'startswith'

Arcitec · 2022-03-29T13:06:14Z

Edit for anyone reading this thread in the future: You can skip the back and forth testing between me and Daniel. There is/was a bug in Pipenv that prevents installation of PEP 503 packages. There are multiple bug report tickets about it. The last known working version of Pipenv was 2022.1.8. The March 2022 versions are broken. I expect that Pipenv versions made after March 2022 will be fixed soon.

@DanielPerezJensen Thanks for posting the full traceback. It shows that your error is just a bug inside pipenv. Try updating pipenv. If that fails, try updating Python itself (you can go up to Python 3.10 now if you want to use the latest Torch version).

The bug is on this line of code in pipenv, where their code tries to do a string.startswith() on list object:

  File "/usr/local/lib/python3.8/dist-packages/pipenv/core.py", line 2114, in do_install
    index, verify_ssl=index.startswith("https:")
AttributeError: 'list' object has no attribute 'startswith'

PS: There are newer versions of Torch out now. Check my guide steps above for how you can update the commands to the latest version. You will need to follow my guide steps to set the correct, new version number in the command.

DanielPerezJensen · 2022-03-29T13:14:22Z

@DanielPerezJensen Thanks for posting the full traceback. It shows that your error is just a bug inside pipenv. Try updating pipenv. If that fails, try updating Python itself.

The bug is on this line of code in pipenv:
  File "/usr/local/lib/python3.8/dist-packages/pipenv/core.py", line 2114, in do_install
    index, verify_ssl=index.startswith("https:")
AttributeError: 'list' object has no attribute 'startswith'
PS: There are newer versions of Torch out now. Check my guide steps above for how you can update the commands to the latest version.

Thanks for the quick response!

In my case I need that previous version, and I figured since I could find the proper version in the link it should work. After updating pipenv I still get an error when I run the command:

pipenv install --extra-index-url https://download.pytorch.org/whl/cu113/ "torch==1.10.1+cu113"

The error trace is detailed below:

Installing torch==1.10.1+cu113...
Adding torch to Pipfile's [packages]...
✔ Installation Succeeded 
Pipfile.lock (b2335c) out of date, updating to (33817e)...
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
✘ Locking Failed! 

CRITICAL:pipenv.patched.notpip._internal.resolution.resolvelib.factory:Could not find a version that satisfies the requirement torch==1.10.1+cu113 (from versions: 1.4.0, 1.5.0, 1.5.1, 1.6.0, 1.7.0, 1.7.1, 1.8.0, 1.8.1, 1.9.0, 1.9.1, 1.10.0, 1.10.1, 1.10.2, 1.11.0)
[ResolutionFailure]:   File "/home/daniel/.local/lib/python3.8/site-packages/pipenv/resolver.py", line 743, in _main
[ResolutionFailure]:       resolve_packages(pre, clear, verbose, system, write, requirements_dir, packages, dev)
[ResolutionFailure]:   File "/home/daniel/.local/lib/python3.8/site-packages/pipenv/resolver.py", line 704, in resolve_packages
[ResolutionFailure]:       results, resolver = resolve(
[ResolutionFailure]:   File "/home/daniel/.local/lib/python3.8/site-packages/pipenv/resolver.py", line 685, in resolve
[ResolutionFailure]:       return resolve_deps(
[ResolutionFailure]:   File "/home/daniel/.local/lib/python3.8/site-packages/pipenv/utils.py", line 1398, in resolve_deps
[ResolutionFailure]:       results, hashes, markers_lookup, resolver, skipped = actually_resolve_deps(
[ResolutionFailure]:   File "/home/daniel/.local/lib/python3.8/site-packages/pipenv/utils.py", line 1127, in actually_resolve_deps
[ResolutionFailure]:       resolver.resolve()
[ResolutionFailure]:   File "/home/daniel/.local/lib/python3.8/site-packages/pipenv/utils.py", line 905, in resolve
[ResolutionFailure]:       raise ResolutionFailure(message=str(e))
[pipenv.exceptions.ResolutionFailure]: Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  You can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
  Hint: try $ pipenv lock --pre if it is a pre-release dependency.
ERROR: No matching distribution found for torch==1.10.1+cu113

I get the same error when trying to use the latest version with the command
pipenv install --extra-index-url https://download.pytorch.org/whl/cu113/ "torch==1.11.0+cu113"

What exactly am I doing wrong?

Arcitec · 2022-03-29T13:47:00Z

@DanielPerezJensen You aren't doing anything wrong! :)

Your Python runtime is a bad version or an unsupported version (such as the "M" version instead of standard Python).

I did a test with your command for the latest Torch version, and I used the latest version of the official CPython 3.8 (since I see that you use Python 3.8).

 ~/D/testpy  pipenv install --python=3.8
Creating a virtualenv for this project...
Pipfile: /home/mxuser/Downloads/testpy/Pipfile
Using /home/mxuser/.pyenv/versions/3.8.12/bin/python3.8 (3.8.12) to create virtualenv...
⠼ Creating virtual environment...created virtual environment CPython3.8.12.final.0-64 in 144ms
  creator CPython3Posix(dest=/home/mxuser/.local/share/virtualenvs/testpy-o3WQlgT4, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/mxuser/.local/share/virtualenv)
    added seed packages: pip==21.3.1, setuptools==60.5.0, wheel==0.37.1
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator

✔ Successfully created virtual environment! 
Virtualenv location: /home/mxuser/.local/share/virtualenvs/testpy-o3WQlgT4
Creating a Pipfile for this project...
Pipfile.lock not found, creating...
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
Updated Pipfile.lock (db4242)!
Installing dependencies from Pipfile.lock (db4242)...
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 0/0 — 00:00:00
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.

 ~/D/testpy  pipenv install --extra-index-url https://download.pytorch.org/whl/cu113/ "torch==1.11.0+cu113"
Installing torch==1.11.0+cu113...
Adding torch to Pipfile's [packages]...
✔ Installation Succeeded 
Pipfile.lock (db4242) out of date, updating to (7d86a3)...
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
✔ Success! 
Updated Pipfile.lock (7d86a3)!
Installing dependencies from Pipfile.lock (7d86a3)...
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 0/0 — 00:00:00
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.

 ~/D/testpy  pipenv install numpy                                                                          
Installing numpy...
Adding numpy to Pipfile's [packages]...
✔ Installation Succeeded 
Pipfile.lock (7d86a3) out of date, updating to (fe714e)...
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
✔ Success! 
Updated Pipfile.lock (fe714e)!
Installing dependencies from Pipfile.lock (fe714e)...
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 0/0 — 00:00:00
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.

 ~/D/testpy  pipenv run python   
Python 3.8.12 (default, Jan 27 2022, 01:41:52) 
[GCC 11.2.1 20211203 (Red Hat 11.2.1-7)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> print("Hello World")
Hello World
>>>

Be sure that you have the latest 64-bit version of CPython, not some alternative interpreter or weird version of Python. Because that would mean that it can't find the package, since the Torch packages are only made for CPython 64-bit.

Also be sure that you have the latest patch for your version of Python. In your case it means Python 3.8.13.

Furthermore, Torch is only made for Linux + Windows. There are no Mac versions.

You're going to have to change Python interpreter to one that Torch supports, that's for sure.

DanielPerezJensen · 2022-03-29T14:00:30Z

@Bananaman

You are of course under no obligation to help me, so thanks a ton already.

I am a bit confused about your statement regarding my interpreter. What commands should I use to determine my CPython version, for example?

I am using Ubuntu 20.04, with the default Python 3.8.10. I am not using some non-standard version of Python/CPython.

The way I have installed PyTorch currently is by simply using pip in my pipenv. This has worked thus far, so I am pretty sure there are no installation problems from that end.

Sorry for my vague, non-specific response, but I am simply not sure what else to specify. Perhaps I can give more detail about my system?

Arcitec · 2022-03-29T14:20:16Z

@DanielPerezJensen No problem, but I don't have much more time today and have to go now. My suggestion is to bypass the system-wide Ubuntu version of Python since they may have messed with its source code. Ubuntu/Debian always messes with packages and does a ton of patching/backporting/modifications.

You can install the official Python from source quite easily by installing pyenv instead. Pipenv will use pyenv to download and compile Python from source code.

You can then use commands such as pipenv install --python=3.10 to force pyenv to download and install a certain version (but be aware that you must run this "force python version" command when you create the Pipenv, it can't be run later after you've already got a Pipfile).

You can also manually call pyenv to install certain versions to override your system package's version, since pipenv prefers the pyenv-versions of Python when available but otherwise falls back to the system-wide version if that version exists there. So if you want specifically Python 3.8, which is already on your system, then you need to force pyenv to install it first, so that pipenv will see it and prefer it. For example:

export SETUPTOOLS_USE_DISTUTILS=stdlib
pyenv install 3.8.13
pipenv install --python=3.8

You should see some line such as "Using /home/mxuser/.pyenv/versions/3.8.13/bin/python3.8 (3.8.13) to create virtualenv...", then you know that pipenv has selected the pyenv version of Python.

PS: The setuptools line is there to fix a bug in some Python versions. Certain versions of Python or setuptools need that extra environment flag to be compiled successfully (see https://bugs.python.org/issue46401 and pypa/setuptools#3031 if you are curious).

I'll share my full pyenv + pipenv installation guide with you. Everything below is copied from my personal documents. If you follow these steps and then install Python via pyenv, you will most likely have a working installation after that. I say most likely, because mine is installed that way and works as you saw above! :)

I hope that this works out for you. At least, you'll finally have pyenv on your system, which is a great tool!

Python

Pyenv (Python Version Management)

This is a tool for installing different versions of Python on your system. They're isolated from the system's own Python, for stability.
First, install all dependencies required for compiling Python (since pyenv installs every version from source code). The official list is at https://github.com/pyenv/pyenv/wiki#suggested-build-environment so visit that site and run the dependency installation commands for your distro.
Now just install pyenv. This script will git-clone the main repository and popular plugins (doctor, installer, update, virtualenv and which-ext) into ~/.pyenv. Ignore the warning about adding the tools to your "PATH", since we'll do that properly for your shell next.

curl -s -S -L https://raw.githubusercontent.com/pyenv/pyenv-installer/master/bin/pyenv-installer | bash

Now we need to setup our shell's environment variables. You can copy and paste all of these lines at once into your terminal to run every command. This sets it up for both ZSH and Bash. These instructions have been edited compared to the official docs, because Fedora doesn't use .profile or .zprofile files. And while we do have .bash_profile, it doesn't run when you spawn bash as a sub-shell, so we need to use .bashrc instead (which also ensures that our path overrides will happen after .bashrc overrides, to ensure that our Python shims have the highest priority. This adjusted setup works perfectly on Fedora (and should work on all other distros), unlike the original "official" instructions.

echo '# Load pyenv configuration' >> ~/.zshrc
echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.zshrc
echo 'export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.zshrc
echo 'eval "$(pyenv init --path)"' >> ~/.zshrc
echo 'eval "$(pyenv init -)"' >> ~/.zshrc

echo '# Load pyenv configuration' >> ~/.bashrc
echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.bashrc
echo 'export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.bashrc
echo 'eval "$(pyenv init --path)"' >> ~/.bashrc
echo 'eval "$(pyenv init -)"' >> ~/.bashrc

Restart your shell or terminal window to load the updated configuration.
Now run a command to check if your system has all packages/dependencies for building Python versions from source. It should say that you are ready to build, without warnings that any problems were detected.

pyenv doctor

We will not be adding the pyenv-virtualenv plugin to your shell init scripts, because we'll be managing that via pipenv instead.
Likewise, you won't be using pyenv itself, since it will be entirely managed by pipenv instead. So don't bother using its tools and commands for setting Python versions or per-folder versions, etc. It would be counter-productive to mix it and pipenv.
Lastly, to update the pyenv tools at any time, simply run the following command to update all of your main and plugin git repositories. We'll be making an alias for updating both this and pipenv later, to simplify the process.

pyenv update

Pipenv (Python Project Environment Management)

This tool automatically manages your project dependencies, virtualenvs and Python versions (when pyenv is also installed). It simplifies everything into one tool which replaces pip, python, anaconda/miniconda, pyenv, virtualenv, etc.
NOTE: Fedora has a pipenv package, but it's severely out of date. We'll instead install the latest version via pip.
Run the following command to install it and all dependencies into your local user's Python packages folder. This prevents it from clashing with or overwriting system packages. They'll instead end up in ~/.local/lib/python3.10/site-packages/ or whatever your current "default" Python version is on the system at the moment. When your system updates its Python interpreter, you'll probably have to reinstall the pipenv package.

pip install --user pipenv

If you were using ZSH, the command most likely warned you that the local bin path doesn't exist in your PATH. This means that pipenv cannot be executed yet. Bash already takes care of that automatically, but ZSH needs some help to add the same folders so that it's found by your system.
Open ~/.zshrc in a text editor, and add the following code at the bottom of the file, to ensure that the path is modified and exported as the final step so that all subprocesses also receive the latest/final PATH. The actual code is taken verbatim from how ~/.bashrc solves the issue.

# User specific environment
if ! [[ "$PATH" =~ "$HOME/.local/bin:$HOME/bin:" ]]
then
    PATH="$HOME/.local/bin:$HOME/bin:$PATH"
fi
export PATH

Recommended: If you want to be a perfectionist and ensure that Bash's PATH works the same as ZSH's, by putting pipenv as a higher priority than pyenv's path, simply open ~/.bashrc and move the "Load pyenv configuration" section to above the "User specific environment" section now.
Restart your shell or terminal window to load the updated configuration.
Check if pipenv is working, by running this command.

pipenv --version

Next, we'll install the shell autocompletions. Sadly they add about half a second to the shell startup time if you add them the normal way, which is totally unacceptable for me, especially since the completion scripts for both ZSH and Bash are totally static and don't require future updates. Therefore, we will instead hardcode the completion code into the shell config files.

echo '# Add pipenv autocompletion' >> ~/.zshrc
_PIPENV_COMPLETE=zsh_source pipenv >> ~/.zshrc

echo '# Add pipenv autocompletion' >> ~/.bashrc
_PIPENV_COMPLETE=bash_source pipenv >> ~/.bashrc

Feel free to manually edit those config files to move the autocompletion code to a nicer location in the files.
Restart your shell again, to apply the autocompletion.
Test the autocompletion by typing pipenv in and pressing Tab. It should autocomplete to pipenv install . You'll also notice that there's a delay when the autocompletion happens. This delay is the Python interpreter startup time and would have happened on every shell startup if we didn't hardcode the completions, yikes.
Now it's finally time to create the auto-update alias that was mentioned in the pyenv guide earlier.
Open your ~/.zshrc or ~/.bashrc in a text editor and add the following line to the bottom.

alias py-update="pyenv update && pip install --upgrade --upgrade-strategy eager --user pipenv"

Simply run py-update periodically, to upgrade both pyenv and pipenv to their latest versions, along with all pipenv dependencies (that still satisfy the pipenv requirements. This is recommended even if your system is working, since there are constantly bugfixes and feature additions in both projects and their dependencies. You may also have to run the original pipenv install command whenever your system's default Python version changes, since everything is installed into a versioned package directory, as mentioned earlier.
You're now ready to start using Pipenv. Read the project documentation to get started!

DanielPerezJensen · 2022-03-29T14:38:56Z

Thanks for being so helpful, I will definetely look into this. Isolating my Python versions seems like a prudent thing to do, as I've had many problems with overlapping installs in the past (especially before using virtual environments)

DanielPerezJensen · 2022-03-29T15:19:33Z

@Bananaman

After going through your guide, and verifying that indeed I am now using the Python version from pyenv, I still get the same error.

Here is a trace:

daniel@daniel-desktop:~/Documents/test$ pipenv install
Creating a virtualenv for this project...
Pipfile: /home/daniel/Documents/test/Pipfile
Using /home/daniel/.pyenv/versions/3.8.12/bin/python3.8 (3.8.12) to create virtualenv...
⠹ Creating virtual environment...created virtual environment CPython3.8.12.final.0-64 in 120ms
  creator CPython3Posix(dest=/home/daniel/.local/share/virtualenvs/test-Kb_OZVBf, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/daniel/.local/share/virtualenv)
    added seed packages: pip==22.0.4, setuptools==61.0.0, wheel==0.37.1
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator

✔ Successfully created virtual environment! 
Virtualenv location: /home/daniel/.local/share/virtualenvs/test-Kb_OZVBf
Installing dependencies from Pipfile.lock (db4242)...
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 0/0 — 00:00:00
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.

(test) daniel@daniel-desktop:~/Documents/test$ pipenv install --extra-index-url https://download.pytorch.org/whl/cu113/ "torch==1.11.0+cu113"
Installing torch==1.11.0+cu113...
Adding torch to Pipfile's [packages]...
✔ Installation Succeeded 
Pipfile.lock (db4242) out of date, updating to (7d86a3)...
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
✘ Locking Failed! 

CRITICAL:pipenv.patched.notpip._internal.resolution.resolvelib.factory:Could not find a version that satisfies the requirement torch==1.11.0+cu113 (from versions: 1.4.0, 1.5.0, 1.5.1, 1.6.0, 1.7.0, 1.7.1, 1.8.0, 1.8.1, 1.9.0, 1.9.1, 1.10.0, 1.10.1, 1.10.2, 1.11.0)
[ResolutionFailure]:   File "/home/daniel/.local/lib/python3.8/site-packages/pipenv/resolver.py", line 743, in _main
[ResolutionFailure]:       resolve_packages(pre, clear, verbose, system, write, requirements_dir, packages, dev)
[ResolutionFailure]:   File "/home/daniel/.local/lib/python3.8/site-packages/pipenv/resolver.py", line 704, in resolve_packages
[ResolutionFailure]:       results, resolver = resolve(
[ResolutionFailure]:   File "/home/daniel/.local/lib/python3.8/site-packages/pipenv/resolver.py", line 685, in resolve
[ResolutionFailure]:       return resolve_deps(
[ResolutionFailure]:   File "/home/daniel/.local/lib/python3.8/site-packages/pipenv/utils.py", line 1398, in resolve_deps
[ResolutionFailure]:       results, hashes, markers_lookup, resolver, skipped = actually_resolve_deps(
[ResolutionFailure]:   File "/home/daniel/.local/lib/python3.8/site-packages/pipenv/utils.py", line 1127, in actually_resolve_deps
[ResolutionFailure]:       resolver.resolve()
[ResolutionFailure]:   File "/home/daniel/.local/lib/python3.8/site-packages/pipenv/utils.py", line 905, in resolve
[ResolutionFailure]:       raise ResolutionFailure(message=str(e))
[pipenv.exceptions.ResolutionFailure]: Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  You can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
  Hint: try $ pipenv lock --pre if it is a pre-release dependency.
ERROR: No matching distribution found for torch==1.11.0+cu113

As you can see in the first trace I am using the proper version from pyenv, and in the second trace I get the same error.

I followed all your instructions I believe. No idea what might be causing this.

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[[source]]
url = "https://download.pytorch.org/whl/cu113/"
verify_ssl = true
name = "downloadpytorch"

[packages]
torch = "==1.11.0+cu113"

[dev-packages]

[requires]
python_version = "3.8"

My pipfile looks like this now, after running that command. It just fails to lock this I suppose. Not sure how to go about fixing that.

Arcitec · 2022-03-29T15:37:00Z

@DanielPerezJensen Wow, that is ridiculous. Everything should work now. You have the exact same self-compiled version of 64-bit CPython 3.8.12 as I do. And we both used the exact same pipenv pytorch install command.

It could be two remaining things:

A pipenv bug. I will update my pipenv, clear the pipenv package cache, and try a new install with your exact commands. This will take half an hour to download the packages anew.
A CDN problem where Pytorch isn't hosting that version for your country. The error message you get says that there is no "+cu113" version in the repo you added:

CRITICAL:pipenv.patched.notpip._internal.resolution.resolvelib.factory:
Could not find a version that satisfies the requirement torch==1.11.0+cu113
(from versions: 1.4.0, 1.5.0, 1.5.1, 1.6.0, 1.7.0, 1.7.1, 1.8.0, 1.8.1, 1.9.0, 1.9.1, 1.10.0, 1.10.1, 1.10.2, 1.11.0)
[ResolutionFailure]:   File "/home/daniel/.local/lib/python3.8/site-packages/pipenv/resolver.py", line 743, in _main

DanielPerezJensen · 2022-03-29T15:48:21Z

Yeah, I wish I had a more thorough understanding of these things. But I simply cannot come up with a reason for this, and I can't really find anyone else with the same problem. The error about cu113 not existing is also quite weird, since I can see the package in the provided link.

…

On Tue, 29 Mar 2022, 17:37 Bananaman, ***@***.***> wrote: @DanielPerezJensen <https://github.com/DanielPerezJensen> Wow, that is ridiculous. Everything should work now. You have the *exact* same self-compiled version of 64-bit CPython 3.8.12 as I do. And we both used the exact same pipenv pytorch install command. It could be two remaining things: 1. A pipenv bug. I will update my pipenv, clear the pipenv package cache, and try a new install with your exact commands. 2. A CDN problem. The error message you get says that there is no "+cu113" version in the repo you added: CRITICAL:pipenv.patched.notpip._internal.resolution.resolvelib.factory:Could — Reply to this email directly, view it on GitHub <#4961 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHRZGW5OJAXXN7RRADIT3RTVCMPSRANCNFSM5OZM7KTA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

Arcitec · 2022-03-29T15:51:06Z

I ran py-update (the alias you see in my guide), to update pyenv and pipenv to the latest versions. Now my versions of both tools match yours (mine were a bit outdated).
Next, I did a recompile of CPython 3.8.12. It was already installed so I overwrote it.

 ~  export SETUPTOOLS_USE_DISTUTILS=stdlib
 ~  pyenv install 3.8.12

pyenv: /home/mxuser/.pyenv/versions/3.8.12 already exists
continue with installation? (y/N) y
Downloading Python-3.8.12.tar.xz...
-> https://www.python.org/ftp/python/3.8.12/Python-3.8.12.tar.xz
Installing Python-3.8.12...
Installed Python-3.8.12 to /home/mxuser/.pyenv/versions/3.8.12

After that, I reset the Pipenv cache. This deletes all downloaded packages. In fact it deletes all local/temporary data, so that it's a completely fresh install.

pipenv --clear      
Clearing caches...

I made a test folder and told Pipenv to use Python 3.8.12. You can see that we have the EXACT same 64-bit Python version: CPython3.8.12.final.0-64, and also the exact same "seed packages" (added seed packages: pip==22.0.4, setuptools==61.0.0, wheel==0.37.1)...

 ~  cd ~/Downloads 
 ~/Downloads  mkcd test_torch
 ~/D/test_torch  pipenv install --python=3.8.12
Creating a virtualenv for this project...
Pipfile: /home/mxuser/Downloads/test_torch/Pipfile
Using /home/mxuser/.pyenv/versions/3.8.12/bin/python3.8 (3.8.12) to create virtualenv...
⠼ Creating virtual environment...created virtual environment CPython3.8.12.final.0-64 in 257ms
  creator CPython3Posix(dest=/home/mxuser/.local/share/virtualenvs/test_torch-wEjNt0dX, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/mxuser/.local/share/virtualenv)
    added seed packages: pip==22.0.4, setuptools==61.0.0, wheel==0.37.1
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator

✔ Successfully created virtual environment! 
Virtualenv location: /home/mxuser/.local/share/virtualenvs/test_torch-wEjNt0dX
Creating a Pipfile for this project...
Pipfile.lock not found, creating...
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
Updated Pipfile.lock (db4242)!
Installing dependencies from Pipfile.lock (db4242)...
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 0/0 — 00:00:00
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.

Now I am running "the" command (same as I did earlier and same as you did). Results will take half an hour due to the Pipenv issues which makes it download every version of the package instead of just the necessary one. I will post the results when they are ready.

pipenv install --extra-index-url https://download.pytorch.org/whl/cu113/ "torch==1.11.0+cu113"

If this works, you may want to do a quick test by creating a Fedora 35 virtual machine, install pyenv and pipenv in it (easy with my guide), and then see if the installation finds the package in there. I use Fedora, not Ubuntu/Debian. Maybe there is some issue at the Ubuntu/Debian system level, although I can't think of how that would work since we custom-compile Python and isolate the virtualenv packages.

The only other potential issue, if Fedora fails too, is that PyTorch's CDN is hiding the packages from you.

Anyway, we'll soon know if the command fails for me too now that I've updated pyenv and pipenv to the latest.

DanielPerezJensen · 2022-03-29T16:02:46Z

I will replicate the exact steps you take, so we can compare. This should be as fresh as it's gonna get installation-wise. Hopefully we can create more clarity. Thanks for all the help so far!

DanielPerezJensen · 2022-03-29T16:05:23Z

I just ran them all, the command did not take half an hour on my machine.

I get the same error:

daniel@daniel-desktop:~/Documents$ pipenv install --extra-index-url https://download.pytorch.org/whl/cu113/ "torch==1.11.0+cu113"
Installing torch==1.11.0+cu113...
Adding torch to Pipfile's [packages]...
✔ Installation Succeeded 
Pipfile.lock (db4242) out of date, updating to (7d86a3)...
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
✘ Locking Failed! 

CRITICAL:pipenv.patched.notpip._internal.resolution.resolvelib.factory:Could not find a version that satisfies the requirement torch==1.11.0+cu113 (from versions: 1.4.0, 1.5.0, 1.5.1, 1.6.0, 1.7.0, 1.7.1, 1.8.0, 1.8.1, 1.9.0, 1.9.1, 1.10.0, 1.10.1, 1.10.2, 1.11.0)
[ResolutionFailure]:   File "/home/daniel/.local/lib/python3.8/site-packages/pipenv/resolver.py", line 743, in _main
[ResolutionFailure]:       resolve_packages(pre, clear, verbose, system, write, requirements_dir, packages, dev)
[ResolutionFailure]:   File "/home/daniel/.local/lib/python3.8/site-packages/pipenv/resolver.py", line 704, in resolve_packages
[ResolutionFailure]:       results, resolver = resolve(
[ResolutionFailure]:   File "/home/daniel/.local/lib/python3.8/site-packages/pipenv/resolver.py", line 685, in resolve
[ResolutionFailure]:       return resolve_deps(
[ResolutionFailure]:   File "/home/daniel/.local/lib/python3.8/site-packages/pipenv/utils.py", line 1398, in resolve_deps
[ResolutionFailure]:       results, hashes, markers_lookup, resolver, skipped = actually_resolve_deps(
[ResolutionFailure]:   File "/home/daniel/.local/lib/python3.8/site-packages/pipenv/utils.py", line 1127, in actually_resolve_deps
[ResolutionFailure]:       resolver.resolve()
[ResolutionFailure]:   File "/home/daniel/.local/lib/python3.8/site-packages/pipenv/utils.py", line 905, in resolve
[ResolutionFailure]:       raise ResolutionFailure(message=str(e))
[pipenv.exceptions.ResolutionFailure]: Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  You can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
  Hint: try $ pipenv lock --pre if it is a pre-release dependency.
ERROR: No matching distribution found for torch==1.11.0+cu113

Arcitec · 2022-03-29T16:07:36Z

@DanielPerezJensen Good news (for you).

It's a bug in the newest version of Pipenv.

The question is which commit/update to Pipenv broke Pipenv.

 ~/D/test_torch  pipenv install --extra-index-url https://download.pytorch.org/whl/cu113/ "torch==1.11.0+cu113"
Installing torch==1.11.0+cu113...
Adding torch to Pipfile's [packages]...
✔ Installation Succeeded 
Pipfile.lock (db4242) out of date, updating to (7d86a3)...
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
✘ Locking Failed! 

CRITICAL:pipenv.patched.notpip._internal.resolution.resolvelib.factory:Could not find a version that satisfies the requirement torch==1.11.0+cu113 (from versions: 1.4.0, 1.5.0, 1.5.1, 1.6.0, 1.7.0, 1.7.1, 1.8.0, 1.8.1, 1.9.0, 1.9.1, 1.10.0, 1.10.1, 1.10.2, 1.11.0)
[ResolutionFailure]:   File "/home/mxuser/.local/lib/python3.10/site-packages/pipenv/resolver.py", line 743, in _main
[ResolutionFailure]:       resolve_packages(pre, clear, verbose, system, write, requirements_dir, packages, dev)
[ResolutionFailure]:   File "/home/mxuser/.local/lib/python3.10/site-packages/pipenv/resolver.py", line 704, in resolve_packages
[ResolutionFailure]:       results, resolver = resolve(
[ResolutionFailure]:   File "/home/mxuser/.local/lib/python3.10/site-packages/pipenv/resolver.py", line 685, in resolve
[ResolutionFailure]:       return resolve_deps(
[ResolutionFailure]:   File "/home/mxuser/.local/lib/python3.10/site-packages/pipenv/utils.py", line 1398, in resolve_deps
[ResolutionFailure]:       results, hashes, markers_lookup, resolver, skipped = actually_resolve_deps(
[ResolutionFailure]:   File "/home/mxuser/.local/lib/python3.10/site-packages/pipenv/utils.py", line 1127, in actually_resolve_deps
[ResolutionFailure]:       resolver.resolve()
[ResolutionFailure]:   File "/home/mxuser/.local/lib/python3.10/site-packages/pipenv/utils.py", line 905, in resolve
[ResolutionFailure]:       raise ResolutionFailure(message=str(e))
[pipenv.exceptions.ResolutionFailure]: Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  You can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
  Hint: try $ pipenv lock --pre if it is a pre-release dependency.
ERROR: No matching distribution found for torch==1.11.0+cu113

DanielPerezJensen · 2022-03-29T16:08:36Z

Interesting, perhaps we can raise this as an issue on Pipenv's repo. If it's not intended behaviour.

Arcitec · 2022-03-29T16:09:02Z

@DanielPerezJensen Yes a bug report ticket is necessary. Can you write it and link to this ticket? It's a very simple bug to describe: The latest version of Pipenv no longer finds PyTorch even though older versions of Pipenv did.

DanielPerezJensen · 2022-03-29T16:09:43Z

#5021

Seems others also have this issue for other packages.

DanielPerezJensen · 2022-03-29T16:17:50Z

Rolling back to version 2022.1.8 seems to allow me to install pytorch in this fashion. Although locking is taking a long time right now (more than 5 minutes atm), but I think that is normal?

Arcitec · 2022-03-29T16:24:55Z

Rolling back to version 2022.1.8 seems to allow me to install pytorch in this fashion.

Yep all will be fine now! :)

Although locking is taking a long time right now (more than 5 minutes atm), but I think that is normal?

It's normal as I mentioned here: #4961 (comment)

They need to improve Pipenv someday to be more intelligent about what it downloads from PEP 503 repos. You can read more about why this is a hard problem for them to solve, in the linked ticket in that comment. :)

It's a small design flaw in Pipenv which becomes really painful in this situation due to the huge size of PyTorch's packages. For most other packages they're so small that you don't notice anything.

For me it takes ~30 minutes to install Torch on a 100mbit connection. After that, the packages are cached and install/reinstall instantly.

noamsgl · 2022-04-27T14:15:54Z

Are there any plans to fix this bug in future versions of pipenv?

matteius · 2022-04-27T16:07:12Z

Sorry @noamsgl, it is not clear which issue you are referring to.

noamsgl · 2022-04-28T09:54:44Z

Thanks @matteius for the reply.
I was using pipenv v. 2022.4.21 and was getting the same trace as Daniel and @Bananaman in #4961 (comment)

Downgrading to pipenv v. 2022.1.8 fixed the issue.
That's what I am referring to.

matteius · 2022-04-28T12:03:04Z

@noamsgl Probably you are asking about index restricted packages, and how to specify the package index going forward, is the short answer. There are several places it has been discussed already and the documentation was recently updated: https://pipenv.pypa.io/en/latest/advanced/#specifying-package-indexes

See also these for starters:

"ERROR: No matching distribution found" in updating Pipfile.lock in installing a package from private Package Repository of GitLab.com #5021
ERROR: No matching distribution found for torch==1.11.0+cu113 when using --extra-index-url #5022

rlleshi · 2022-08-15T12:50:20Z

The latest version 2022.8.15 also doesn't work.

Error text: 
ERROR: Could not find a version that satisfies the requirement torch==1.10.1+cpu (from versions: 1.7.1, 1.8.0, 1.8.1, 1.9.0, 1.9.1, 1.10.0, 1.10.1, 1.10.2, 1.11.0, 1.12.0, 1.12.1)
ERROR: No matching distribution found for torch==1.10.1+cpu

As mentioned above in the comments, the 2022.1.8 works.

matteius · 2022-08-15T13:02:43Z

@rlleshi What does your Pipfile look like?

rlleshi · 2022-08-15T13:07:50Z

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
flask = "*"
flask-restful = "*"
flask-sqlalchemy = "*"
flask-cors = "*"
flask-migrate = "*"
psycopg2 = "*"
pyyaml = "*"
python-dotenv = "*"
jsonschema = "*"

[dev-packages]
pytest = "*"
rich = "*"
pre-commit = "*"

[requires]
python_version = "3.9"

matteius · 2022-08-15T13:24:18Z

@rlleshi This is because your default index is pypi, so you need to specify the package index for torch==1.10.1+cpu For example, this package is available up to Python 3.9 -- so I specified python 3.9 and then installed via:
pipenv install torch==1.10.1+cpu --index https://download.pytorch.org/whl/

Now my Pipfile looks like:

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[[source]]
url = "https://download.pytorch.org/whl/"
verify_ssl = true
name = "downloadpytorch"

[packages]
requests = "*"
torch = {version = "==1.10.1+cpu", index = "downloadpytorch"}

[dev-packages]

[requires]
python_version = "3.9"

matteius@matteius-VirtualBox:~/pipenv-triage/pytorch$ pipenv install torch==1.10.1+cpu --index https://download.pytorch.org/whl/
Installing torch==1.10.1+cpu...
Adding torch to Pipfile's [packages]...
✔ Installation Succeeded 
Pipfile.lock not found, creating...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
✔ Success! 
Locking [dev-packages] dependencies...
Updated Pipfile.lock (8ee97b)!
Installing dependencies from Pipfile.lock (8ee97b)...
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 6/6 — 00:00:00
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.

This is the Pipfile.lock:

{
    "_meta": {
        "hash": {
            "sha256": "b82559e975590a4c92d380ba48c77e80d3ebee3be365e2ef5066443ef38ee97b"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.9"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            },
            {
                "name": "downloadpytorch",
                "url": "https://download.pytorch.org/whl/",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "certifi": {
            "hashes": [
                "sha256:84c85a9078b11105f04f3036a9482ae10e4621616db313fe045dd24743a0820d",
                "sha256:fe86415d55e84719d75f8b69414f6438ac3547d2078ab91b67e779ef69378412"
            ],
            "markers": "python_full_version >= '3.6.0'",
            "version": "==2022.6.15"
        },
        "charset-normalizer": {
            "hashes": [
                "sha256:5189b6f22b01957427f35b6a08d9a0bc45b46d3788ef5a92e978433c7a35f8a5",
                "sha256:575e708016ff3a5e3681541cb9d79312c416835686d054a23accb873b254f413"
            ],
            "markers": "python_full_version >= '3.6.0'",
            "version": "==2.1.0"
        },
        "idna": {
            "hashes": [
                "sha256:84d9dd047ffa80596e0f246e2eab0b391788b0503584e8945f2368256d2735ff",
                "sha256:9d643ff0a55b762d5cdb124b8eaa99c66322e2157b69160bc32796e824360e6d"
            ],
            "markers": "python_version >= '3.5'",
            "version": "==3.3"
        },
        "requests": {
            "hashes": [
                "sha256:7c5599b102feddaa661c826c56ab4fee28bfd17f5abca1ebbe3e7f19d7c97983",
                "sha256:8fefa2a1a1365bf5520aac41836fbee479da67864514bdb821f31ce07ce65349"
            ],
            "index": "pypi",
            "version": "==2.28.1"
        },
        "torch": {
            "hashes": [
                "sha256:062b54eda877ebe3ff7c042f3dbabe39ffadf37534d4f5ca6648b72cedc57544",
                "sha256:485b19a4f7f85c70e6ce4292a214de744553004d6aa91a894b7540cfbdf33dd8",
                "sha256:4aaf4aacfb85dc9273032d184b39c32c65f45362a0cc80c2bbb83673db6a4dd0",
                "sha256:69b4a8c01d5c36bd1d9f6d0b33162cf971504483b0dfeab196f6ef2b0dc87878",
                "sha256:85f6f9dcecc76822f3fc90cbdd3ea7014656b94b93fa74eb98c0c4ac1ef6e5b8",
                "sha256:8bed73eed3ef970aa90a5836c11d1f6723992d6a5246500372f7ecc0b203e2a5",
                "sha256:afc985ce1c1eee129ba06fd6dcb10ad5df21459f49bf696829700d3fdf78aaad",
                "sha256:c20f88e6e4a318d96eb5789cbceff764233104413132cf9230f7134104b3d252"
            ],
            "index": "downloadpytorch",
            "version": "==1.10.1+cpu"
        },
        "typing-extensions": {
            "hashes": [
                "sha256:25642c956049920a5aa49edcdd6ab1e06d7e5d467fc00e0506c44ac86fbfca02",
                "sha256:e6d2677a32f47fc7eb2795db1dd15c1f34eff616bcaf2cfb5e997f854fa1c4a6"
            ],
            "markers": "python_version >= '3.7'",
            "version": "==4.3.0"
        },
        "urllib3": {
            "hashes": [
                "sha256:c33ccba33c819596124764c23a97d25f32b28433ba0dedeb77d873a38722c9bc",
                "sha256:ea6e8fb210b19d950fab93b60c9009226c63a28808bc8386e05301e25883ac0a"
            ],
            "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4, 3.5' and python_version < '4'",
            "version": "==1.26.11"
        }
    },
    "develop": {}
}

latest pipenv, version 2022.8.15

rlleshi · 2022-08-15T14:06:48Z

aha, in the latest versions of pipenv the --extra-index-url flag has been renamed to --index.

I used pipenv install "torch==1.10.1+cpu" --extra-index-url https://download.pytorch.org/whl/

matteius · 2022-08-15T14:11:15Z

@rlleshi We removed support of the --extra-index-url because it implied the ability to search for packages across more than one index, which was suspectable to package confusion attacks. In order to solve for that, pipenv became more restrictive, such that the behavior is now to search only the default index for the packages that do not explicitly specify an alternative index. You can read more details here: https://pipenv.pypa.io/en/latest/advanced/#specifying-package-indexes

dmexs · 2022-11-23T04:20:20Z

When I run try to run pipenv install --index https://download.pytorch.org/whl/ "torch==1.13.1+cpu"

I get the following error:
ERROR: Could not find a version that satisfies the requirement torch==1.13.1+cpu (from versions: 1.4.0, 1.5.0, 1.5.1, 1.6.0, 1.7.0, 1.7.1, 1.8.0, 1.8.1, 1.9.0, 1.9.1, 1.10.0, 1.10.1, 1.10.2, 1.11.0, 1.12.0, 1.12.1, 1.13.0) ERROR: No matching distribution found for torch==1.13.1+cpu

With:
Python 3.8.5
pipenv 2022.11.11
pip 22.3.1

marcospgp · 2023-02-17T15:55:08Z

Is this functionality still broken? I could not manage to install pytorch using pipenv, with all attempts failing to find any pytorch package ("Could not find a version that satisfies the requirement" error with no available versions listed).

I tried the solutions described in this thread to no avail.

I managed to install pytorch and other dependencies using a manually created venv just fine.

matteius · 2023-02-17T16:01:54Z

@marcospgp perhaps open a new issue, but what command are you running --I just ran @dmexs command and it worked fine:

matteius@matteius-VirtualBox:~/pipenv-triage/pytorch-test$ pipenv install --index https://download.pytorch.org/whl/ "torch==1.13.1+cpu"
Creating a virtualenv for this project...
Pipfile: /home/matteius/pipenv-triage/pytorch-test/Pipfile
Using /home/matteius/.pyenv/versions/3.11.0/bin/python3 (3.11.0) to create virtualenv...
⠏ Creating virtual environment...created virtual environment CPython3.11.0.final.0-64 in 679ms
creator CPython3Posix(dest=/home/matteius/.virtualenvs/pytorch-test-sgFyUZQK, clear=False, no_vcs_ignore=False, global=False)
seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/matteius/.local/share/virtualenv)
added seed packages: pip==23.0, setuptools==67.1.0, wheel==0.38.4
activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator

✔ Successfully created virtual environment!
Virtualenv location: /home/matteius/.virtualenvs/pytorch-test-sgFyUZQK
Creating a Pipfile for this project...
Installing torch==1.13.1+cpu...
Resolving torch==1.13.1+cpu...
Installing...
Adding torch to Pipfile's [packages] ...
✔ Installation Succeeded
Pipfile.lock not found, creating...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
✔ Success!
Locking [dev-packages] dependencies...
Updated Pipfile.lock (606b65cc78e1c68b9b6dd850ff65a855cf32c97204fe46bd1f565a95d9405d15)!
Installing dependencies from Pipfile.lock (405d15)...
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.

marcospgp · 2023-02-17T16:47:06Z

@matteius

python -m pipenv install --index https://download.pytorch.org/whl/ "torch==1.13.1+cpu"
Creating a virtualenv for this project...
Pipfile: C:\Users\testuser\Documents\GitHub\pipenv-test\Pipfile
Using C:/Program Files/Python311/python.exe (3.11.1) to create virtualenv...
[==  ] Creating virtual environment...created virtual environment CPython3.11.1.final.0-64 in 4547ms
  creator CPython3Windows(dest=C:\Users\testuser\.virtualenvs\pipenv-test-JAMSETJU, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=C:\Users\testuser\AppData\Local\pypa\virtualenv)
    added seed packages: pip==23.0, setuptools==67.1.0, wheel==0.38.4
  activators BashActivator,BatchActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator

Successfully created virtual environment!
Virtualenv location: C:\Users\testuser\.virtualenvs\pipenv-test-JAMSETJU
Creating a Pipfile for this project...
Installing torch==1.13.1+cpu...
Resolving torch==1.13.1+cpu...
Installing...
[=== ] Installing torch...[31m[1mError: [0m An error occurred while installing [32mtorch==1.13.1+cpu[0m!
Error text: Looking in indexes: https://download.pytorch.org/whl/, 
https://pypi.org/simple, https://pypi.org/simple

[36mERROR: Could not find a version that satisfies the requirement torch==1.13.1+cpu (fromversions: none)
ERROR: No matching distribution found for torch==1.13.1+cpu
[0m
Installation Failed

Python 3.10.5 (then tried 3.7.0, same result)

global scope (no venv) pip freeze:

certifi==2022.12.7
distlib==0.3.6
filelock==3.9.0
pipenv==2023.2.4
platformdirs==3.0.0
virtualenv==20.19.0
virtualenv-clone==0.5.7

matteius · 2023-02-18T01:43:25Z

@marcospgp Are you running from within a virtual machine or on bare metal hardware?

Also not sure if it matters, but why are you invoking pipenv with python -m ahead of it?

These are the available distros of "torch==1.13.1+cpu":

torch-1.13.1+cpu-cp310-cp310-linux_x86_64.whl
torch-1.13.1+cpu-cp310-cp310-win_amd64.whl
torch-1.13.1+cpu-cp311-cp311-linux_x86_64.whl
torch-1.13.1+cpu-cp37-cp37m-linux_x86_64.whl
torch-1.13.1+cpu-cp37-cp37m-win_amd64.whl
torch-1.13.1+cpu-cp38-cp38-linux_x86_64.whl
torch-1.13.1+cpu-cp38-cp38-win_amd64.whl
torch-1.13.1+cpu-cp39-cp39-linux_x86_64.whl
torch-1.13.1+cpu-cp39-cp39-win_amd64.whl

Which python version and which platform and CPU are you on?

marcospgp · 2023-02-18T15:25:23Z

@matteius Running on windows 10 bare metal. I'm running pipenv through python -m because I installed it as per the instructions which as far as I understand doesn't add pipenv to the path variable.

Which python version and which platform and CPU are you on?

mentioned above, and I'm not sure CPU matters since pip install torch worked right away, but it's intel.

thinker0 · 2023-04-14T09:49:25Z

pipenv install --index https://download.pytorch.org/whl/cpu/ "torch==1.13.1"

Can I search the CPU version by doing this?

MetLee · 2023-05-08T08:27:33Z

pipenv install --index https://download.pytorch.org/whl/cpu/ "torch==1.13.1"
Can I search the CPU version by doing this?

@thinker0 pipenv install torch

scott-vsi · 2023-05-12T14:14:00Z

Something that tripped me up is that it seems torch and torchvision must be installed with the same bugfix version. For example,

torchvision = {version="==0.14.1+cu117", index="pytorch"}
torch = {version="==1.13.1+cu117", index="pytorch"}

Trying to use torchvision==0.14.0+cu117 will not work! (I know this is because torch and torchvision are very closely tied; I just didn't realize that extended to the bugfix version).

torchvision = {version="==0.14.0+cu117", index="pytorch"}
torch = {version="==1.13.1+cu117", index="pytorch"}

It fails with this error (which is not that helpful; it would be nice if it mentioned torch or torchvision):

[pipenv.exceptions.ResolutionFailure]: Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  You can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
  Hint: try $ pipenv lock --pre if it is a pre-release dependency.
ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

davidedelpapa · 2023-06-23T15:27:05Z

Since searching for "pipenv install torch" refers to this guide, can we update the first solution?
it seems (at least for the CPU version) that --extra-index-url is not working, while the command:
pipenv install --index https://download.pytorch.org/whl/cpu "torch==2.0.1+cpu"
works just fine.

twolfvb · 2023-06-27T15:44:17Z

Since searching for "pipenv install torch" refers to this guide, can we update the first solution? it seems (at least for the CPU version) that --extra-index-url is not working, while the command: pipenv install --index https://download.pytorch.org/whl/cpu "torch==2.0.1+cpu" works just fine.

That did not work for me, but

pipenv install --index https://download.pytorch.org/whl/cpu "torch==2.0.1" worked. Note that I deleted the +cpu from the package's name, but kept your url that ends with /cpu.

datalifenyc · 2023-09-23T20:42:37Z

Use the Pytorch whl download page, https://download.pytorch.org/whl/torch_stable.html, to determine the versions you want to install.

Command

pipenv install --index https://download.pytorch.org/whl/ "torch==2.0.1+cu118" "torchvision==0.15.2+cu118" "torchaudio==2.0.2+cu118"

Output

Installing torch==2.0.1+cu118...
Resolving torch==2.0.1+cu118...
Installation Succeeded
Installing torchvision==0.15.2+cu118...
Resolving torchvision==0.15.2+cu118...                                                                                         
Installation Succeeded
Installing torchaudio==2.0.2+cu118...
Resolving torchaudio==2.0.2+cu118...                                                                                           
Installation Succeeded
Pipfile.lock (343c5a) out of date, updating to (b56edd)...                                                                     
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
Success!
...
Downloading file torch-2.0.1+cu118-cp311-cp311-win_amd64.whl to obtain hash...
Downloading file torchvision-0.15.2+cu118-cp311-cp311-win_amd64.whl to obtain
hash...
Downloading file torchaudio-2.0.2+cu118-cp311-cp311-win_amd64.whl to obtain
hash...
Locking [dev-packages] dependencies...                                                                                         
Updated Pipfile.lock (<hash>)!
Installing dependencies from Pipfile.lock (<hash>)...
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.

Trezorro · 2024-01-26T11:25:39Z

Although this threat has helped me a lot, I was set back an hour because I did not know that torch is not compatible with python >3.12. Be advised.

I finally figured out the error using the compatibility matrix here: https://github.com/pytorch/pytorch/blob/main/RELEASE.md#release-compatibility-matrix

Also, even with python 3.8, I was unable to get pipenv to install pytorch 1.7.0 , which was specified by my target codebase. I still have no idea why. Running on an intel mac with pyenv and pipenv. Cpu specific commands did not help.

As a note to pipenv developers.
Either pipenv, pytorch or both should make this more obvious.
Pipenv only returned the generic error of: No distribution found. Could this not tell me what options I do have, or that it has to do with python version specifiers?

Arcitec changed the title ~~[BUG] Pipenv doesn't read all packages from very large 3rd party repo?~~ [BUG] Pipenv doesn't read all packages from very large 3rd party repo? (PyTorch) Feb 18, 2022

Arcitec closed this as completed Feb 19, 2022

Arcitec changed the title ~~[BUG] Pipenv doesn't read all packages from very large 3rd party repo? (PyTorch)~~ GUIDE: How to install PyTorch via Pipenv (and how to add other 3rd Party Repositories). Feb 19, 2022

This was referenced Feb 19, 2022

Can't use an index file for some reason #4649

Closed

Pip package URL is not a valid index pytorch/pytorch#39779

Closed

Arcitec mentioned this issue Mar 29, 2022

"ERROR: No matching distribution found" in updating Pipfile.lock in installing a package from private Package Repository of GitLab.com #5021

Closed

drewgilliam mentioned this issue Jun 6, 2022

PEP 503 facebookresearch/detectron2#4314

Open

ghost mentioned this issue Aug 10, 2022

CUDA capability sm_86 is not compatible with the current PyTorch installation lucidrains/deep-daze#203

Open

matteius added the Type: Question ❔ This is a question or a request for support. label Aug 15, 2022

tomasff mentioned this issue Oct 7, 2022

PEP 503-standard repo URL pyg-team/pytorch_geometric#5625

Open

GUIDE: How to install PyTorch via Pipenv (and how to add other 3rd Party Repositories). #4961

GUIDE: How to install PyTorch via Pipenv (and how to add other 3rd Party Repositories). #4961

Comments

Arcitec commented Feb 18, 2022 • edited Loading

Issue description

Arcitec commented Feb 19, 2022 • edited Loading

Arcitec commented Feb 19, 2022

Arcitec commented Feb 19, 2022 • edited Loading

Solved: This is how to install Torch CORRECTLY in Pipenv. Most people do this incorrectly even in various "solutions" I've seen from other people... I've researched to the bottom of this problem and found out the real, intended solution that everyone should use.

Now you may notice something: There's no ".html" in my repo filenames.

You may notice something else: I always specify a version with +cpu or +cu113 or whatever.

If you want to figure out the URL to the correct repos/versions in the future, do as follows:

One final note: ALWAYS RESPECT THE TORCH PYTHON VERSIONS.

Arcitec commented Feb 19, 2022 • edited Loading

DanielPerezJensen commented Mar 29, 2022

Arcitec commented Mar 29, 2022 • edited Loading

DanielPerezJensen commented Mar 29, 2022

Arcitec commented Mar 29, 2022 • edited Loading

DanielPerezJensen commented Mar 29, 2022

Arcitec commented Mar 29, 2022 • edited Loading

Python

Pyenv (Python Version Management)

Pipenv (Python Project Environment Management)

DanielPerezJensen commented Mar 29, 2022

DanielPerezJensen commented Mar 29, 2022 • edited Loading

Arcitec commented Mar 29, 2022 • edited Loading

DanielPerezJensen commented Mar 29, 2022 via email

Arcitec commented Mar 29, 2022 • edited Loading

DanielPerezJensen commented Mar 29, 2022

DanielPerezJensen commented Mar 29, 2022

Arcitec commented Mar 29, 2022

It's a bug in the newest version of Pipenv.

DanielPerezJensen commented Mar 29, 2022

Arcitec commented Mar 29, 2022

DanielPerezJensen commented Mar 29, 2022

DanielPerezJensen commented Mar 29, 2022 • edited Loading

Arcitec commented Mar 29, 2022 • edited Loading

noamsgl commented Apr 27, 2022

matteius commented Apr 27, 2022

noamsgl commented Apr 28, 2022

matteius commented Apr 28, 2022

rlleshi commented Aug 15, 2022 • edited Loading

matteius commented Aug 15, 2022

rlleshi commented Aug 15, 2022

matteius commented Aug 15, 2022

rlleshi commented Aug 15, 2022

matteius commented Aug 15, 2022

dmexs commented Nov 23, 2022 • edited Loading

marcospgp commented Feb 17, 2023 • edited Loading

matteius commented Feb 17, 2023

marcospgp commented Feb 17, 2023 • edited Loading

matteius commented Feb 18, 2023

marcospgp commented Feb 18, 2023

thinker0 commented Apr 14, 2023

MetLee commented May 8, 2023

scott-vsi commented May 12, 2023

davidedelpapa commented Jun 23, 2023

twolfvb commented Jun 27, 2023

datalifenyc commented Sep 23, 2023

Trezorro commented Jan 26, 2024 • edited Loading

Arcitec commented Feb 18, 2022 •

edited

Loading

Arcitec commented Feb 19, 2022 •

edited

Loading

Arcitec commented Feb 19, 2022 •

edited

Loading

You may notice something else: I always specify a version with `+cpu` or `+cu113` or whatever.

Arcitec commented Feb 19, 2022 •

edited

Loading

Arcitec commented Mar 29, 2022 •

edited

Loading

Arcitec commented Mar 29, 2022 •

edited

Loading

Arcitec commented Mar 29, 2022 •

edited

Loading

DanielPerezJensen commented Mar 29, 2022 •

edited

Loading

Arcitec commented Mar 29, 2022 •

edited

Loading

Arcitec commented Mar 29, 2022 •

edited

Loading

DanielPerezJensen commented Mar 29, 2022 •

edited

Loading

Arcitec commented Mar 29, 2022 •

edited

Loading

rlleshi commented Aug 15, 2022 •

edited

Loading

dmexs commented Nov 23, 2022 •

edited

Loading

marcospgp commented Feb 17, 2023 •

edited

Loading

marcospgp commented Feb 17, 2023 •

edited

Loading

Trezorro commented Jan 26, 2024 •

edited

Loading