Can not install azure-cli using pip install --pre azure-cli --extra-index-url https://azurecliprod.blob.core.windows.net/edge

[sarwatwplayfab]

az feedback auto-generates most of the information requested below, as of CLI version 2.0.62

Describe the bug
We recently started seeing an issue with the following:

pip install --pre azure-cli --extra-index-url https://azurecliprod.blob.core.windows.net/edge
This was running fine till around a week or so ago but now we see that pip is always looking for the correct verison of urllib3 and other libraries. This goes on FOREVER until the timeout is reached for the pipeline and then the pipeline fails. I think this is related to the latest pip version and backtracking issue but I am not sure how to resolve this.

Any ideas how I can solve this?

Log from the pipeline that is running this command
Requirement already satisfied: msrest>=0.6.21 in /opt/hostedtoolcache/Python/3.9.5/x64/lib/python3.9/site-packages (from azure-keyvault-administration==4.0.0b3->azure-cli) (0.6.21)

Requirement already satisfied: azure-core<2.0.0,>=1.7.0 in /opt/hostedtoolcache/Python/3.9.5/x64/lib/python3.9/site-packages (from azure-keyvault-administration==4.0.0b3->azure-cli) (1.12.0)

Collecting msrestazure<2.0.0,>=0.4.32

Downloading msrestazure-0.6.4-py2.py3-none-any.whl (40 kB)

Collecting python-dateutil<3,>=2.1.0

Downloading python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB)

Requirement already satisfied: cffi in /opt/hostedtoolcache/Python/3.9.5/x64/lib/python3.9/site-packages (from azure-datalake-store~=0.0.49->azure-cli) (1.13.2)

Collecting vsts

Downloading vsts-0.1.25-py3-none-any.whl (2.3 MB)

Collecting jinja2

Downloading Jinja2-3.0.1-py3-none-any.whl (133 kB)

Collecting azure-nspkg>=2.0.0

Downloading azure_nspkg-3.0.2-py3-none-any.whl (1.5 kB)

Collecting azure-mgmt-nspkg>=2.0.0

Downloading azure_mgmt_nspkg-3.0.2-py3-none-any.whl (1.6 kB)

Collecting azure-mgmt-datalake-nspkg>=2.0.0

Downloading azure_mgmt_datalake_nspkg-3.0.1-py3-none-any.whl (1.7 kB)

Requirement already satisfied: pycparser in /opt/hostedtoolcache/Python/3.9.5/x64/lib/python3.9/site-packages (from cffi->azure-datalake-store~=0.0.49->azure-cli) (2.19)

Collecting pathlib2

Downloading pathlib2-2.3.5-py2.py3-none-any.whl (18 kB)

Collecting invoke<2.0,>=1.3

Downloading invoke-1.5.0-py3-none-any.whl (211 kB)

Requirement already satisfied: pyyaml in /opt/hostedtoolcache/Python/3.9.5/x64/lib/python3.9/site-packages (from knack~=0.8.2->azure-cli-core==2.25.0.post20210615155724->azure-cli) (5.3)

Collecting tabulate

Downloading tabulate-0.8.9-py3-none-any.whl (25 kB)

Collecting pygments

Downloading Pygments-2.9.0-py3-none-any.whl (1.0 MB)

Collecting colorama

Downloading colorama-0.4.4-py2.py3-none-any.whl (16 kB)

Collecting PyJWT[crypto]<3,>=1.0.0

Downloading PyJWT-2.1.0-py3-none-any.whl (16 kB)

Requirement already satisfied: isodate>=0.6.0 in /opt/hostedtoolcache/Python/3.9.5/x64/lib/python3.9/site-packages (from msrest>=0.6.21->azure-keyvault-administration==4.0.0b3->azure-cli) (0.6.0)

Requirement already satisfied: certifi>=2017.4.17 in /opt/hostedtoolcache/Python/3.9.5/x64/lib/python3.9/site-packages (from msrest>=0.6.21->azure-keyvault-administration==4.0.0b3->azure-cli) (2019.11.28)

Requirement already satisfied: requests-oauthlib>=0.5.0 in /opt/hostedtoolcache/Python/3.9.5/x64/lib/python3.9/site-packages (from msrest>=0.6.21->azure-keyvault-administration==4.0.0b3->azure-cli) (1.3.0)

Requirement already satisfied: pyparsing>=2.0.2 in /opt/hostedtoolcache/Python/3.9.5/x64/lib/python3.9/site-packages (from packaging~=20.9->azure-cli) (2.4.7)

Collecting bcrypt>=3.1.3

Downloading bcrypt-3.2.0-cp36-abi3-manylinux2010_x86_64.whl (63 kB)

Collecting pynacl>=1.0.1

Downloading PyNaCl-1.4.0-cp35-abi3-manylinux1_x86_64.whl (961 kB)

Collecting PyJWT[crypto]<3,>=1.0.0

Downloading PyJWT-2.0.1-py3-none-any.whl (15 kB)

Downloading PyJWT-2.0.0-py3-none-any.whl (15 kB)

Downloading PyJWT-2.0.0a2-py3-none-any.whl (15 kB)

Downloading PyJWT-2.0.0a1-py3-none-any.whl (16 kB)

Requirement already satisfied: idna<2.9,>=2.5 in /opt/hostedtoolcache/Python/3.9.5/x64/lib/python3.9/site-packages (from requests~=2.22->azure-cli-core==2.25.0.post20210615155724->azure-cli) (2.8)

Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /opt/hostedtoolcache/Python/3.9.5/x64/lib/python3.9/site-packages (from requests~=2.22->azure-cli-core==2.25.0.post20210615155724->azure-cli) (1.25.8)

Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /opt/hostedtoolcache/Python/3.9.5/x64/lib/python3.9/site-packages (from requests~=2.22->azure-cli-core==2.25.0.post20210615155724->azure-cli) (3.0.4)

Requirement already satisfied: oauthlib>=3.0.0 in /opt/hostedtoolcache/Python/3.9.5/x64/lib/python3.9/site-packages (from requests-oauthlib>=0.5.0->msrest>=0.6.21->azure-keyvault-administration==4.0.0b3->azure-cli) (3.1.0)

INFO: pip is looking at multiple versions of urllib3 to determine which version is compatible with other requirements. This could take a while.

Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1

Downloading urllib3-1.25.11-py2.py3-none-any.whl (127 kB)

Downloading urllib3-1.25.10-py2.py3-none-any.whl (127 kB)

Downloading urllib3-1.25.9-py2.py3-none-any.whl (126 kB)

Downloading urllib3-1.25.7-py2.py3-none-any.whl (125 kB)

Downloading urllib3-1.25.6-py2.py3-none-any.whl (125 kB)

Downloading urllib3-1.25.5-py2.py3-none-any.whl (125 kB)

Downloading urllib3-1.25.4-py2.py3-none-any.whl (125 kB)

Downloading urllib3-1.25.3-py2.py3-none-any.whl (150 kB)

To Reproduce
In an azure pipeline (yaml) file, write a script and run:
pip install --pre azure-cli --extra-index-url https://azurecliprod.blob.core.windows.net/edge

Yaml setup details
2021-06-15T19:15:20.0479257Z ##[section]Starting: Initialize job
2021-06-15T19:15:20.0480963Z Agent name: 'Hosted Agent'
2021-06-15T19:15:20.0481450Z Agent machine name: 'fv-az40-742'
2021-06-15T19:15:20.0481770Z Current agent version: '2.187.2'
2021-06-15T19:15:20.0544581Z ##[group]Operating System
2021-06-15T19:15:20.0545069Z Ubuntu
2021-06-15T19:15:20.0545303Z 20.04.2
2021-06-15T19:15:20.0545519Z LTS
2021-06-15T19:15:20.0545772Z ##[endgroup]
2021-06-15T19:15:20.0546057Z ##[group]Virtual Environment
2021-06-15T19:15:20.0546409Z Environment: ubuntu-20.04
2021-06-15T19:15:20.0546712Z Version: 20210606.1

Expected behavior
Azure cli is installed

Environment summary
Linux based pipeline

Additional context
This was working fine around 2 weeks ago but stopped working and just over the weekend (June 18th onwards) started working again. I am not sure if this is working temporarily.

[yonzhan]

installation

[jiasli]

Per these lines:

INFO: pip is looking at multiple versions of urllib3 to determine which version is compatible with other requirements. This could take a while.

Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1

This is a dependency from old versions of requests (similar to #18499 (comment)). Have you requested old versions of requests to be installed? I explicitly bumped requests to requests~=2.25.1 in #18518 to eliminate possible urllib3 dependency conflicts.

I tested with a clean virtual environment but couldn't repro.

[sarwatwplayfab]

How do you even request old versions of requests to be installed?

[jiasli]

Something like requests~=2.24.0. Also I am not sure if your python already has a build-in requests. Could you run pip list before installing Azure CLI?

[sarwatwplayfab]

pip list shows:
2021-06-23T05:46:32.9550595Z [command]/usr/bin/bash --noprofile --norc /home/vsts/work/_temp/0acddf70-9d3d-486c-97b0-db3dd06dc294.sh
2021-06-23T05:46:33.2524592Z Package Version
2021-06-23T05:46:33.2525700Z ------------------------ ----------
2021-06-23T05:46:33.2526118Z attrs 19.3.0
2021-06-23T05:46:33.2526595Z azure-core 1.12.0
2021-06-23T05:46:33.2527037Z azure-storage-blob 12.8.0
2021-06-23T05:46:33.2527647Z azure-storage-file-share 12.0.0
2021-06-23T05:46:33.2528090Z azure-storage-queue 12.1.0
2021-06-23T05:46:33.2528404Z certifi 2019.11.28
2021-06-23T05:46:33.2528705Z cffi 1.13.2
2021-06-23T05:46:33.2528996Z chardet 3.0.4
2021-06-23T05:46:33.2529285Z cryptography 2.8
2021-06-23T05:46:33.2529553Z idna 2.8
2021-06-23T05:46:33.2529844Z isodate 0.6.0
2021-06-23T05:46:33.2530136Z jsonschema 3.2.0
2021-06-23T05:46:33.2530576Z more-itertools 8.3.0
2021-06-23T05:46:33.2530888Z msrest 0.6.21
2021-06-23T05:46:33.2531190Z oauthlib 3.1.0
2021-06-23T05:46:33.2531684Z packaging 20.4
2021-06-23T05:46:33.2531963Z pip 21.1.1
2021-06-23T05:46:33.2532252Z pluggy 0.13.1
2021-06-23T05:46:33.2532700Z py 1.8.1
2021-06-23T05:46:33.2532989Z pycparser 2.19
2021-06-23T05:46:33.2533451Z pyfakefs 4.0.2
2021-06-23T05:46:33.2533749Z pyparsing 2.4.7
2021-06-23T05:46:33.2534049Z pyrsistent 0.16.0
2021-06-23T05:46:33.2534348Z pytest 5.4.3
2021-06-23T05:46:33.2534646Z pytz 2019.3
2021-06-23T05:46:33.2535128Z PyYAML 5.3
2021-06-23T05:46:33.2535434Z requests 2.22.0
2021-06-23T05:46:33.2535899Z requests-oauthlib 1.3.0
2021-06-23T05:46:33.2536225Z setuptools 56.0.0
2021-06-23T05:46:33.2536538Z six 1.14.0
2021-06-23T05:46:33.2536850Z urllib3 1.25.8
2021-06-23T05:46:33.2537166Z wcwidth 0.2.4
2021-06-23T05:46:33.5576031Z WARNING: You are using pip version 21.1.1; however, version 21.1.2 is available.

[jiasli]

Your build-in python has requests pre-installed:

2021-06-23T05:46:33.2535434Z requests 2.22.0

which requires

https://github.com/psf/requests/blob/v2.22.0/setup.py#L47

    'urllib3>=1.21.1,<1.26,!=1.25.0,!=1.25.1',

This conflicts with Azure CLI's dependency:

azure-cli/src/azure-cli-core/setup.py

Line 62 in 162be08

'urllib3[secure]>=1.26.5',

pip install won't automatically upgrade existing packages (request) without --upgrade flag, so it fails to resolve the dependency and goes into almost endless backtracking.

azure-cli-core now forces

azure-cli/src/azure-cli-core/setup.py

Line 60 in afbc53a

'requests~=2.25.1',

Now the issue should have been resolved by #18518.

[sarwatwplayfab]

I am having this issue pop up again. When my pipeline tries to install Azure-cli, I get the same issue:

Collecting azure-mgmt-containerinstance~=1.4
Downloading azure_mgmt_containerinstance-1.5.0-py2.py3-none-any.whl (96 kB)
Collecting azure-cli-core==2.27.0.post20210803065115
Downloading https://azurecliprod.blob.core.windows.net/edge/azure-cli-core/azure_cli_core-2.27.0.post20210803065115-py3-none-any.whl (179 kB)
Collecting azure-mgmt-reservations==0.6.0
Downloading azure_mgmt_reservations-0.6.0-py2.py3-none-any.whl (33 kB)
Collecting azure-mgmt-hdinsight~=8.0.0
Downloading azure_mgmt_hdinsight-8.0.0-py2.py3-none-any.whl (109 kB)
Collecting azure-mgmt-managedservices~=1.0
Downloading azure_mgmt_managedservices-1.0.0-py2.py3-none-any.whl (38 kB)
Collecting scp~=0.13.2
Downloading scp-0.13.6-py2.py3-none-any.whl (8.2 kB)
Collecting azure-multiapi-storage~=0.6.2
Downloading azure_multiapi_storage-0.6.2-py2.py3-none-any.whl (4.3 MB)
Collecting azure-mgmt-redis~=13.0.0
Downloading azure_mgmt_redis-13.0.0-py2.py3-none-any.whl (75 kB)
Collecting azure-mgmt-storage~=18.0.0
Downloading azure_mgmt_storage-18.0.0-py2.py3-none-any.whl (1.6 MB)
Collecting urllib3[secure]>=1.26.5
Downloading urllib3-1.26.6-py2.py3-none-any.whl (138 kB)
Collecting azure-cli-telemetry==1.0.6.*
Downloading https://azurecliprod.blob.core.windows.net/edge/azure-cli-telemetry/azure_cli_telemetry-1.0.6.post20210803065115-py3-none-any.whl (11 kB)
Collecting pyopenssl>=17.1.0
Downloading pyOpenSSL-20.0.1-py2.py3-none-any.whl (54 kB)
Collecting cryptography<3.4,>=3.2
Downloading cryptography-3.3.2-cp36-abi3-manylinux2010_x86_64.whl (2.6 MB)
Collecting humanfriendly<10.0,>=4.7
Downloading humanfriendly-9.2-py2.py3-none-any.whl (86 kB)
Collecting azure-mgmt-core<1.3.0,>=1.2.0
Downloading azure_mgmt_core-1.2.2-py2.py3-none-any.whl (21 kB)
Requirement already satisfied: six~=1.12 in /opt/hostedtoolcache/Python/3.9.6/x64/lib/python3.9/site-packages (from azure-cli-core==2.27.0.post20210803065115->azure-cli) (1.14.0)
Collecting paramiko<3.0.0,>=2.0.8
Downloading paramiko-2.7.2-py2.py3-none-any.whl (206 kB)
Collecting knack~=0.8.2
Downloading knack-0.8.2-py3-none-any.whl (59 kB)
Collecting msal<2.0.0,>=1.10.0
Downloading msal-1.13.0-py2.py3-none-any.whl (67 kB)
Collecting psutil~=5.8
Downloading psutil-5.8.0-cp39-cp39-manylinux2010_x86_64.whl (293 kB)
Collecting azure-common~=1.1
Downloading azure_common-1.1.27-py2.py3-none-any.whl (12 kB)
Collecting pkginfo>=1.5.0.1
Downloading pkginfo-1.7.1-py2.py3-none-any.whl (25 kB)
Collecting PyJWT>=2.1.0
Downloading PyJWT-2.1.0-py3-none-any.whl (16 kB)
Collecting jmespath
Downloading jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
Collecting adal~=1.2.7
Downloading adal-1.2.7-py2.py3-none-any.whl (55 kB)
Collecting requests[socks]=2.25.1
Downloading requests-2.25.1-py2.py3-none-any.whl (61 kB)
Collecting argcomplete=1.8
Downloading argcomplete-1.12.3-py2.py3-none-any.whl (38 kB)
Collecting portalocker~=1.6
Downloading portalocker-1.7.1-py2.py3-none-any.whl (10 kB)
Collecting applicationinsights<0.12,>=0.11.1

Collecting invoke<2.0,>=1.3
Downloading invoke-1.6.0-py3-none-any.whl (212 kB)
Collecting pathlib2
Downloading pathlib2-2.3.6-py2.py3-none-any.whl (17 kB)
Collecting pygments
Downloading Pygments-2.9.0-py3-none-any.whl (1.0 MB)
Collecting tabulate
Downloading tabulate-0.8.9-py3-none-any.whl (25 kB)
Requirement already satisfied: pyyaml in /opt/hostedtoolcache/Python/3.9.6/x64/lib/python3.9/site-packages (from knack~=0.8.2->azure-cli-core==2.27.0.post20210803065115->azure-cli) (5.3)
Collecting colorama
Downloading colorama-0.4.4-py2.py3-none-any.whl (16 kB)
Requirement already satisfied: certifi>=2017.4.17 in /opt/hostedtoolcache/Python/3.9.6/x64/lib/python3.9/site-packages (from msrest>=0.6.21->azure-keyvault-administration==4.0.0b3->azure-cli) (2019.11.28)
Requirement already satisfied: requests-oauthlib>=0.5.0 in /opt/hostedtoolcache/Python/3.9.6/x64/lib/python3.9/site-packages (from msrest>=0.6.21->azure-keyvault-administration==4.0.0b3->azure-cli) (1.3.0)
Requirement already satisfied: isodate>=0.6.0 in /opt/hostedtoolcache/Python/3.9.6/x64/lib/python3.9/site-packages (from msrest>=0.6.21->azure-keyvault-administration==4.0.0b3->azure-cli) (0.6.0)
Requirement already satisfied: pyparsing>=2.0.2 in /opt/hostedtoolcache/Python/3.9.6/x64/lib/python3.9/site-packages (from packaging~=20.9->azure-cli) (2.4.7)
Collecting bcrypt>=3.1.3
Downloading bcrypt-3.2.0-cp36-abi3-manylinux2010_x86_64.whl (63 kB)
Collecting deprecated
Downloading Deprecated-1.2.12-py2.py3-none-any.whl (9.5 kB)
Requirement already satisfied: idna<2.9,>=2.5 in /opt/hostedtoolcache/Python/3.9.6/x64/lib/python3.9/site-packages (from requests<3,>=2.0.0->adal~=1.2.7->azure-cli-core==2.27.0.post20210803065115->azure-cli) (2.8)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /opt/hostedtoolcache/Python/3.9.6/x64/lib/python3.9/site-packages (from requests<3,>=2.0.0->adal~=1.2.7->azure-cli-core==2.27.0.post20210803065115->azure-cli) (3.0.4)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /opt/hostedtoolcache/Python/3.9.6/x64/lib/python3.9/site-packages (from requests<3,>=2.0.0->adal~=1.2.7->azure-cli-core==2.27.0.post20210803065115->azure-cli) (1.25.8)
Requirement already satisfied: oauthlib>=3.0.0 in /opt/hostedtoolcache/Python/3.9.6/x64/lib/python3.9/site-packages (from requests-oauthlib>=0.5.0->msrest>=0.6.21->azure-keyvault-administration==4.0.0b3->azure-cli) (3.1.0)
Collecting PySocks!=1.5.7,>=1.5.6
Downloading PySocks-1.7.1-py3-none-any.whl (16 kB)
Collecting urllib3[secure]>=1.26.5
Downloading urllib3-1.26.5-py2.py3-none-any.whl (138 kB)
INFO: pip is looking at multiple versions of urllib3 to determine which version is compatible with other requirements. This could take a while.
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1
Downloading urllib3-1.25.11-py2.py3-none-any.whl (127 kB)
Downloading urllib3-1.25.10-py2.py3-none-any.whl (127 kB)
Downloading urllib3-1.25.9-py2.py3-none-any.whl (126 kB)
Downloading urllib3-1.25.7-py2.py3-none-any.whl (125 kB)
Downloading urllib3-1.25.6-py2.py3-none-any.whl (125 kB)
Downloading urllib3-1.25.5-py2.py3-none-any.whl (125 kB)
Downloading urllib3-1.25.4-py2.py3-none-any.whl (125 kB)
Downloading urllib3-1.25.3-py2.py3-none-any.whl (150 kB)
Downloading urllib3-1.25.2-py2.py3-none-any.whl (150 kB)
Downloading urllib3-1.24.3-py2.py3-none-any.whl (118 kB)
Downloading urllib3-1.24.2-py2.py3-none-any.whl (131 kB)
Downloading urllib3-1.24.1-py2.py3-none-any.whl (118 kB)
Downloading urllib3-1.24-py2.py3-none-any.whl (117 kB)
Downloading urllib3-1.23-py2.py3-none-any.whl (133 kB)
Downloading urllib3-1.22-py2.py3-none-any.whl (132 kB)
Downloading urllib3-1.21.1-py2.py3-none-any.whl (131 kB)
INFO: pip is looking at multiple versions of urllib3 to determine which version is compatible with other requirements. This could take a while.
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
INFO: pip is looking at multiple versions of sshtunnel to determine which version is compatible with other requirements. This could take a while.
Collecting sshtunnel~=0.1.4
Downloading sshtunnel-0.1.4.tar.gz (48 kB)
INFO: pip is looking at multiple versions of six to determine which version is compatible with other requirements. This could take a while.
Collecting six~=1.12
Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
Downloading six-1.15.0-py2.py3-none-any.whl (10 kB)
Downloading six-1.13.0-py2.py3-none-any.whl (10 kB)
Downloading six-1.12.0-py2.py3-none-any.whl (10 kB)
INFO: pip is looking at multiple versions of sshtunnel to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of scp to determine which version is compatible with other requirements. This could take a while.