Private repos and --skip-lock strange behavior #5654

scastria · 2023-04-17T00:13:57Z

Issue description

There seems to be some issue with private repos and the usage of --skip-lock. I have a private repo in AWS CodeArttifact. If I setup my Pipfile with the private repo as a [[source]] listed at the top and specify my private package with the index pointing to the private repo, pipenv install works but pipenv install --skip-lock does not. However, if I remove the [[source]] at the top and remove the index setting on the private package, but setup PIP_EXTRA_INDEX_URL as env variable in my shell, then pipenv install --skip-lock DOES work. It behaves as if using --skip-lock does not pass the credentials to the private repo to allow it to resolve.

Expected result

pipenv install --skip-lock to resolve private package

Actual result

stuff > pipenv install --skip-lock --verbose
Using python: 3.10
Path to python: /opt/homebrew/bin/python3
Creating a virtualenv for this project...
Pipfile: /Users/myusername/stuff/Pipfile
Using /opt/homebrew/bin/python3 (3.10.8) to create virtualenv...
⠙ Creating virtual environment...created virtual environment CPython3.10.8.final.0-64 in 112ms
  creator CPython3Posix(dest=/Users/myusername/.local/share/virtualenvs/stuff-4QD5uP5M, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/Users/myusername/Library/Application Support/virtualenv)
    added seed packages: pip==23.0.1, setuptools==67.6.1, wheel==0.40.0
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator

✔ Successfully created virtual environment!
Virtualenv location: /Users/myusername/.local/share/virtualenvs/stuff-4QD5uP5M
Installing dependencies from Pipfile...
Writing supplied requirement line to temporary file: 'mycompany-utils==0.0.203'
Install Phase: Standard Requirements
Preparing Installation of 'mycompany-utils'
$ /Users/myusername/.local/share/virtualenvs/stuff-4QD5uP5M/bin/python /opt/homebrew/lib/python3.10/site-packages/pipenv/patched/pip/__pip-runner__.py install -i https://aws:MYTOKEN@data-engineering-XXXXXXXXXX.d.codeartifact.us-west-2.amazonaws.com/pypi/data_engineering/simple/ --no-input --upgrade --exists-action=i -r /var/folders/qd/sndx6g1d447d2vx_rcmp2yzr0000gn/T/pipenv-_i4qbgs4-requirements/pipenv-zg2b9_x3-hashed-reqs.txt
Using source directory: '/Users/myusername/.local/share/virtualenvs/stuff-4QD5uP5M/src'
Looking in indexes: https://aws:****@data-engineering-XXXXXXXXXX.d.codeartifact.us-west-2.amazonaws.com/pypi/data_engineering/simple/

Collecting mycompany-utils==0.0.203

  Using cached https://data-engineering-XXXXXXXXXX.d.codeartifact.us-west-2.amazonaws.com/pypi/data_engineering/simple/mycompany-utils/0.0.203/mycompany_utils-0.0.203-py3-none-any.whl (50 kB)

ERROR: Could not find a version that satisfies the requirement PyYAML (from mycompany-utils) (from versions: none)
ERROR: No matching distribution found for PyYAML
An error occurred while installing mycompany-utils==0.0.203! Will try again.
Installing initially failed dependencies...
Writing supplied requirement line to temporary file: 'mycompany-utils==0.0.203'
Install Phase: Standard Requirements
Preparing Installation of 'mycompany-utils'
$ /Users/myusername/.local/share/virtualenvs/stuff-4QD5uP5M/bin/python /opt/homebrew/lib/python3.10/site-packages/pipenv/patched/pip/__pip-runner__.py install -i https://aws:MYTOKEN@data-engineering-XXXXXXXXXX.d.codeartifact.us-west-2.amazonaws.com/pypi/data_engineering/simple/ --no-input --upgrade --no-use-pep517 --exists-action=i -r /var/folders/qd/sndx6g1d447d2vx_rcmp2yzr0000gn/T/pipenv-_i4qbgs4-requirements/pipenv-eqcwadvi-hashed-reqs.txt
Using source directory: '/Users/myusername/.local/share/virtualenvs/stuff-4QD5uP5M/src'
Looking in indexes: https://aws:****@data-engineering-XXXXXXXXXX.d.codeartifact.us-west-2.amazonaws.com/pypi/data_engineering/simple/

Collecting mycompany-utils==0.0.203

  Using cached https://data-engineering-XXXXXXXXXX.d.codeartifact.us-west-2.amazonaws.com/pypi/data_engineering/simple/mycompany-utils/0.0.203/mycompany_utils-0.0.203-py3-none-any.whl (50 kB)

ERROR: Could not find a version that satisfies the requirement boto3 (from mycompany-utils) (from versions: none)
ERROR: No matching distribution found for boto3
[pipenv.exceptions.InstallError]: ERROR: Could not find a version that satisfies the requirement boto3 (from mycompany-utils) (from versions: none)
[pipenv.exceptions.InstallError]: ERROR: No matching distribution found for boto3
ERROR: Couldn't install package: [Requirement(_name='mycompany-utils', vcs=None, req=NamedRequirement(name='mycompany-utils', version='==0.0.203', req=Requirement.parse('mycompany-utils==0.0.203'), extras=[], editable=False, _parsed_line=<Line (editable=False, name=mycompany-utils, path=None, uri=None, extras=(), markers=None, vcs=None, specifier===0.0.203, pyproject=None, pyproject_requires=None, pyproject_backend=None, ireq=mycompany-utils==0.0.203)>), markers=None, _specifiers='==0.0.203', index='data_engineering', editable=False, hashes=frozenset(), extras=(), abstract_dep=None, _line_instance=<Line (editable=False, name=mycompany-utils, path=None, uri=None, extras=(), markers=None, vcs=None, specifier===0.0.203, pyproject=None, pyproject_requires=None, pyproject_backend=None, ireq=mycompany-utils==0.0.203)>, _ireq=None)]
 Package installation failed...

Steps to replicate

Provide the steps to replicate (which usually at least includes the commands and the Pipfile).

Create Pipfile with [[source]]:

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[[source]]
url = "https://aws:${CODEARTIFACT_AUTH_TOKEN}@data-engineering-XXXXXXXXX.d.codeartifact.us-west-2.amazonaws.com/pypi/data_engineering/simple/"
verify_ssl = true
name = "data_engineering"

[packages]
mycompany_utils = {version="==0.0.203", index="data_engineering"}

[dev-packages]

[requires]
python_version = "3.10"

export CODEARTIFACT_AUTH_TOKEN with valid AWS codeartifact token
pipenv install --skip-lock
FAILURE
Edit Pipfile to NOT use [[source]], but rely on a PIP_EXTRA_INDEX_URL:

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
greenstreet_utils = "==0.0.203"

[dev-packages]

[requires]
python_version = "3.10"

pipenv --rm
export PIP_EXTRA_INDEX_URL with valid AWS codeartifact URL with valid token
pipenv install --skip-lock
SUCCESS

Please run $ pipenv --support, and paste the results here. Don't put backticks (`) around it! The output already contains Markdown formatting.

$ pipenv --support

Pipenv version: '2023.3.20'

Pipenv location: '/opt/homebrew/lib/python3.10/site-packages/pipenv'

Python location: '/opt/homebrew/opt/[email protected]/bin/python3.10'

OS Name: 'posix'

User pip version: '23.0.1'

user Python installations found:

3.10.8: /opt/homebrew/bin/python3
3.10.8: /usr/local/bin/python
3.9.15: /opt/homebrew/bin/python3.9
3.9.6: /usr/bin/python3
3.8.15: /opt/homebrew/bin/python3.8

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.10.8',
 'os_name': 'posix',
 'platform_machine': 'arm64',
 'platform_python_implementation': 'CPython',
 'platform_release': '22.4.0',
 'platform_system': 'Darwin',
 'platform_version': 'Darwin Kernel Version 22.4.0: Mon Mar  6 20:59:28 PST '
                     '2023; root:xnu-8796.101.5~3/RELEASE_ARM64_T6000',
 'python_full_version': '3.10.8',
 'python_version': '3.10',
 'sys_platform': 'darwin'}

System environment variables:

MANPATH
TERM_PROGRAM
SHELL
TERM
HOMEBREW_REPOSITORY
TMPDIR
GOBIN
TERM_PROGRAM_VERSION
ADOPS_PAT
OLDPWD
TERM_SESSION_ID
USER
LDAP_USERNAME
CODEARTIFACT_AUTH_TOKEN
SSH_AUTH_SOCK
HOMEBREW_NO_AUTO_UPDATE
PATH
__CFBundleIdentifier
PWD
JAVA_HOME
LANG
LDAP_PASSWORD
XPC_FLAGS
PS1
XPC_SERVICE_NAME
SHLVL
HOME
SHELL_SESSION_HISTORY
HOMEBREW_PREFIX
LOGNAME
GOPATH
DEV_CLIENT_ID
INFOPATH
HOMEBREW_CELLAR
BITBUCKET_PAT
DEV_CLIENT_SECRET
_
__CF_USER_TEXT_ENCODING
PIP_DISABLE_PIP_VERSION_CHECK
PIP_PYTHON_PATH
PYTHONDONTWRITEBYTECODE
PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv–specific environment variables:

Debug–specific environment variables:

PATH: /opt/homebrew/bin:/opt/homebrew/sbin:/Users/myusername/.go/bin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/share/dotnet:~/.dotnet/tools:/Library/Apple/usr/bin:/Library/Frameworks/Mono.framework/Versions/Current/Commands:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin
SHELL: /bin/bash
LANG: en_US.UTF-8
PWD: /Users/myusername/stuff

The text was updated successfully, but these errors were encountered:

matteius · 2023-04-17T00:54:58Z

@scastria Can you try adding to your Pipfile:

[pipenv]
install_search_all_sources = true

More details: https://pipenv.pypa.io/en/latest/indexes/

scastria · 2023-04-17T00:59:36Z

@matteius Unfortunately, that does not make a difference. Also, just as a reminder, running WITHOUT --skip-lock works fine even without the above install_search_all suggestion

matteius · 2023-04-17T01:03:28Z

@scastria side question -- why even use --skip-lock? It bypasses most of the benefits and security features of pipenv.

What if you leave the install_search_all suggestion and swap the order of your sources so your private source is the default? You would have to specify anything that needs to come from pypi though. What I am wondering is if your private package is requiring another private package and then failing to resolve it.

Additionally, --skip-lock is broken in general, have you tried the patch branch for this issue? #5653

scastria · 2023-04-17T01:09:52Z

@matteius That is a good question. My company started using pipenv last year around May and back then, it would sometimes take 10 minutes to run a pipenv install. We discovered that if we added --skip-lock, it only took 30 seconds or less. Therefore, we all got in the habit of avoiding the lock file and have never looked back. But I do agree that we are bypassing a lot of the great features of pipenv by doing so.

Our private package only depends on public packages.

Changing the default repo/index and specifying public packages to come from pypi would be too big of an impact on all of our repos. Let me try the patch branch...

matteius · 2023-04-17T01:18:01Z

@scastria Yeah, I personally would like to remove the --skip-lock but I think there would be some push back so I am going to hold on that for now. There have been some big performance improvements in pipenv since the time your company started using pipenv. I suspect based on the timeline of my development, that the issue here is --skip-lock never properly accounted for source specifications in the Pipfile (I am speculating) and I'll take that a step further and say pip 23.x made some changes that maybe compounded this fact.

When I originally did my work on index restricted packages, I mostly ignored the code paths for skip-lock because as a feature it didn't make sense to me, and working from the Pipfile is a whole different beast than working from the lockfile in so many ways. Let me know though if that skip lock branch doesn't resolve it and I can tag this issue appropriately.

scastria · 2023-04-17T01:19:05Z

@matteius Not sure I did it right, but I installed the patch branch with this??

pip install git+https://github.com/pypa/pipenv.git@issue-5367-fix

That gave me version: 2023.3.19.dev0

Using this version had no effect with my original Pipfile.

However, it DID work after adding the install_search_all setting!

matteius · 2023-04-17T01:20:59Z

@scastria Cool, so the take away there is that pip 23.x basically caused users to required the install_search_all_sources = true setting, I've seen a number of reports resolved with that after we vendor'd in pip 23 this year. The main take-away is skip-lock was broken since at least September when I did the named package categories work and refactor.

matteius · 2023-04-17T01:23:21Z

@scastria Also would love to implore you to see if the performance of pipenv sync is better for you in newer versions and just run with the lockfile. All of the python package managers have been improving their performance over the last year: https://lincolnloop.github.io/python-package-manager-shootout/

scastria · 2023-04-17T01:25:47Z

Thanks, I will try.

matteius · 2023-04-18T10:49:19Z

Expecting a release in the next week.

matteius added the PR: awaiting-review The PR related to this issue is awaiting review by a maintainer. label Apr 17, 2023

matteius mentioned this issue Apr 17, 2023

Fix issue with skip-lock #5653

Merged

2 tasks

matteius closed this as completed Apr 18, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Private repos and --skip-lock strange behavior #5654

Private repos and --skip-lock strange behavior #5654

scastria commented Apr 17, 2023 •

edited

Loading

matteius commented Apr 17, 2023

scastria commented Apr 17, 2023 •

edited

Loading

matteius commented Apr 17, 2023

scastria commented Apr 17, 2023

matteius commented Apr 17, 2023

scastria commented Apr 17, 2023

matteius commented Apr 17, 2023

matteius commented Apr 17, 2023

scastria commented Apr 17, 2023

matteius commented Apr 18, 2023

Private repos and --skip-lock strange behavior #5654

Private repos and --skip-lock strange behavior #5654

Comments

scastria commented Apr 17, 2023 • edited Loading

Issue description

Expected result

Actual result

Steps to replicate

matteius commented Apr 17, 2023

scastria commented Apr 17, 2023 • edited Loading

matteius commented Apr 17, 2023

scastria commented Apr 17, 2023

matteius commented Apr 17, 2023

scastria commented Apr 17, 2023

matteius commented Apr 17, 2023

matteius commented Apr 17, 2023

scastria commented Apr 17, 2023

matteius commented Apr 18, 2023

scastria commented Apr 17, 2023 •

edited

Loading

scastria commented Apr 17, 2023 •

edited

Loading