Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pipfile.lock out of date, but installation uses --system... re-building lockfile must happen in isolation. #2273

Closed
max-arnold opened this issue May 28, 2018 · 17 comments
Labels
Category: Docker Issue affects docker builds. Type: Bug 🐛 This issue is a bug. Type: Regression This issue is a regression of a previous behavior.

Comments

@max-arnold
Copy link

max-arnold commented May 28, 2018

pipenv complains about installation using --system flag and does not update lock file on subsequent package installs.

$ python -m pipenv.help output

Pipenv version: '2018.05.18'

Pipenv location: '/Users/user/.virtualenvs/quicktest/lib/python3.6/site-packages/pipenv'

Python location: '/Users/user/.virtualenvs/quicktest/bin/python'

Other Python installations in PATH:

  • 2.6: /opt/local/bin/python2.6

  • 2.6: /opt/local/bin/python2.6

  • 2.7: /opt/local/bin/python2.7

  • 2.7: /opt/local/bin/python2.7

  • 2.7: /usr/bin/python2.7

  • 3.6: /opt/local/bin/python3.6m

  • 3.6: /Users/user/.virtualenvs/quicktest/bin/python3.6

  • 3.6: /opt/local/bin/python3.6

  • 3.6.5: /Users/user/.virtualenvs/quicktest/bin/python

  • 3.6.5: /opt/local/bin/python

  • 2.7.10: /usr/bin/python

  • 2.7.15: /opt/local/bin/python2

  • 3.6.5: /Users/user/.virtualenvs/quicktest/bin/python3

  • 3.6.5: /opt/local/bin/python3

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.6.5',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '17.5.0',
 'platform_system': 'Darwin',
 'platform_version': 'Darwin Kernel Version 17.5.0: Fri Apr 13 19:32:32 PDT '
                     '2018; root:xnu-4570.51.2~1/RELEASE_X86_64',
 'python_full_version': '3.6.5',
 'python_version': '3.6',
 'sys_platform': 'darwin'}

System environment variables:

  • TERM_SESSION_ID
  • SSH_AUTH_SOCK
  • COLORFGBG
  • ITERM_PROFILE
  • XPC_FLAGS
  • LANG
  • PWD
  • SHELL
  • TERM_PROGRAM_VERSION
  • TERM_PROGRAM
  • PATH
  • DISPLAY
  • COLORTERM
  • COMMAND_MODE
  • TERM
  • HOME
  • TMPDIR
  • USER
  • XPC_SERVICE_NAME
  • LOGNAME
  • SHLVL
  • OLDPWD
  • HISTSIZE
  • HISTFILE
  • SAVEHIST
  • WORDCHARS
  • EDITOR
  • DJROOT
  • PYROOT
  • PYLROOT
  • JAVA_HOME
  • FPATH
  • VIRTUALENV_PYTHON
  • VIRTUALENVWRAPPER_PROJECT_FILENAME
  • VIRTUALENVWRAPPER_WORKON_CD
  • VIRTUALENVWRAPPER_SCRIPT
  • WORKON_HOME
  • VIRTUALENVWRAPPER_HOOK_DIR
  • DOCBOOK_XSL_URL
  • XML_CATALOG_FILES
  • ES_HOME
  • GOPATH
  • NVM_DIR
  • PS1
  • VIRTUAL_ENV
  • _
  • PYTHONDONTWRITEBYTECODE
  • PIP_PYTHON_PATH

Pipenv–specific environment variables:

Debug–specific environment variables:

  • PATH: /Users/user/.virtualenvs/quicktest/bin:/opt/local/bin:/opt/local/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin
  • SHELL: /opt/local/bin/zsh
  • LANG: ru_RU.UTF-8
  • PWD: /tmp
  • VIRTUAL_ENV: /Users/user/.virtualenvs/quicktest

Contents of Pipfile ('/private/tmp/Pipfile'):

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
pygments = "*"
fabric = "*"

[dev-packages]

[requires]
python_version = "3.6"

Contents of Pipfile.lock ('/private/tmp/Pipfile.lock'):

{
    "_meta": {
        "hash": {
            "sha256": "4fef9e909141947fb573b0345f2129099de0f89c3f36a5b8164a15fcc0fea9e8"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.6"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "pygments": {
            "hashes": [
                "sha256:78f3f434bcc5d6ee09020f92ba487f95ba50f1e3ef83ae96b9d5ffa1bab25c5d",
                "sha256:dbae1046def0efb574852fab9e90209b23f556367b5a320c0bcb871c77c3e8cc"
            ],
            "index": "pypi",
            "version": "==2.2.0"
        }
    },
    "develop": {}
}
Steps to replicate
% mkvirtualenv quicktest
Running virtualenv with interpreter /opt/local/bin/python3.6
Using base prefix '/opt/local/Library/Frameworks/Python.framework/Versions/3.6'
New python executable in /Users/user/.virtualenvs/quicktest/bin/python3.6
Also creating executable in /Users/user/.virtualenvs/quicktest/bin/python
Installing setuptools, pip, wheel...done.
virtualenvwrapper.user_scripts creating /Users/user/.virtualenvs/quicktest/bin/predeactivate
virtualenvwrapper.user_scripts creating /Users/user/.virtualenvs/quicktest/bin/postdeactivate
virtualenvwrapper.user_scripts creating /Users/user/.virtualenvs/quicktest/bin/preactivate
virtualenvwrapper.user_scripts creating /Users/user/.virtualenvs/quicktest/bin/postactivate
virtualenvwrapper.user_scripts creating /Users/user/.virtualenvs/quicktest/bin/get_env_details
Requirement already up-to-date: pip in /Users/user/.virtualenvs/quicktest/lib/python3.6/site-packages (10.0.1)
Collecting pipenv
  Using cached https://files.pythonhosted.org/packages/8d/fe/4faa519acfb984015dde4e2973d89e47baf6a9cd81e9d58e2f2d3c47fb6f/pipenv-2018.5.18-py3-none-any.whl
Collecting virtualenv-clone>=0.2.5 (from pipenv)
  Using cached https://files.pythonhosted.org/packages/6d/c2/dccb5ccf599e0c5d1eea6acbd058af7a71384f9740179db67a9182a24798/virtualenv_clone-0.3.0-py2.py3-none-any.whl
Collecting certifi (from pipenv)
  Using cached https://files.pythonhosted.org/packages/7c/e6/92ad559b7192d846975fc916b65f667c7b8c3a32bea7372340bfe9a15fa5/certifi-2018.4.16-py2.py3-none-any.whl
Collecting virtualenv (from pipenv)
  Using cached https://files.pythonhosted.org/packages/b6/30/96a02b2287098b23b875bc8c2f58071c35d2efe84f747b64d523721dc2b5/virtualenv-16.0.0-py2.py3-none-any.whl
Requirement not upgraded as not directly required: setuptools>=36.2.1 in /Users/user/.virtualenvs/quicktest/lib/python3.6/site-packages (from pipenv) (39.2.0)
Installing collected packages: virtualenv-clone, certifi, virtualenv, pipenv
Successfully installed certifi-2018.4.16 pipenv-2018.5.18 virtualenv-16.0.0 virtualenv-clone-0.3.0

The first installed package is locked just fine:

(quicktest) user@otg /tmp!% hash -r

(quicktest) user@otg /tmp % pipenv install pygments
Creating a Pipfile for this project…
Installing pygments…
Collecting pygments
  Using cached https://files.pythonhosted.org/packages/02/ee/b6e02dc6529e82b75bb06823ff7d005b141037cb1416b10c6f00fc419dca/Pygments-2.2.0-py2.py3-none-any.whl
Installing collected packages: pygments
Successfully installed pygments-2.2.0

Adding pygments to Pipfile's [packages]…
Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Updated Pipfile.lock (fea9e8)!
Installing dependencies from Pipfile.lock (fea9e8)…
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 1/1 — 00:00:00

Subsequent package installs do complain about --system flag:

(quicktest) user@otg /tmp % pipenv install fabric
Installing fabric…
Collecting fabric
  Using cached https://files.pythonhosted.org/packages/ab/66/c4f9a1f6e50fbf783aab49eef620cb2ec5a8a1eb6cf834347beab4d381a3/fabric-2.1.3-py2.py3-none-any.whl
Collecting invoke<2.0,>=1.0 (from fabric)
  Using cached https://files.pythonhosted.org/packages/6c/66/9e232c59e61f0a0b6552d68419a5c5a5dba368e105fdbfd2b6c74c402234/invoke-1.0.0-py3-none-any.whl
Collecting cryptography>=1.1 (from fabric)
  Using cached https://files.pythonhosted.org/packages/40/87/acdcf84ce6d25a7db1c113f4b9b614fd8d707b7ab56fbf17cf18cd26a627/cryptography-2.2.2-cp34-abi3-macosx_10_6_intel.whl
Collecting paramiko>=2.4 (from fabric)
  Using cached https://files.pythonhosted.org/packages/3e/db/cb7b6656e0e7387637ce850689084dc0b94b44df31cc52e5fc5c2c4fd2c1/paramiko-2.4.1-py2.py3-none-any.whl
Collecting cffi>=1.7; platform_python_implementation != "PyPy" (from cryptography>=1.1->fabric)
  Using cached https://files.pythonhosted.org/packages/8e/be/40b1bc2c3221acdefeb9dab6773d43cda7543ed0d8c8df8768f05af2d01e/cffi-1.11.5-cp36-cp36m-macosx_10_6_intel.whl
Collecting idna>=2.1 (from cryptography>=1.1->fabric)
  Using cached https://files.pythonhosted.org/packages/27/cc/6dd9a3869f15c2edfab863b992838277279ce92663d334df9ecf5106f5c6/idna-2.6-py2.py3-none-any.whl
Collecting six>=1.4.1 (from cryptography>=1.1->fabric)
  Using cached https://files.pythonhosted.org/packages/67/4b/141a581104b1f6397bfa78ac9d43d8ad29a7ca43ea90a2d863fe3056e86a/six-1.11.0-py2.py3-none-any.whl
Collecting asn1crypto>=0.21.0 (from cryptography>=1.1->fabric)
  Using cached https://files.pythonhosted.org/packages/ea/cd/35485615f45f30a510576f1a56d1e0a7ad7bd8ab5ed7cdc600ef7cd06222/asn1crypto-0.24.0-py2.py3-none-any.whl
Collecting pynacl>=1.0.1 (from paramiko>=2.4->fabric)
  Using cached https://files.pythonhosted.org/packages/74/8e/a6c0d340972d9e2f1a405aaa3f2460950b4c0337f92db0291a4355974529/PyNaCl-1.2.1-cp36-cp36m-macosx_10_6_intel.whl
Collecting bcrypt>=3.1.3 (from paramiko>=2.4->fabric)
  Using cached https://files.pythonhosted.org/packages/7e/59/d48fd712941da1a5d6490964a37bb3de2e526965b6766273f6a7049ee590/bcrypt-3.1.4-cp36-cp36m-macosx_10_6_intel.whl
Collecting pyasn1>=0.1.7 (from paramiko>=2.4->fabric)
  Using cached https://files.pythonhosted.org/packages/a0/70/2c27740f08e477499ce19eefe05dbcae6f19fdc49e9e82ce4768be0643b9/pyasn1-0.4.3-py2.py3-none-any.whl
Collecting pycparser (from cffi>=1.7; platform_python_implementation != "PyPy"->cryptography>=1.1->fabric)
Installing collected packages: invoke, pycparser, cffi, idna, six, asn1crypto, cryptography, pynacl, bcrypt, pyasn1, paramiko, fabric
Successfully installed asn1crypto-0.24.0 bcrypt-3.1.4 cffi-1.11.5 cryptography-2.2.2 fabric-2.1.3 idna-2.6 invoke-1.0.0 paramiko-2.4.1 pyasn1-0.4.3 pycparser-2.18 pynacl-1.2.1 six-1.11.0

Adding fabric to Pipfile's [packages]…
Pipfile.lock (fea9e8) out of date, but installation uses --system... re-building lockfile must happen in isolation. Please rebuild lockfile in a virtualenv. Continuing anyway...
Installing dependencies from Pipfile.lock (fea9e8)…
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 1/1 — 00:00:00
@ns-cweber
Copy link

ns-cweber commented Jun 12, 2018

No idea if this is related, but I'm getting a version mismatch error when I run pipenv lock && pipenv install --system --deploy --ignore-pipfile. I'm running the command this way because I noticed the error when I would pipenv lock during development, but the install command would fail in CI, so I juxtaposed them to be sure. Strangely, this error isn't consistent for the same exact source code; sometimes it fails and other times it succeeds, but the hash returned from the pipenv lock is always the same. I've also run cksum on the lockfile to make sure it's the exact same in all cases.

EDIT: I noticed in the trials that are succeeding, our pipenv is changing its version from 2018.05.18 to 11.10.1 (because someone pinned our pipenv version in the Pipfile to 11.10.1--not sure how it got installed since all of the previous pipenv install commands were failing). In short, both versions of pipenv build the same lock file but 2018.05.18 fails to install from that lock file, expecting a different version.

@max-arnold
Copy link
Author

@techalchemy I installed the latest master branch, but the issue is still present.

@techalchemy
Copy link
Member

Ok thanks for the update, guess this one wasn’t as simple. Still need to capture these with test coverage

@techalchemy techalchemy added Type: Bug 🐛 This issue is a bug. Category: Docker Issue affects docker builds. Type: Regression This issue is a regression of a previous behavior. labels Jun 15, 2018
@techalchemy
Copy link
Member

@max-arnold are you still experiencing this? I can't reproduce this.

@max-arnold
Copy link
Author

max-arnold commented Jun 16, 2018

Yes

user@otg /tmp % which python3.6  # installed via macports

/opt/local/bin/python3.6

user@otg /tmp % mkvirtualenv -p python3.6 quicktest

Running virtualenv with interpreter /opt/local/bin/python3.6
Using base prefix '/opt/local/Library/Frameworks/Python.framework/Versions/3.6'
...
virtualenvwrapper.user_scripts creating /Users/user/.virtualenvs/quicktest/bin/get_env_details


(quicktest) user@otg /tmp % pip install git+https://github.com/pypa/pipenv.git@e326ce6e071c8b93f9b451cbf912284811952122

Collecting git+https://github.com/pypa/pipenv.git@e326ce6e071c8b93f9b451cbf912284811952122
  Cloning https://github.com/pypa/pipenv.git (to revision e326ce6e071c8b93f9b451cbf912284811952122) to /private/var/folders/fh/j4kp2bt97gxb89plh9h4gxx80000gn/T/pip-req-build-mzqcvs0e
...
Successfully installed certifi-2018.4.16 pipenv-2018.5.18 virtualenv-16.0.0 virtualenv-clone-0.3.0


(quicktest) user@otg /tmp % hash -r

(quicktest) user@otg /tmp % pipenv install pygments

Creating a Pipfile for this project…
Installing pygments…
...
Updated Pipfile.lock (fea9e8)!
Installing dependencies from Pipfile.lock (fea9e8)…
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 1/1 — 00:00:00

(quicktest) user@otg /tmp % pipenv install requests

Installing requests…
...
Successfully installed chardet-3.0.4 idna-2.7 requests-2.19.1 urllib3-1.23

Adding requests to Pipfile's [packages]…
Pipfile.lock (fea9e8) out of date, but installation uses --system... re-building lockfile must happen in isolation. Please rebuild lockfile in a virtualenv. Continuing anyway...
Installing dependencies from Pipfile.lock (fea9e8)…
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 1/1 — 00:00:00

(quicktest) user@otg /tmp % pipenv --version

pipenv, version 2018.05.18

(quicktest) user@otg /tmp % grep requests Pipfile.lock

(quicktest) user@otg /tmp!% grep requests Pipfile

requests = "*"

techalchemy pushed a commit that referenced this issue Jun 16, 2018
- Fixes #2273
- Hopefully for real this time

Signed-off-by: Dan Ryan <[email protected]>
@techalchemy
Copy link
Member

Can you try the fix I jsut put up? I'm not 100% sure it will work but hopefully

@max-arnold
Copy link
Author

(quicktest) user@otg /tmp!% pipenv install pygments
Creating a Pipfile for this project…
Traceback (most recent call last):
  File "/Users/user/.virtualenvs/quicktest/bin/pipenv", line 11, in <module>
    sys.exit(cli())
  File "/Users/user/.virtualenvs/quicktest/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/Users/user/.virtualenvs/quicktest/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/Users/user/.virtualenvs/quicktest/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/user/.virtualenvs/quicktest/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/user/.virtualenvs/quicktest/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/Users/user/.virtualenvs/quicktest/lib/python3.6/site-packages/pipenv/cli.py", line 416, in install
    selective_upgrade=selective_upgrade,
  File "/Users/user/.virtualenvs/quicktest/lib/python3.6/site-packages/pipenv/core.py", line 1798, in do_install
    if (system or allow_global) and not (PIPENV_VIRTUALENV):
NameError: name 'allow_global' is not defined

@techalchemy
Copy link
Member

Thanks. Guess I didn’t quite fix it ;)

@atleta
Copy link

atleta commented Jun 16, 2018

I'm also experiencing this. I'm on linux and used plain old virtualenv to create my env. python version is 3.5.3. Installed pipenv (pipenv==2018.5.18) inside the virtualenv and running it form there.

@uranusjr
Copy link
Member

@atleta You can work around this bug right now. Pipenv detects it is run in a virtualenv automatically and does the right thing. You don’t need to use --system to run Pipenv inside it :)

@techalchemy
Copy link
Member

@max-arnold ok, the new version definitely fixes it for real, I actually tested it first this time, sorry about that :) will have it in the release

@max-arnold
Copy link
Author

Thank you Dan!

Works for me too:

Adding requests to Pipfile's [packages]…
Pipfile.lock (fea9e8) out of date, updating to (b7f4ec)…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Updated Pipfile.lock (b7f4ec)!
Installing dependencies from Pipfile.lock (b7f4ec)…
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 6/6 — 00:00:02

@atleta
Copy link

atleta commented Jun 17, 2018

@uranusjr Maybe I'm misunderstanding the original report, but I haven't used the system flag.

@uranusjr
Copy link
Member

@atleta If you didn’t use --system or --global, the issue is separate to this one. This only happens when you specify one of both of the flags.

@techalchemy
Copy link
Member

@uranusjr @atleta there were a bunch of bugs with this implementation actually and they are all at least sort of related, but depending which bug you're talking about you might have seen an error message without passing in --system, the above case we were hashing out was for a case where you use your own virtualenv and pipenv interprets it as if you had used --system. That is fixed now, along with a few others (--deploy --system was sometimes just making a virtualenv for no reason, for instance).

@MAbdullahAlvi
Copy link

(quicktest) user@otg /tmp!% pipenv install pygments
Creating a Pipfile for this project…
Traceback (most recent call last):
  File "/Users/user/.virtualenvs/quicktest/bin/pipenv", line 11, in <module>
    sys.exit(cli())
  File "/Users/user/.virtualenvs/quicktest/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/Users/user/.virtualenvs/quicktest/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/Users/user/.virtualenvs/quicktest/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/user/.virtualenvs/quicktest/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/user/.virtualenvs/quicktest/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/Users/user/.virtualenvs/quicktest/lib/python3.6/site-packages/pipenv/cli.py", line 416, in install
    selective_upgrade=selective_upgrade,
  File "/Users/user/.virtualenvs/quicktest/lib/python3.6/site-packages/pipenv/core.py", line 1798, in do_install
    if (system or allow_global) and not (PIPENV_VIRTUALENV):
NameError: name 'allow_global' is not defined

Brother! can you find any solution to this problem so please tell me that

@matteius
Copy link
Member

matteius commented Sep 4, 2022

@MAbdullahAlvi I am not sure seeing as how the latest published version of pipenv 2022.9.4 does not have a line if (system or allow_global) and not (PIPENV_VIRTUALENV): in core.py at all yet alone on line 1798. Please ensure you are on the latest version and open a new complete issue report if you require help further.

Also I noticed you are using python 3.6 which is no longer supported. Best regards.

@pypa pypa locked as off-topic and limited conversation to collaborators Sep 4, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Category: Docker Issue affects docker builds. Type: Bug 🐛 This issue is a bug. Type: Regression This issue is a regression of a previous behavior.
Projects
None yet
Development

No branches or pull requests

7 participants