Environment markers in dependencies should be merged, not overriden by each other

[Jamim]

Issue description

Pipenv.lock file might contain different markers after pipenv lock.

This issue is quite annoying when you deal with a Pipfile that contains a lot of requirements especially when most of them are outdated.

Expected result

Markers remain unchanged after pipenv lock if nothing has been changed in Pipfile and on the PyPI.

Actual result

Markers change unpredictably.

Steps to replicate

You can use this script to reproduce the bug.
Please see the asciicast for the details.

Environment

I experience this issue

• on latest Gentoo with latest pipenv from PyPI and from master of this repo
• on Ubuntu 16.04 with latest pipenv from PyPI

$ pipenv --support # Gentoo

Pipenv version: '2018.7.1.dev0'

Pipenv location: '/home/mim/src/pipenv/pipenv'

Python location: '/usr/bin/python3.7'

Other Python installations in PATH:

• 2.7: /usr/bin/python2.7

• 2.7: /usr/bin/python2.7

• 3.6: /usr/bin/python3.6m

• 3.6: /usr/bin/python3.6

• 3.7: /usr/bin/python3.7m

• 3.7: /usr/bin/python3.7

• 3.7.0: /usr/bin/python

• 2.7.15: /usr/bin/python2

• 3.7.0: /usr/bin/python3

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.7.0',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '4.17.10-gentoo',
 'platform_system': 'Linux',
 'platform_version': '#1 SMP Wed Jul 25 23:37:33 +03 2018',
 'python_full_version': '3.7.0',
 'python_version': '3.7',
 'sys_platform': 'linux'}

System environment variables:

• LC_ALL
• LS_COLORS
• STY
• ANDROID_HOME
• XDG_MENU_PREFIX
• LANG
• LESS
• DISPLAY
• OPENGL_PROFILE
• CONFIG_PROTECT_MASK
• EDITOR
• GPG_TTY
• COLORTERM
• ASCIINEMA_REC
• JAVA_HOME
• GCC_SPECS
• SSH_AUTH_SOCK
• GLADE_CATALOG_PATH
• VBOX_APP_HOME
• HUSHLOGIN
• USER
• GLADE_MODULE_PATH
• PAGER
• DESKTOP_SESSION
• PWD
• HOME
• MANPAGER
• WINDOW
• SSH_AGENT_PID
• GSETTINGS_BACKEND
• XDG_DATA_DIRS
• JDK_HOME
• GLADE_PIXMAP_PATH
• GTK_MODULES
• MAIL
• WINDOWPATH
• CONFIG_PROTECT
• SHELL
• TERM
• VTE_VERSION
• JAVAC
• XDG_CURRENT_DESKTOP
• OPENCL_PROFILE
• ANDROID_SWT
• MOZ_GMP_PATH
• SHLVL
• LANGUAGE
• MANPATH
• WINDOWID
• LOGNAME
• DBUS_SESSION_BUS_ADDRESS
• XSESSION
• XAUTHORITY
• ANT_HOME
• XDG_CONFIG_DIRS
• PATH
• INFOPATH
• TERMCAP
• SESSION_MANAGER
• LESSOPEN
• _
• PYTHONDONTWRITEBYTECODE
• PIP_PYTHON_PATH

Pipenv–specific environment variables:

Debug–specific environment variables:

• PATH: /home/mim/.local/bin:/home/mim/.local/bin:/usr/x86_64-pc-linux-gnu/gcc-bin/8.1.0:/usr/lib/llvm/6/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin:/opt/android-sdk-update-manager/tools:/opt/android-sdk-update-manager/platform-tools:/home/mim/.npm-global/bin:/home/mim/.npm-global/bin:
• SHELL: /bin/bash
• EDITOR: /bin/nano
• LANG: ru_RU.utf8
• PWD: /home/mim

---

$ pipenv --support # Ubuntu 16.04

Pipenv version: '2018.7.1'

Pipenv location: '/home/mim/.local/lib/python3.6/site-packages/pipenv'

Python location: '/usr/bin/python3.6'

Other Python installations in PATH:

• 2.7: /usr/bin/python2.7

• 2.7: /usr/bin/python2.7

• 3.5: /usr/bin/python3.5m

• 3.5: /usr/bin/python3.5

• 3.6: /usr/bin/python3.6m

• 3.6: /usr/bin/python3.6

• 2.7.12: /usr/bin/python

• 2.7.12: /usr/bin/python2

• 3.5.2: /usr/bin/python3

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.6.6',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '4.13.0-45-generic',
 'platform_system': 'Linux',
 'platform_version': '#50~16.04.1-Ubuntu SMP Wed May 30 11:18:27 UTC 2018',
 'python_full_version': '3.6.6',
 'python_version': '3.6',
 'sys_platform': 'linux'}

System environment variables:

• XDG_VTNR
• LC_PAPER
• XDG_SESSION_ID
• LC_ADDRESS
• XDG_GREETER_DATA_DIR
• LC_MONETARY
• CLUTTER_IM_MODULE
• SESSION
• GPG_AGENT_INFO
• XDG_MENU_PREFIX
• TERM
• SHELL
• QT_LINUX_ACCESSIBILITY_ALWAYS_ON
• WINDOWID
• LC_NUMERIC
• UPSTART_SESSION
• GNOME_KEYRING_CONTROL
• GTK_MODULES
• USER
• LS_COLORS
• QT_ACCESSIBILITY
• LC_TELEPHONE
• XDG_SESSION_PATH
• UNITY_HAS_3D_SUPPORT
• XDG_SEAT_PATH
• SSH_AUTH_SOCK
• SESSION_MANAGER
• DEFAULTS_PATH
• XDG_CONFIG_DIRS
• UNITY_DEFAULT_PROFILE
• PATH
• DESKTOP_SESSION
• QT_QPA_PLATFORMTHEME
• QT_IM_MODULE
• LC_IDENTIFICATION
• __fish_sysconfdir
• XDG_SESSION_TYPE
• PWD
• JOB
• XMODIFIERS
• __fish_datadir
• LANG
• GNOME_KEYRING_PID
• MANDATORY_PATH
• GDM_LANG
• LC_MEASUREMENT
• IM_CONFIG_PHASE
• COMPIZ_CONFIG_PROFILE
• PAPERSIZE
• GDMSESSION
• SESSIONTYPE
• GTK2_MODULES
• XDG_SEAT
• SHLVL
• HOME
• LANGUAGE
• GNOME_DESKTOP_SESSION_ID
• UPSTART_INSTANCE
• __fish_help_dir
• XDG_SESSION_DESKTOP
• UPSTART_EVENTS
• LOGNAME
• COMPIZ_BIN_PATH
• XDG_DATA_DIRS
• QT4_IM_MODULE
• DBUS_SESSION_BUS_ADDRESS
• LESSOPEN
• UPSTART_JOB
• INSTANCE
• XDG_RUNTIME_DIR
• DISPLAY
• XDG_CURRENT_DESKTOP
• GTK_IM_MODULE
• LESSCLOSE
• LC_TIME
• __fish_bin_dir
• XAUTHORITY
• LC_NAME
• COLORTERM
• _
• PYTHONDONTWRITEBYTECODE
• PIP_PYTHON_PATH

Pipenv–specific environment variables:

Debug–specific environment variables:

• PATH: /home/mim/bin:/home/mim/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/mim/.npm-global/bin
• SHELL: /bin/bash
• LANG: en_US.UTF-8
• PWD: /home/mim

---

[nikolay-saskovets]

I've also faced with the same issue:

I've tried the script from @Jamim and here are my results:

diff --git a/Pipfile.lock b/Pipfile.lock
index d77c7e4..c306df2 100644
--- a/Pipfile.lock
+++ b/Pipfile.lock
@@ -80,7 +80,7 @@
                 "sha256:330cc27ccbf7f1e992e69fef78261dc7c6569012cf397db8d3de0234e6c937ca",
                 "sha256:a7bb0f2cf3a3fd1ab2732cb49eba4252c2af4240442415b4abce3b87022a8f50"
             ],
-            "markers": "python_version < '3.0'",
+            "markers": "python_version < '3.3'",
             "version": "==1.0.2"
         },
         "mock": {

Something was changed on attempt 5

my env on Ubuntu 18.04:

$ pipenv --support

Pipenv version: '2018.7.1'

Pipenv location: '/home/develop/.local/lib/python3.6/site-packages/pipenv'

Python location: '/usr/bin/python3'

Other Python installations in PATH:

• 2.7: /usr/bin/python2.7

• 2.7: /usr/bin/python2.7

• 3.6: /usr/bin/python3.6m

• 3.6: /usr/bin/python3.6

• 2.7.15: /usr/bin/python

• 2.7.15: /usr/bin/python2

• 3.6.5: /usr/bin/python3

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.6.5',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '4.15.0-29-generic',
 'platform_system': 'Linux',
 'platform_version': '#31-Ubuntu SMP Tue Jul 17 15:39:52 UTC 2018',
 'python_full_version': '3.6.5',
 'python_version': '3.6',
 'sys_platform': 'linux'}

System environment variables:

• CLUTTER_IM_MODULE
• NVM_DIR
• LS_COLORS
• LC_MEASUREMENT
• LESSCLOSE
• LC_PAPER
• LC_MONETARY
• LANG
• GDM_LANG
• HISTCONTROL
• DISPLAY
• HISTTIMEFORMAT
• QT_STYLE_OVERRIDE
• WORKON_HOME
• EDITOR
• COLORTERM
• XDG_VTNR
• SSH_AUTH_SOCK
• MANDATORY_PATH
• LC_NAME
• XDG_SESSION_ID
• XDG_GREETER_DATA_DIR
• USER
• DESKTOP_SESSION
• QT4_IM_MODULE
• DEFAULTS_PATH
• PWD
• HOME
• SSH_AGENT_PID
• XDG_SESSION_TYPE
• XDG_DATA_DIRS
• XDG_SESSION_DESKTOP
• LC_ADDRESS
• LC_NUMERIC
• VISUAL
• TERM
• SHELL
• XDG_SEAT_PATH
• XMODIFIERS
• GPG_AGENT_INFO
• COLORFGBG
• XDG_SEAT
• SHLVL
• LANGUAGE
• WINDOWID
• LC_TELEPHONE
• GDMSESSION
• LOGNAME
• DBUS_SESSION_BUS_ADDRESS
• XDG_RUNTIME_DIR
• HOSTFILE
• XAUTHORITY
• XDG_SESSION_PATH
• XDG_CONFIG_DIRS
• PATH
• LC_IDENTIFICATION
• PS1
• HISTSIZE
• HISTFILESIZE
• LESSOPEN
• GTK_IM_MODULE
• LC_TIME
• OLDPWD
• _
• PYTHONDONTWRITEBYTECODE
• PIP_PYTHON_PATH

Pipenv–specific environment variables:

Debug–specific environment variables:

• PATH: /home/develop/bin/my:/home/develop/bin:/home/develop/bin/my:/home/develop/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/root/bin/:/usr/local/rvm/bin/
• SHELL: /bin/bash
• EDITOR: vim
• LANG: en_US.UTF-8
• PWD: /home/develop/tmp

---

[uranusjr]

@nikolay-saskovets What is the package that produces the different output? This information would save us much time to investigate. Without looking into this too deeply, it is likely due to how a certain package is distributed in an undeterministic way, but it would be much easier to explain this with a concrete example.

[nikolay-saskovets]

@uranusjr this output is about installing those two packages pytest-cov pytest-vcr. And markers are changes for one of their dependency — the package funcsigs:

develop:$ git diff -U5
diff --git a/Pipfile.lock b/Pipfile.lock
index d77c7e4..c306df2 100644
--- a/Pipfile.lock
+++ b/Pipfile.lock
@@ -78,11 +78,11 @@
         "funcsigs": {
             "hashes": [
                 "sha256:330cc27ccbf7f1e992e69fef78261dc7c6569012cf397db8d3de0234e6c937ca",
                 "sha256:a7bb0f2cf3a3fd1ab2732cb49eba4252c2af4240442415b4abce3b87022a8f50"
             ],
-            "markers": "python_version < '3.0'",
+            "markers": "python_version < '3.3'",
             "version": "==1.0.2"
         },

But it seems like such issues may caught with some other packages from PyPi. We are faced with such problem in a real project with other packages, too.

I think @Jamim may provide more info as «topic starter» :-)

[uranusjr]

Thanks! I think I have an idea of what’s going on, but I’ll need some time to investigate it further to make sure my guess is correct. It would be super helpful if more details could be provideded in the meantime.

What would be helpful are:

• Output of pipenv lock --verbose
• Full Pipfile.lock results
• Whether pipenv lock --lock --verbose changes the output (and if it does, what exactly changed)
• pipenv graph and pipenv run pip list results on both environments

[Jamim]

Thank you, @nikolay-saskovets!

Hi @uranusjr,

The simplest way to reproduce the issue is to run the script from Steps to replicate.
It installs pytest-cov and pytest-vcr and locks the dependencies in a loop until the Pipfile.lock has changed. I specially made this example using as less requirements as possible.

In the example the issue affects funcsigs==1.0.2 only, but in the real project it affects:

• backports.functools-lru-cache==1.5
• configparser==3.5.0
• contextlib2==0.5.5
• funcsigs==1.0.2

So every time when I run pipenv lock it changes some markers :-/

There are a lot of pipenv lock --verbose at the asciicast. I hope it should be really helpful.

[Jamim]

@uranusjr You can found Pipfile and Pipfile.lock in this repository.
It's the repository that was created during the asciicast recording.

[Jamim]

pipenv graph:

pytest-cov==2.5.1
  - coverage [required: >=3.7.1, installed: 4.5.1]
  - pytest [required: >=2.6.0, installed: 3.6.3]
    - atomicwrites [required: >=1.0, installed: 1.1.5]
    - attrs [required: >=17.4.0, installed: 18.1.0]
    - funcsigs [required: Any, installed: 1.0.2]
    - more-itertools [required: >=4.0.0, installed: 4.2.0]
      - six [required: >=1.0.0,<2.0.0, installed: 1.11.0]
    - pluggy [required: >=0.5,<0.7, installed: 0.6.0]
    - py [required: >=1.5.0, installed: 1.5.4]
    - setuptools [required: Any, installed: 40.0.0]
    - six [required: >=1.10.0, installed: 1.11.0]
pytest-vcr==0.3.0
  - pytest [required: >=3.0.0, installed: 3.6.3]
    - atomicwrites [required: >=1.0, installed: 1.1.5]
    - attrs [required: >=17.4.0, installed: 18.1.0]
    - funcsigs [required: Any, installed: 1.0.2]
    - more-itertools [required: >=4.0.0, installed: 4.2.0]
      - six [required: >=1.0.0,<2.0.0, installed: 1.11.0]
    - pluggy [required: >=0.5,<0.7, installed: 0.6.0]
    - py [required: >=1.5.0, installed: 1.5.4]
    - setuptools [required: Any, installed: 40.0.0]
    - six [required: >=1.10.0, installed: 1.11.0]
  - vcrpy [required: Any, installed: 1.13.0]
    - contextlib2 [required: Any, installed: 0.5.5]
    - mock [required: Any, installed: 2.0.0]
      - funcsigs [required: >=1, installed: 1.0.2]
      - pbr [required: >=0.11, installed: 4.2.0]
      - six [required: >=1.9, installed: 1.11.0]
    - PyYAML [required: Any, installed: 3.13]
    - six [required: >=1.5, installed: 1.11.0]
    - wrapt [required: Any, installed: 1.10.11]

pipenv run pip list:

Package        Version
-------------- -------
atomicwrites   1.1.5  
attrs          18.1.0 
contextlib2    0.5.5  
coverage       4.5.1  
funcsigs       1.0.2  
mock           2.0.0  
more-itertools 4.2.0  
pbr            4.2.0  
pip            18.0   
pluggy         0.6.0  
py             1.5.4  
pytest         3.6.3  
pytest-cov     2.5.1  
pytest-vcr     0.3.0  
PyYAML         3.13   
setuptools     40.0.0 
six            1.11.0 
vcrpy          1.13.0 
wheel          0.31.1 
wrapt          1.10.11

[uranusjr]

Quick update: I think the problem is a race condition, for funcsig it’s between mock and pytest. They are picking up different conditions from each source. They should be instead consolidated into one instead (ideally python_version < '3.3', but I don’t think we have that level of merker parsing ability, and would need to settle with python_version <'3.0' or python_version < '3.3').

[techalchemy]

is this only affecting python_version? This is not straightforward and also it's not really a bug, they are functionally identical which is why it doesn't matter. I get that it is an annoyance, but it is not a resolution bug, it is a bug with having to ignore lockfile changes sometimes

[proofit404]

For some reason, pipenv can add "markers": null, to the lock file.

Heroku fails to deploy a project before I clean up all the markers from the lock file manually.

[techalchemy]

oh boy the null markers thing is actually quite bad, but i think that's fixed

merging markers is incredibly difficult it turns out, I've spent like several months on this over in sarugaku/passa#59 and it still isn't quite finished

If anyone is very good at set math, you can probably help with that...

[nickmyatt]

I am experiencing the same issue (fluctuating python_version marker) when installing backports.functools-lru-cache.

Does a given Pipfile.lock actually support mutliple python_versions anyway? Quoting #2683 (comment)

Hi, this has been proposed multiple times (the most informational thread is #1050, I believe), and the resolution is to refrain from doing it, since Pipenv wants to focus on the application aspect of Python development. In an application context, and it is wildly considered best practice to support only one Python minor version in a Python application.

We schedule pipenv update to scan for new dependencies, and generate merge requests if Pipfile.lock has changed. Nondeterministic environment markers is obviously a problem for such a workflow as it generates spurious MRs. My current solution to the problem is

sed -i '/"markers": "python_version/d' Pipfile.lock

before committing Pipfile.lock. Couldn't pipenv just strip the python_version markers out whenever

[requires]
python_version = "..."

is present?

[uranusjr]

Does a given Pipfile.lock actually support mutliple python_versions anyway?

It does not, but a missing python_version is supported.

[matteius]

I believe this has been resolve and/or is stale.