-
-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Platform sensitive lock mechanism #5561
Comments
I am going to attempt to shed some light on this problem/request and explain why this is a bigger issue than the pipenv maintainers or contributors to pipenv are able to solve for, why it would require work from pip itself and why that may, unfortunately, not happen.
This part feels somewhat doable, though it kind of feels like it would be a similar implementation to
I believe this certainly would require updates from pip to the resolver in order to support -- its not something we can just patch and modify the pip resolver to accomplish. The problem here is that pip doesn't actually support a public interface into the resolver mechanisms and asking for improvements to the resolver therefore is complicated. I've been told prior that unless the feature directly supports pip and its public interface we are unlikely to see such improvements make it into pip internals. You can find more details in this thread which has some related threads tagged #5379
I agree that you would have to have twine twice in your Pipfile to solve this using named categories, but does that really mean it does not provide a workaround for the issue? I get that conceptually there may be better ideas of how a resolver could behave, but the crux of the problem seems to be that pipenv uses pip's resolver, but many of pip's core developers would prefer this weren't the case. Given then, I feel there is hardly a path forward in terms of actually developing a solution for this outside of convincing the pip community that we need a public interface to the resolver. The history here appears to be that pipenv used to have its own bespoke resolver, which numerous bug reports and later on pip reworked their internal resolver and pipenv adopted it because it solved numerous bug reports. We cannot just go back to inventing our own resolver; we instead need support from pip to help solve the cross platform resolving issues but again this is not guaranteed because so far the pip resolver hasn't been designed around resolving transitive dependencies that do not pertain to the system the resolution is happening on. That is why I feel there is no solution in sight. I've thought we could enhance the work around of named categories further, perhaps for example environment variables that specify which categories |
Thanks for your detailed answer. I got now a better insight how
I can be a workaround but basically it destroys the concept (at least as I understand it and maybe you as well). Having two categories with the same content is basically nothing else as having two lock-files. The packages and there versions can differ and nobody notice anything because he's only using his category. The solution, I would like to see, would use the same dependency tree as far as possible. It's quite unusual that you ran into problems with dependencies on a second or higher level. Most of the problems are on first level, so on your direct dependencies. If there is an issue with another dependency, most likely the maintainer of the dependency in between would care. |
It is a good question and one I should spend more time looking into, but my minimal understanding is that poetry does not use the pip resolver. I am not sure what it is using, though. The transitive issue I mentioned I think applies to what you are calling direct dependencies, basically packages that are related to a package you require. There is probably additional details in this thread #5130 that would interest you. |
Almost, the "direct" dependency would be a requirement of my package. But I think you already explained in a very nice way why it is that hard to fix.
Yeah, I started with this thread and noticed that it might does not fit 100%. Anyhow, I think I got my answer. You and our collaborators are not able to fix this issue on short period for fair technical reasons. Unfortunately, this means for me that I have explore other solutions. Please feel free to close. |
Thanks for your understanding @kolibri91 -- I will leave open for the foreseeable future: it's something we want to solve, and further documents the problem. |
I am working on a path forward for cross platform dependencies, and I think I have something that might stick. #5892 I think it only consider sys_platform markers right now, but I think it makes sense to apply it to all marker types. Please have a look at the above PR. here is an example:
` |
Is your feature request related to a problem? Please describe.
I'm facing a problem regarding platform-dependent dependencies of packages used by my own package. For example, I'm using
black
during development and for that reason it's added as a package in the [dev-packages]-section of Pipfile. Another tool istwine
.black
has a windows only dependency calledcolorama
. If you want to track this in our lock-file over multiple platforms, you could add it with a marker to your Pipfile. With this marker, it will be almost untouched once when you runpipenv lock --dev
on a linux platform.twine
has a dependency calledkeyring
and this package has other linux only dependencies likeSecretStorage
,jeepney
,cryptography
, etc. You could use the same workaround here but:Last but not least, also the proposed workaround does not work 100% correct because it removes the "index"-attribute of packages marked with the other platform.
Describe the solution you'd like
I would like to see a solution where the "lock"-mechanism does not touch any attribute of packages with a mis-match in their markers.
This means, if I lock on windows, the packages with the marker
sys_platform == 'linux'
are unchanged. Further, I would like to see a weak inheritance of the markers. If I add a package likekeyring
with a marker to the Pipfile and the resolver notices that there are dependencies, all dependencies should "inherit" the marker. If other packages with no marker would need the package as well, then the marker should be removed so that only the weakest marker survives.Describe alternatives you've considered
I checked the named categories and they does not solve the issue. Using named categories would mean, that I have to enter
twine
twice into the Pipfile, once in windows-group and once in a linux-group.Additional context
I'm using windows10 and ubuntu20.04 with python 3.8.10 and
pipenv==2022.12.19
$ pipenv --support
Pipenv version:
'2022.12.19'
Pipenv location:
'C:\\work\\.venv-pipenv38\\lib\\site-packages\\pipenv'
Python location:
'C:\\work\\.venv-pipenv38\\Scripts\\python.exe'
OS Name:
'nt'
User pip version:
'22.3.1'
user Python installations found:
3.10.5
:C:\DEV\tools\python\py310\python.exe
3.9.13
:C:\DEV\tools\python\py39\python.exe
3.8.10
:C:\DEV\tools\python\py38\python.exe
3.7.8
:C:\DEV\tools\python\py37\python.exe
3.6.8
:C:\DEV\tools\python\py36\python.exe
PEP 508 Information:
System environment variables:
ALLUSERSPROFILE
ANDROID_NDK_HOME
APPDATA
BLENDER_EXECUTABLE
CAMLIBS
CLM_ACTIVATION_WIZARD
CLM_HOME
CLM_NEXT_HOME
COMMONPROGRAMFILES
COMPUTERNAME
COMSPEC
CONAN_CMAKE_GENERATOR
CONAN_MAKE_PROGRAM
CONAN_SSH_CHECKOUT
CONAN_USER_HOME
CONAN_USER_HOME_SHORT
CONAN_USE_ALWAYS_SHORT_PATHS
CONAN_V2_MODE
COMMONPROGRAMFILES(X86)
COMMONPROGRAMW6432
DRIVERDATA
EXEPATH
FLEXLM_TIMEOUT
HELIOS_EXECUTABLE
HOME
HOMEDRIVE
HOMEPATH
IOLIBS
JAVA_HOME
LOCALAPPDATA
LOGONSERVER
MSYSTEM
NEXTHINK
NUMBER_OF_PROCESSORS
OLDPWD
OS
ONEDRIVE
ONEDRIVECOMMERCIAL
ONLINESERVICES
PATH
PATHEXT
PIPENV_IGNORE_VIRTUALENVS
PIPENV_KEEP_OUTDATED
PIPENV_TOKEN
PIPENV_USERNAME
PIPENV_VENV_IN_PROJECT
PLINK_PROTOCOL
PROCESSOR_ARCHITECTURE
PROCESSOR_IDENTIFIER
PROCESSOR_LEVEL
PROCESSOR_REVISION
PROGRAMFILES
PS1
PSMODULEPATH
PUBLIC
PWD
PROGRAMDATA
PROGRAMFILES(X86)
PROGRAMW6432
PYCHARM COMMUNITY EDITION
PYCHARM
REGIONCODE
SHLVL
SYSTEMDRIVE
SYSTEMROOT
TEMP
TERM
TMP
TOXENV
USERDNSDOMAIN
USERDOMAIN
USERDOMAIN_ROAMINGPROFILE
USERNAME
USERPROFILE
VBOX_MSI_INSTALL_PATH
VIRTUAL_ENV
VS140COMNTOOLS
WINDIR
WORK
WSLENV
WT_PROFILE_ID
WT_SESSION
ZES_ENABLE_SYSMAN
_
PLATFORMCODE
PIP_DISABLE_PIP_VERSION_CHECK
PIP_PYTHON_PATH
PYTHONDONTWRITEBYTECODE
PYTHONFINDER_IGNORE_UNSUPPORTED
Pipenv–specific environment variables:
PIPENV_IGNORE_VIRTUALENVS
:1
PIPENV_KEEP_OUTDATED
:0
PIPENV_TOKEN
: ****PIPENV_USERNAME
: ****PIPENV_VENV_IN_PROJECT
:1
Debug–specific environment variables:
PATH
:C;C:\work\.venv-pipenv38\Scripts;C:\Program Files\Git\mingw64\bin;C:\Program Files\Git\usr\bin;C:\Users\****\bin;C:\Program Files (x86)\Common Files\Oracle\Java\javapath;C:\Program Files\AdoptOpenJDK\jre-11.0.3.2-openj9\bin;C:\windows\system32;C:\windows;C:\windows\System32\Wbem;C:\windows\System32\WindowsPowerShell\v1.0;C:\windows\System32\OpenSSH;C:\Program Files\WindowsPowerShell\Scripts;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0;C:\WINDOWS\System32\OpenSSH;C:\Program Files\MATLAB\R2019b\bin;C:\Program Files\TortoiseGit\bin;C:\Program Files\dotnet;C:\Program Files\Docker\Docker\resources\bin;C:\Program Files\Git\cmd;C:\Users\****\AppData\Local\Programs\Microsoft VS Code\bin;C:\Program Files\JetBrains\PyCharm 2020.2.2\bin;C:\Users\****\AppData\Local\Programs\MiKTeX\miktex\bin\x64;C:\DEV\tools;C:\Program Files\CMake\bin;C:\Program Files (x86)\Graphviz2.38\bin;C:\Users\****\AppData\Local\Pandoc;C:\DEV\android\platform-tools;C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\MSBuild\Current\Bin
PWD
:C:/work/python-packages/reality-capture-models
VIRTUAL_ENV
:C:\work\.venv-pipenv38
Contents of
Pipfile
('C:\work\python-packages\reality-capture-models\Pipfile'):Contents of
Pipfile.lock
('C:\work\python-packages\reality-capture-models\Pipfile.lock'):</details/
The text was updated successfully, but these errors were encountered: