Solve upgrade can create incompatible dependency versions in lock file + other bugs #6282

matteius · 2024-10-23T11:42:26Z

The issue

pipenv upgrade relied on the dependent packages being updated, but sometimes the package being requested for update is a dependency of another package in the tree. This applies a more sophisticated approach to considering both ends of that equation.

Fixes #6281
Fixes #6273
Fixes #6280
Fixes #6217
Fixes #6149
Fixes #6144
Fixes #6165
Fixed #6115

Probably Fixes #6249
Probably Fixes #6154

Possibly fixes #6116
Possibly fixes #6237

The fix

Adds test that fails on main branch but passes here:

=============================================================================================== warnings summary ================================================================================================
..\..\.virtualenvs\pipenv-fJ_HixU3\Lib\site-packages\_pytest\config\__init__.py:1441: 25 warnings
  C:\Users\matte\.virtualenvs\pipenv-fJ_HixU3\Lib\site-packages\_pytest\config\__init__.py:1441: PytestConfigWarning: Unknown config option: plugins

    self._warn_or_fail_if_strict(f"Unknown config option: {key}\n")

tests\integration\conftest.py:47
  C:\Users\matte\projects\pipenv\tests\integration\conftest.py:47: RuntimeWarning: Failed connecting to internet: http://httpbin.org/ip
    warnings.warn(

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
============================================================================================ short test summary info ============================================================================================
FAILED tests/integration/test_upgrade.py::test_pipenv_dependency_incompatibility_resolution - AssertionError: google-api-core was not updated to a compatible version despite the protobuf update
assert '==2.18.0' != '==2.18.0'
=================================================================================== 1 failed, 26 warnings in 64.37s (0:01:04) ===================================================================================

The checklist

Associated issue
A news fragment in the news/ directory to describe this fix with the extension .bugfix.rst, .feature.rst, .behavior.rst, .doc.rst. .vendor.rst. or .trivial.rst (this will appear in the release changelog). Use semantic line breaks and name the file after the issue number or the PR #.

…s now

…nd reversing those dependencies to see what truly must be considered

…rrect reverse results, rewrite the reverse test to use --json-tree to prevent consistency in testing issues, and convert off click echo

…ses + pip session requests performance issue patch

…rify interface to resolver category

tests/integration/test_lock.py

tests/integration/test_install_markers.py

tests/unit/test_utils.py

tests/integration/test_install_markers.py

tests/integration/test_cli.py

tests/integration/test_upgrade.py

tests/integration/test_install_vcs.py

pipenv/utils/resolver.py

pipenv/utils/project.py

pipenv/routines/update.py

matteius · 2024-10-26T04:05:39Z

pipenv/routines/update.py

-        categories = ["develop"]
-    elif not categories or "packages" in categories:
-        categories = ["default"]
+    if not categories:


If the original authors had just made the package group names identical between Pipfile and lock it would have saved 50% of my headache here.

pipenv/routines/graph.py

matteius · 2024-10-26T04:08:04Z

pipenv/utils/requirements.py

@@ -223,7 +223,7 @@ def requirement_from_lockfile(
        if "extras" in package_info
        else ""
    )
-    pip_line = f"{package_name}{extras}=={version}{os_markers}{markers}{hashes}"


The cause of our version specifiers being too strict, because this method is actually used in more places than just requirement_from_lockfile 👎

pyproject.toml

.github/PULL_REQUEST_TEMPLATE.md

oz123 · 2024-10-29T20:28:19Z

pipenv/resolver.py

-            k in entry_dict
-            for k in ["sys_platform", "python_version", "os_name", "platform_machine"]
-        ):
-            return None, entry_dict


You are replicating code from plette here why not just import it?

oz123 · 2024-10-29T20:29:13Z

pipenv/resolver.py


    @property
-    def original_markers(self):
-        original_markers, lockfile_dict = self.get_markers_from_dict(self.lockfile_dict)


These local methods might be added to plette.

oz123 · 2024-10-29T20:29:49Z

pipenv/resolver.py

+        if isinstance(self.extras, list):
+            self.extras = set(self.extras)
+        if isinstance(self.hashes, list):
+            self.hashes = set(self.hashes)



Same as comment above.

pipenv/routines/graph.py

pyproject.toml

tests/integration/test_cli.py

matteius added 2 commits October 23, 2024 07:38

improvements to pipenv reverse to allow json output as pipdeptree doe…

49b0c1f

…s now

Improve update routine logic by considering whats already installed a…

68562b8

…nd reversing those dependencies to see what truly must be considered

matteius mentioned this pull request Oct 23, 2024

pipenv update/upgrade can create incompatible dependency versions in lock file #6281

Closed

matteius added 9 commits October 23, 2024 20:13

convert this code to the prefered output mechanism

e6cabda

edge case

40d347c

correction

1ed6447

convert this code off click

928b6e9

refactor to reduce install phases, add pre-sync phase to determine co…

73aaf37

…rrect reverse results, rewrite the reverse test to use --json-tree to prevent consistency in testing issues, and convert off click echo

Add new test that demonstrates the issue 6281

7bf15fb

add news fragment

2594e3b

fix ruff

3b086b6

fix test

870226d

matteius force-pushed the issue-6281 branch from ae2b85c to b1018a8 Compare October 24, 2024 04:20

matteius changed the title ~~Issue 6281~~ Solve pipenv update/upgrade can create incompatible dependency versions in lock file Oct 24, 2024

matteius marked this pull request as ready for review October 24, 2024 04:23

matteius requested a review from oz123 October 24, 2024 04:23

matteius force-pushed the issue-6281 branch from b1018a8 to 1f8cee7 Compare October 24, 2024 04:40

This was referenced Oct 24, 2024

pyarrow error on FreeBSD 13.2 #6133

Open

Dependency resolution error upgrading a package by a patch version with no changes to dependencies #6154

Closed

Updating packages with non canonical names errors out #6116

Closed

matteius force-pushed the issue-6281 branch 3 times, most recently from e68751e to c34fa08 Compare October 24, 2024 22:47

Replace the old Entry class with something simpler based on data-clas…

ef04cd2

…ses + pip session requests performance issue patch

matteius force-pushed the issue-6281 branch from c34fa08 to ef04cd2 Compare October 24, 2024 22:59

matteius marked this pull request as draft October 24, 2024 23:22

matteius added 4 commits October 24, 2024 19:31

Replace the old Entry class with something simpler based on data-clas…

bc30825

…ses + pip session requests performance issue patch

fix tests by including subdir and adjusting expected req string; clea…

4888042

…rify interface to resolver category

fix remaining tests

f7bbae8

fix remaining tests

ee09fb1

Remove peep process

7968287

matteius force-pushed the issue-6281 branch from 4590a92 to 7968287 Compare October 26, 2024 03:36

matteius changed the title ~~Solve pipenv update/upgrade can create incompatible dependency versions in lock file~~ Solve upgrade can create incompatible dependency versions in lock file + other bugs Oct 26, 2024