poetry update with git-based dependencies updates poetry.lock but not the virtual environment

[IaroslavR]

• I am on the latest Poetry version.
• I have searched the issues of this repo and believe that this is not a duplicate.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Ubuntu 18.04.5 LTS
• Poetry version: 1.1.6
• Link of a Gist with the contents of your pyproject.toml file: https://gist.github.com/IaroslavR/cfe6372de78b45c8be80b819f0a6112e

Issue

Almost identical to #2921 with only one difference, I have a git dependency that is defined with a revision instead of branch. When I change rev="v0.1.2" to rev="v0.1.3" poetry.lock successfully updated with the correct commit resolved_reference but the library in the virtual environment remains checked-out at the commit that was latest when poetry install was last run.

[wakemaster39]

I have a proposed fix for this in #3867, poetry is not detecting it is a git installation properly.

[iainvm]

The fix put in for #3867 doesn't seem to be working for many, and isn't working for me either, but the issue has been closed. So Hopefully this open issue will get some traction?

[aptlin]

Any updates?

[Ivorforce]

I can confirm the fix in #3867 works for me (git rev= syntax).

However, the latest poetry release does not include the fix yet (poetry 1.1.8 from August 19th).

If you are willing to try a poetry pre-release:

poetry self update --preview

Note that poetry.lock files created by the 1.2 preview may be incompatible with those of the stable versions.

If you want the fix as a hotfix to poetry 1.1.x:

# Extract the venv path from this file (second line):
cat `which poetry`
# Add the hotfix to this file. For me, it needed to go into line 68.
nano <VENV>/lib/python3.8/site-packages/poetry/repositories/installed_repository.py

The hotfix, for quick reference (credit to @wakemaster39):

                                # this is required to handle pathlib oddity on win32 python==3.5
                                path = lib.joinpath(path)
                        paths.add(path)

        src_path = env.path / "src" / name
        if not paths and src_path.exists():
            paths.add(src_path)

        return paths

    @classmethod

[mrtolkien]

The 1.2 preview crashes for me, but this hotfix works perfectly on 1.1.8. Thanks for sharing it!

[phillipuniverse]

The workaround that @Ivorforce worked for me but I was using the legacy installer and the file to patch is at ~/.poetry/lib/poetry/repositories/installed_repository.py.

[PabloCastellano]

This is still happening to me on 1.1.12. How can we move this forward?

I tried @Ivorforce workaround but it's not perfect, since all git dependencies are cloned even if the version has not changed:

Check these results:

1. Without the workaround

$ poetry install
Installing dependencies from lock file

No dependencies to install or update

2. After applying the workaround

$ poetry install
Installing dependencies from lock file

Package operations: 0 installs, 5 updates, 0 removals

  • Updating dj-stripe (2.5.1 0d22b4e -> 2.5.1 de40328)
  • Updating django-bootstrap-scss-static (4.5.3 c4a1291 -> 4.5.3 c4a1291)
  • Updating django-daterangefilter (1.0.0 f1a4339 -> 1.0.0 f1a4339)
  • Updating django-tailwind (3.1.1 d340149 -> 3.1.1 d340149)
  • Updating pyswagger (0.8.39 7b15c12 -> 0.8.39 7b15c12)

It worked! But I only expected to update dj-stripe. And from the results it was indeed the only one that updated the version.

But what happens if I run it again?

$ poetry install
Installing dependencies from lock file

Package operations: 0 installs, 5 updates, 0 removals

  • Updating dj-stripe (2.5.1 de40328 -> 2.5.1 de40328)
  • Updating django-bootstrap-scss-static (4.5.3 c4a1291 -> 4.5.3 c4a1291)
  • Updating django-daterangefilter (1.0.0 f1a4339 -> 1.0.0 f1a4339)
  • Updating django-tailwind (3.1.1 d340149 -> 3.1.1 d340149)
  • Updating pyswagger (0.8.39 7b15c12 -> 0.8.39 7b15c12)

What? So all git dependencies are updated even if the version didn't change 🤔

[ghost]

as a quick workaround

• remove the depdendency,
• runnin poetry install
• add it again
• runpoetry install

[jessielaf]

Is there an update on this issue?

[evdevdev]

I can confirm this issue is still present. Is there anything I can do to help document its reproducibility and/or help with implementing a fix?

[zerlok]

Same here.

I've organized a workaround into a simple bash script. Maybe it will help somebody.

# poetry-reinstall.sh
#!/bin/bash

uri=${1?"URI is required"}
shift

type=${uri%+*}
type_uri=${uri#*+}

case $type in
        git)
                git_uri=${type_uri##.git*}
                git_name_with_ref=${git_uri##*/}
                git_name=${git_name_with_ref#@*}
                name=${git_name%.git*}

                poetry remove ${name} && poetry lock && poetry add ${@} ${uri} 
                ;;
        *)
                echo "unsupported type ${type}"
                exit 1
                ;;
esac

Usage

# install dependency at first time
$ poetry add git+ssh://[email protected]/zerlok/asynchron.git@main

# reinstall dependency after update in the main branch
$ bash poetry-reinstall.sh git+ssh://[email protected]/zerlok/asynchron.git@main

[dimbleby]

Works fine with poetry 1.1.13:

msgpack = { git = "https://github.com/msgpack/msgpack-python", rev = "v1.0.3" }

after installing:

$ python -c 'import msgpack; print(msgpack.version)'
(1, 0, 3)

bump the rev to v1.0.4, both the lockfile and the installed package are updated:

$ poetry update
Updating dependencies
Resolving dependencies... (8.2s)

Writing lock file

Package operations: 0 installs, 1 update, 0 removals

  • Updating msgpack (1.0.3 -> 1.0.4 b5acfd5)

$ python -c 'import msgpack; print(msgpack.version)'
(1, 0, 4)

This can be closed

[indigoviolet]

Does not work out of the box for me with poetry 1.1.13.

However, here is a hacked together recipe that uses dasel to parse pyproject.toml. I use it with just to get the library argument, so here it is in justfile format, but it can be easily converted to a vanilla shell script:

# Update Poetry git dependency
poetry_reinstall_git LIB:
    #!/usr/bin/env bash
    uri=$(dasel select -f pyproject.toml -s '.tool.poetry.dependencies.{{LIB}}.git')
    rev=$(dasel select -f pyproject.toml -s '.tool.poetry.dependencies.{{LIB}}.rev')
    poetry remove {{LIB}} && poetry lock && poetry add "git+$uri@$rev"

[antdking]

It looks to work on 1.1.3 when there's been a package version change.
If the version number changes, the new source gets checked out.
No version change, no fresh checkout.

[finswimmer]

This should be solved in the latest Poetry version (1.2.2).

Feel free to leave a comment if you disagree.

[truenicoco]

I still experience this on 1.6.1. poetry update seems to update the dep (updating blah-blah-blah... with the proper commit hashes) but the version in the venv does not update. I need to manually pip uninstall blah-blah-blah && poetry install for the git dependencies to actually be updated in the venv.

[fredrikaverpil]

@truenicoco I don't see this behavior with 1.6.1. My venv gets updated as expected. I'm on macOS/zsh and I've installed poetry with pipx (python 3.11).

Maybe check the output of which python, which poetry, echo $VIRTUAL_ENV and poetry env info. I suspect Poetry is updating a different virtual environment than the one you expected.

[truenicoco]

nicoco@xxx ~/s/p/x/matridge (master)> which python
/usr/bin/python
nicoco@xxx ~/s/p/x/matridge (master)> which poetry
/home/nicoco/.local/bin/poetry
nicoco@xxx ~/s/p/x/matridge (master)> echo $VIRTUAL_ENV

nicoco@xxx ~/s/p/x/matridge (master)> poetry env info

Virtualenv
Python:         3.11.3
Implementation: CPython
Path:           /home/nicoco/.cache/pypoetry/virtualenvs/matridge-oqllwicX-py3.11
Executable:     /home/nicoco/.cache/pypoetry/virtualenvs/matridge-oqllwicX-py3.11/bin/python
Valid:          True

System
Platform:   linux
OS:         posix
Python:     3.11.3
Path:       /usr/local
Executable: /usr/local/bin/python3.11

Nothing suspicious here. Other deps get without trouble, it's just that specific git dependency that, despite the Updating... message with commit hashes definitely does not get updated.

[fredrikaverpil]

Can you provide the full logs from when you do this update?

poetry update <pkg> -vvv

[truenicoco]

Not the exact same repo because I did the pip uninstall XXX; poetry install dance, but another one: https://paste.sr.ht/~nicoco/66ac57a1378ed21c0a99c3087ac822e55f3804ae

And I could reproduce the bug, despite what the end of the log states, slidge in the matteridge-4b5Pn8h8-py3.11 virtual env is not at 071647d, but still at 293613b. Let me know if you want to test some more stuff in this state.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.