poetry installed with install-poetry.py does not respect asdf python version

[shawon-crosen]

Due to the way poetry is installed by asdf and the way that poetry decides which python verion it should use, it is only able to use the version of python that is created as a venv in it's install path when it is installed.

Example:
Set asdf to use python 3.8.7 globally

wintermute:~/CircleCi/Github/dbt-data-modeling:(fix_cves_8_05)> asdf global python 3.8.7
wintermute:~/CircleCi/Github/dbt-data-modeling:(fix_cves_8_05)> python --version
Python 3.8.7

Output during poetry install with python version specified as "^3.8" in pyproject.toml:

Creating virtualenv dbt-data-modeling-5cvI_FFT-py3.9 in /Users/shawon/Library/Caches/pypoetry/virtualenvs
Using virtualenv: /Users/shawon/Library/Caches/pypoetry/virtualenvs/dbt-data-modeling-5cvI_FFT-py3.9

Output of poetry env:

Poetry
Version: 1.1.7
Python:  3.9.6

Virtualenv
Python:         3.9.6
Implementation: CPython
Path:           NA

System
Platform: darwin
OS:       posix
Python:   /Users/shawon/.asdf/installs/poetry/1.1.7/venv

I believe this is related to this issue, specifically the comment by finswimmer: python-poetry/poetry#1888

[crflynn]

This seems like an issue with the new installer provided by poetry. The issue is discussed here:

• After the installation of poetry via the new script install-poetry.py poetry ignores pyenv local python version python-poetry/poetry#4199
• Poetry does not use active pyenv when creating virtual environment python-poetry/poetry#651 (comment) <- explanation
• poetry seems to ignore virtualenvs.create python-poetry/poetry#4187 (comment) <- more explanation

I don't know if there is an easy workaround here until they resolve this (if they choose to), but it unfortunately breaks the way we use poetry with asdf.

We could go back to the old installer, and this problem would go away. However we wouldn't be able to use 1.2+ versions of poetry, and the old installer is planned to be deprecated.

I don't think there is much we can easily do here unless poetry changes the way it is installed.

[shawon-crosen]

That makes sense, thank you for the thorough response!

Bummer that the installer changed that, I really liked being able to have different poetry versions available via asdf.

My workaround for now is to install poetry with the script, then just point it to the correct python asdf version by running poetry env $ASDF_INSTALL_PATH/python. That works for now.

Feel free to close this if you'd like, I understand this is pretty much out of your hands.

[crflynn]

I'm wondering if we can incorporate some workaround here to have it use the asdf specified version of python. Otherwise I'm considering using the old installer until it is fully deprecated.

[crflynn]

My workaround for now is to install poetry with the script, then just point it to the correct python asdf version by running poetry env $ASDF_INSTALL_PATH/python. That works for now.

I'm not sure this would get you the virtualenv. I might suggest using

poetry env use $(asdf which python)

instead when creating a new venv, especially if you also specify a local python version using asdf local python x.y.z.

[crflynn]

@smorimoto if I could ask for some bash help I wonder if we can conditionally install with the legacy get-poetry.py if ASDF_INSTALL_VERSION <1.2.0 and use the new install-poetry.py otherwise.

[Kurt-von-Laven]

Hopefully the workaround that worked for me works for you all as well.

[Kurt-von-Laven]

The maintainers of Poetry recently pointed out (c.f., #3345) that they have since resolved this issue on master, but the fix (#4433) has not yet been released. Look for it on poetry > 1.2.0a2 once that is released. At that time, I believe this plugin will be able to stop using the deprecated get-poetry.py even for poetry < 1.2.

[tony]

At that time, I believe this plugin will be able to stop using the deprecated get-poetry.py even for poetry < 1.2.

@Kurt-von-Laven: Correct, since June 2021.

install-poetry.py works officially on 1.1.7 and up. As of 6176ff2 is hard coded for 2.x, so 1.1.7 - 1.1.11 won't be retrieved.

https://github.com/python-poetry/poetry/releases/tag/1.1.7 first release supporting install-poetry.py.

Due to bugs in 1.1.7 and 1.1.8, install-poetry.py may break in usage due to a hash bug: python-poetry/poetry#4409. For that reason I made a PR for 1.1.9+ to not set up users to fail.

asdf-poetry/bin/install

Line 34 in 6176ff2

semver_ge "$ASDF_INSTALL_VERSION" 1.2.0 && vercomp="ge" || vercomp="lt"

This updating this resolves the ModuleNotFoundError: No module named 'cleo' on python 3.10, and also makes us more compliant with poetry

PR at #14 #16

[tony]

If we're receiving ModuleNotFoundError: No module named 'cleo' the new installer fixes it for me.

Installer's new repo: https://github.com/python-poetry/install.python-poetry.org (independent of poetry releases)

curl -sSL https://install.python-poetry.org | python -

We can now use: https://install.python-poetry.org in place of the URL at

asdf-poetry/bin/install

Line 37 in 6176ff2

install_url="https://raw.githubusercontent.com/python-poetry/poetry/master/install-poetry.py"

PR at #15

[Kurt-von-Laven]

There is now an open PR proposing the addition of a setting that would allow the installation of Poetry using the current version of Python. The author, @finswimmer, is asking for feedback. Would there be any concerns with using such an option to install Poetry in this asdf plugin?

[tony]

I've did a test with .python-version in a repo, looks good

My default system python is 3.7.9 but when inside the repo, python correctly shows as 3.10.0

~ ❯ python -V
Python 3.7.9
~ ❯ cd ~/projects/project-with-3.10
❯ python -V
Python 3.10.0

[tony]

@Kurt-von-Laven On a separate note, in the issue you are testing with 1.1.7 (where I still get the error)

if you use poetry 1.1.11 and have .python-version set in your projects, does it obey the python version?

In my testing this is working, but I've installed via this: ASDF_POETRY_INSTALL_URL=https://install.python-poetry.org asdf install poetry 1.1.11

How does it work then?

(It's difficult to track all the issues between this and poetry's tracker, so trying to get an idea what's working and what's not as of now)

[Kurt-von-Laven]

@tony, sorry I completely missed this. I am using .tool-versions rather than .python-version, although I don't think the distinction is salient to any of the issues at hand. I updated to asdf v0.9.0 and reinstalled the latest version of the asdf-poetry plugin. I see the expected version of Python (from .tool-versions) using both poetry 1.1.7 and poetry 1.1.11. I didn't even have to use ASDF_POETRY_INSTALL_URL. I wonder if asdf-poetry is already using the setting introduced by the aforementioned PR?

[crflynn]

This is fixed by #21, though only for poetry 1.2+.

[mwdiers]

On the latest version of asdf (0.11.1) and poetry (1.3.2), this is still a problem. The local python version is ignored by poetry unless I create the environment with poetry env use $(asdf which python)

[Kurt-von-Laven]

Are you able to reproduce the issue after uninstalling Poetry and then reinstalling it while the desired version of Python is active? The local Python version at the time of installation is the one that is used, but it doesn't automatically update.

[SergiyKolesnikov]

On the latest version of asdf (0.11.1) and poetry (1.3.2), this is still a problem. The local python version is ignored by poetry unless I create the environment with poetry env use $(asdf which python)

Setting the virtualenvsprefer-active-python-experimental to true (e.g., in your shell's rc file) will make poetry respect the currently active version of Python. So, my current workflow looks something like this:

export POETRY_VIRTUALENVS_PREFER_ACTIVE_PYTHON=true  # this line is not needed if the env var is set in your shell's rc file
mkdir project-dir
cd project-dir
asdf local python 3.10.9   # change to the required Python version
poetry init --python '~3.10.9'  # change to the required Python version
poetry install

NOTE: I specify --python '~3.10.9' to pin allowed Python version to >=3.10.9, <3.11.0. This is useful if a python package from the project requires a specific minor version of Python. If the Python version is not specified like that, the version will be ^3.10 by default, which will allow all minor versions in the range >=3.10.0, <4.0.0.

Thanks to @finswimmer for posting this python-poetry/poetry#1888 (comment)