Regression: Poetry uses the wrong version of Python

[sargunv]

brew gist-logs <formula> link OR brew config AND brew doctor output

HOMEBREW_VERSION: 3.6.18-42-g4abc51f
ORIGIN: https://github.com/Homebrew/brew
HEAD: 4abc51f32869511eaa29c7feaa30312aec70e1d2
Last commit: 15 hours ago
Core tap ORIGIN: https://github.com/Homebrew/homebrew-core
Core tap HEAD: b52e0ef46820f2454d0d2fe5b9c82af8038ecc53
Core tap last commit: 3 hours ago
Core tap branch: master
HOMEBREW_PREFIX: /opt/homebrew
HOMEBREW_CASK_OPTS: []
HOMEBREW_MAKE_JOBS: 8
Homebrew Ruby: 2.6.10 => /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/bin/ruby
CPU: octa-core 64-bit arm_firestorm_icestorm
Clang: 14.0.0 build 1400
Git: 2.39.0 => /opt/homebrew/bin/git
Curl: 7.84.0 => /usr/bin/curl
macOS: 13.0.1-arm64
CLT: 14.2.0.0.1.1668646533
Xcode: 14.2
Rosetta 2: false

Please note that these warnings are just used to help the Homebrew maintainers
with debugging if you file an issue. If everything you use Homebrew for is
working fine: please don't worry or file an issue; just ignore this. Thanks!

Warning: Putting non-prefixed coreutils in your path can cause GMP builds to fail.

Verification

• My "brew doctor output" says Your system is ready to brew. and am still able to reproduce my issue.
• I ran brew update and am still able to reproduce my issue.
• I have resolved all warnings from brew doctor and that did not fix my problem.

What were you trying to do (and why)?

Trying to poetry install with Python v3.8.13 in my PATH, in a project that requires Python >=3.8.5,<3.10.0 according to its pyproject.toml.

The python in my PATH is v3.8.13 (from pyenv):

The python required by my project using Poetry is >=3.8.5,<3.10.0

So, a poetry install in this project should work.

What happened (include all command output)?

Poetry somehow runs with Python 3.11.1, so poetry install fails as 3.11.1 does not meet the constraints in the project:

Even poetry env list shows the venv is named as if it's a 3.8 venv:

With poetry env info we can see that poetry is ignoring my PATH and instead using the python from its homebrew package:

And here's the culprit. The #! of the poetry bin hardcodes the Python path. EDIT: official installer does too, and does not exhibit the same bug.

What did you expect to happen?

I expected Poetry to use the Python from my PATH to install the project I'm working on (that's compatible with the Python from my PATH).

Step-by-step reproduction instructions (by running brew commands)

brew install pyenv
pyenv install 3.8.13
pyenv global 3.8.13
brew install poetry
poetry init -n --name test --python 3.8.13
poetry install

[sargunv]

This is a regression of #62910, which was fixed in c9b23e8 (Feb 11, 2021). I believe it was broken in 763ca9d (Dec 7, 2021).

[sargunv]

Workaround: poetry env remove --all and poetry env use "$(which python3)" to get the right env.

[sargunv]

Interestingly, in the commit that I believe broke it, there's a test that attempts to test for regressions of #62910, but the test uses poetry env use to explicitly select a Python path and therefore doesn't trigger the bug.

This test no longer exists in the latest version of the formula.

homebrew-core/Formula/poetry.rb

Lines 205 to 214 in 763ca9d

	# Check using different python versions
	# See https://github.com/Homebrew/homebrew-core/issues/62910
	pythons = deps.map(&:to_formula).select { |f| f.name.match?(/^python@3\.\d+$/) }
	cd testpath/"homebrew" do
	inreplace "pyproject.toml", /^python = "\^3.*"/, 'python = "^3"'
	pythons.each do |python|
	system bin/"poetry", "env", "use", python.opt_bin/"python3"
	assert_match python.version.to_s, shell_output("#{bin}/poetry run python --version")
	end
	end

[cho-m]

Homebrew's installation is similar to the documented manual installation (https://python-poetry.org/docs/#installing-manually) as we essentially do a pip install inside a venv.

I think the official installer also just installs into a venv which adds the corresponding venv shebang. Shebangs are generated by pip and not something we modify (and modifying it will probably break usage as it won't find venv).

Given above, this seems more like an upstream issue, maybe python-poetry/poetry#7158

---

EDIT: Quoting from above issue python-poetry/poetry#7158 (comment)

Poetry finds python3 as a suitable executable, because it's version is 3.10.6. Now python3 is passed as an argument to virtualenv. virtualenv runs in Poetry's own venv, so python3 points to Python 3.9.

This sounds like what you are experiencing, i.e. output claims to have found python3 with v3.8.13 but then resolves it to internal poetry venv's v3.11.1.

[sargunv]

This may be fixable upstream, but I'm unable to replicate the problem when poetry is installed with the official install instructions. Something is different between what Brew is doing and what the official installer is doing, and that something is triggering the bug (at least for me).

However, you are correct that the one from the official installer also hardcodes the python in the shebang:

[sargunv]

Will keep an eye on python-poetry/poetry#7221 (PR) to see if the upstream fix also applies here though.

EDIT: that's merged, now following python-poetry/poetry#7357

[the-serious-programmer]

@sargunv thanks so much for the mentioned workaround:

Workaround: poetry env remove --all and poetry env use "$(which python3)" to get the right env.

I was literally spending hours and hours on why my poetry kept falling back to python 3.11 instead of 3.9.
Originally came from: python-poetry/poetry#7158

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[sargunv]

should be fixed in 1.4.0

[rtviii]

Still having this issue.