after poetry 1.2.0 release, poetry 1.1 installation is broken when it is installed with pip install

[sryabkov]

• I have searched the issues of this repo and believe that this is not a duplicate.
• I have searched the documentation and believe that my question is not covered.

Issue

As part of poetry 1.2.0 release, poetry-core 1.1.0 has been released as well. It seems to be incompatible with poetry 1.1.

In our Dockerfiles, we do this

RUN pip install poetry==1.1.4 \
  && poetry config virtualenvs.create false --local

After poetry 1.2.0 release, we are getting the following error:

ImportError: cannot import name 'PyProjectException' from 'poetry.core.pyproject'

The problem is caused by this command:

poetry config virtualenvs.create false --local

Our fix was locking poetry-core to version 1.0.8:

RUN pip install poetry==1.1.4 poetry-core==1.0.8 \
  && poetry config virtualenvs.create false --local

I don't know if this could be considered a bug, but it might be a missed test scenario or a missing warning in documentation. Regardless of the status, I wanted to create this issue in case anybody else has a similar problem.

[neersighted]

You are on an ancient version of Poetry, which had an unbounded poetry-core dep. Update to Poetry 1.1.15 and you will have a locked, compatible core version.

Versions <1.1.14 are unable to communicate with PyPI due to a breaking change on their side anyway.

[sryabkov]

@neersighted

We are in the middle of an upgrade to 1.2.0, but we have about 200 of Dockerfiles and 150+ of them got broken this morning, which completely blocked our CI and triggered an incident. At least now, our CI process is unblocked.

You are on an ancient version of Poetry, which had an unbounded poetry-core dep. Update to Poetry 1.1.15 and you will have a locked, compatible core version.

Thanks for the info.

Can you please confirm that replacing

RUN pip install poetry==1.1.4 poetry-core==1.0.8

with

RUN pip install poetry==1.1.15

won't require the re-creation of the poetry.lock files?

Versions <1.1.14 are unable to communicate with PyPI due to a breaking change on their side anyway.

Interesting, our builds seemed to have been working OK.

[neersighted]

We are in the middle of an upgrade to 1.2.0, but we have about 200 of Dockerfiles and 150+ of them got broken this morning, which completely blocked our CI and triggered an incident. At least now, our CI process is unblocked.

I'm sorry to hear that... Incoming deprecations and important fixes are announced using all the channels we have (Github releases, Discord, our blog/website, the code itself) -- do you have any feedback on how we could ensure you are aware of changes to Poetry that may require action from you?

Can you please confirm that replacing

RUN pip install poetry==1.1.4 poetry-core==1.0.8

with

RUN pip install poetry==1.1.15

won't require the re-creation of the poetry.lock files?

Poetry 1.2 and newer versions of Poetry 1.1 should be able to fully understand any lock file from 1.1.4, though there may be changes to case, punctuation, order, etc that make them not deterministic across versions.

Versions <1.1.14 are unable to communicate with PyPI due to a breaking change on their side anyway.

Interesting, our builds seemed to have been working OK.

You will be able to install, but poetry lock or poetry update on that version should fail as PyPI dropped releases from the API these versions use.

[DavidJVitale]

Our builds broke this morning as well, I believe related to a similar issue that @sryabkov described. We were able to fix it with the help of this thread and from reading poetry outputs. What fixed it for us was changing our build commands from something like this:

curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py -o get-poetry.py
python get-poetry.py --version 1.1.4
ln -s $HOME/.poetry/bin/poetry /usr/bin/poetry
poetry <whatever>

to this:

curl -sSL https://install.python-poetry.org -o get-poetry.py
python get-poetry.py --version 1.1.15
ln -s $HOME/.local/bin/poetry /usr/bin/poetry
poetry <whatever>

We needed to upgrade to version 1.1.15 , pull from https://install.python-poetry.org instead of the github URL, and have our symlink pull from $HOME/.local/bin instead of $HOME/.poetry/bin

It was partially a configuration error on our end, I hope this information is useful for anyone else out there.

[neersighted]

Two suggestions -- don't rename install-poetry.py / install.python-poetry.org to get-poetry.py as it will just confuse people trying to help you in the future.

The second one is if you want a global install of Poetry, just do that! The new installer is intentionally magic-free -- it really just sets up a virtual environment in a convenient way for simple pipeline/single developer use.

Instead try this:

python3 -m venv /opt/poetry
/opt/poetry/bin/pip install poetry==1.1.15
# /opt/poetry/bin/poetry is now available for usage or symlinking

[DavidJVitale]

Good to know, thanks! We will try that out :)

[sryabkov]

@neersighted

do you have any feedback on how we could ensure you are aware of changes to Poetry that may require action from you?

How about adding a special section to CHANGELOG.md for every release?

[neersighted]

@neersighted

do you have any feedback on how we could ensure you are aware of changes to Poetry that may require action from you?

How about adding a special section to CHANGELOG.md for every release?

The changelog is meant to be the canonical project history and to focus on technical changes only. Top-level summary items usually end up in Github releases, and more in-depth explanations in the release announcements...

If you think it would be useful to have detailed release notes somewhere in the repo, what makes files in the repository more visible/accessible than Github Releases or the blog/website?

[sryabkov]

@neersighted I am starting to wonder if the important part is not where but what is published... Sorry for the long post.

Before I start, are there release notes for 1.2.0? I mean besides what accompanies GitHub releases (e.g. https://github.com/python-poetry/poetry/releases/tag/1.2.0)? Maybe I missed something...

---

To declare my biases upfront, I assume that:

1. the majority of regular users do NOT care about pre-releases
2. the release notes need to be written for regular users

To give more context of where I am coming from, we recently did some work to get us to poetry 1.2.0, and we started with 1.2.0b3. We ran into a couple of roadblocks that weren't documented in release notes. We had to experiment and dig through the issues in the repo to find answers. We wrote an internal article about them to help everyone upgrade. It is an internal upgrade guide. Here are the main issues:

1. poetry.lock version and compatibility with poetry 1.1

poetry 1.2 can be used with lock files created by poetry 1.1, however poetry 1.1 is not compatible with poetry.lock created by poetry 1.2. This is related to the facts that poetry 1.2 has a built-in pip and that there was a bug in poetry 1.1.
The issue that seems to be describing this problem is #4242
We tried to temporarily upgrade the CI to poetry 1.2.0b3, and that resolved the problem. However, this is not an acceptable solution because developers, all of whom are running poetry 1.1 locally, would have the same problem when running poetry install locally. What's worse, poetry install might succeed but pytest can fail or there might even be some runtime errors. Everyone would need to move to poetry 1.2.0 at the same time.
There is nothing inside poetry.lock that says if it was created using poetry 1.1 or poetry 1.2. The only way we know how to tell apart the lock files created by 1.2 is look for references to setuptools. They should exist in poetry.lock created by poetry 1.2 and they should be absent in lock files created by poetry 1.1
Because it is not easy to tell apart poetry.lock created by poetry 1.1 from poetry files created by poetry 1.2, please mention in the PR when you create/re-create upgrade poetry.lock using poetry 1.2

2. private repo basic authentication

If you are trying to configure basic authentication for a private PyPi repo (poetry config http-basic.myreponame user password), you are likely to run into an issue described in #2692
This solution worked for us: #2692 (comment)..
The other way to handle this authentication problem is to set the following environment variable:
PYTHON_KEYRING_BACKEND="keyring.backends.null.Keyring"
The latter solution might be more convenient when updating Dockerfiles.

3. installer.max-workers=1 over experimental.new-installer=false

We have observed poetry (both 1.1 and 1.2) getting stuck on downloads (can be seen with poetry lock -vvv). For that reason, in a lot of places we do poetry config experimental.new-installer false
poetry maintainers claim this problem is caused by parallel pip installs when older versions of pip are used. They say the problem doesn't happen or happens much more rarely when recent versions of pip are used. If the problem does happen, they recommend configuring poetry config installer.max-workers 1 instead of poetry config experimental.new-installer false. This advice seems to have solved the issue when testing poetry 1.2.0b3. See #3336 (comment) for broader context.
The installer.max-workers setting was introduced in poetry 1.2b1. See #3516

---

I understand that the distinction between the product issue and the issue with using the product is a subtle one, and as a maintainer, you have to draw the line somewhere. 1.2.0 is a big release, which was long time coming (if you look from the perspective of a regular user who doesn't mess with pre-releases) and I think the poetry project would benefit from making the transition as smooth as possible. So, arguably it might have been justified for someone to comb the repo for issues and put up a public upgrade guide warning users about issues that can hinder or slow down their upgrade process.

For example, I'd argue that a lot of people are going to run into issue 1 above during the upgrade process if they are temporarily mixing poetry 1.1 and 1.2 and that they would have appreciated the heads up.

---

I also want to bring up an important information bit that you mentioned in this thread that I didn't see anywhere else:

"Versions <1.1.14 are unable to communicate with PyPI due to a breaking change on their side anyway."

Shouldn't have this been front and center in the poetry 1.1.15 release notes? To me, this means "upgrade immediately", but there is nothing in the release notes.

---

To sum up, to give you and all other poetry maintainers credit, all the information that is required is available if you search the issues. It could be made easier to find, but that's really a separate aspect of maintaining the product. For 1.2.0, I'd recommend an "Upgrading to 1.2.0" section in the documentation. For other important info, maybe add a "Known Issues" section to the documentation website?

[neersighted]

@sryabkov The canonical guide to upgrading is the release announcement at https://python-poetry.org/blog/announcing-poetry-1.2.0/ -- PRs against that post are welcome.

Regarding 1.1.15, note the version 1.1.14 contains the fix and it is documented in the changelog, the release notes and in a release announcement.

Certainly more documentation would be wonderful -- we are doing our best to keep on top of it, and 1.2 being a massive overdue release has made this especially painful. That being said, I'm not sure that fundamentally we can do anything to communicate changes more widely based on what has been discussed here -- the takeaway seems to be that we should instead focus on the quality and centralization of information.

[sryabkov]

@neersighted FYI, please see python-poetry/website#79

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.