RFC: how to deal with poetry 1.3's new lockfile format

[DMRobertson]

I've spent some time on this today and now want to dump context.

• Dependabot's batch of PRs this morning all propose a big change to the way the lockfile is presented. E.g. Bump ruff from 0.0.215 to 0.0.223 #14851
• Our CI uses poetry 1.2.0, which fails to read this lockfile.
• I am guessing this is due to Update poetry requirement from <1.3.0,>=1.1.15 to >=1.1.15,<1.4.0 in /python/helpers dependabot/dependabot-core#6302 allowing the use of poetry 1.3.
• Poetry 1.3 introduces a new lockfile format: feat: lock file format 2.0 (was: store package files on the package in lockfile) python-poetry/poetry#6393
• In feat: lock file format 2.0 (was: store package files on the package in lockfile) python-poetry/poetry#6393 (comment) it was proposed that the ability to read (not write) the new format should be backported to 1.2.x, which happened here [1.2] feat: forward compatibility for lock file format 2.0 python-poetry/poetry#6608

Options:

0. Stop using dependabot.
1. Ask dependabot authors if there's some way to pin the version of poetry that dependabot uses.
2. Convert to poetry 1.3.x's lockfile format. Require poetry >= 1.2.2.

I assumed that option (0) was undesirable. Put differently, I contend that dependabot is valuable to us. It does a good job of summarising what's changed in dependencies, and also automatically proposes updates when security advisories are made for our dependencies.

Option (1) seemed like a relatively big ask for a fairly small scenario, so I put that to one side. EDIT: The depdendabot authors seem to be aware of this problem, see dependabot/dependabot-core#1556 (comment) and the discussion following it. In dependabot/dependabot-core#1556 (comment) they point to a poetry issue python-poetry/poetry#3316 for enforcing a minimum poetry issue, which remains open. (Note that we actually want a maximum version here).

Option (2) was my next approach. I started this process in
matrix-org/setup-python-poetry#14 and matrix-org/sytest#1326. But thinking about it, I realised that anyone who wants to edit the lockfile will have to install poetry 1.3.x. This includes some parts of CI (e.g. old deps, twisted trunk); so there's no way of being consistent; option(2) is really

2'. Convert to poetry 1.3.x's lockfile format. Require poetry >= 1.2.2 to read it, and >= 1.3.0 to write it.

but writing this down in our docs/CI sounds like a recipe for future confusion. Instead, I propose the simpler

3. Convert to poetry 1.3.x's lockfile format. Require poetry >= 1.3 to use it. (1.2.2 will suffice for read-only operations---but we wouldn't advertise this fact.)

[clokep]

Convert to poetry 1.3.x's lockfile format. Require poetry >= 1.3 to use it. (1.2.2 will suffice for read-only operations---but we wouldn't advertise this fact.)

I think that this is reasonable if we don't have any issues updating to Poetry 1.3.x.

[DMRobertson]

Convert to poetry 1.3.x's lockfile format. Require poetry >= 1.3 to use it. (1.2.2 will suffice for read-only operations---but we wouldn't advertise this fact.)

I think that this is reasonable if we don't have any issues updating to Poetry 1.3.x.

At the risk of tempting fate, matrix-org/sytest#1327 and #14860 seem to have Just Worked ™️!

[DMRobertson]

Sorted in #14860... hopefully.