Publish images for different builds of python

[rarkins]

We need to support multiple builds of python, e.g. at least 3.7 and 3.8 but probably also down to full precision like 3.7.5.

We also need to work out a naming convention, e.g. renovate/poetry:1.0.0-python3.7 and renovate/poetry:1.0.0-python3.8.

[rarkins]

Depends on renovatebot/docker-python#18

[itajaja]

this might be a little nasty if you want to provide full compatibility of any python version/potery version pair. some folks might decide for whatever reason to do poetry:1.0.1 and python 3.8.2. this will lead to a cartesian explosion of base images. I was thinking, maybe it makes sense to download a python version dynamically in the image using something like pyenv? the downside being that it will add much more runtime

[rarkins]

For Bundler we prebuild Ruby images and dynamically install Bundler as it adds only a few seconds. Most package managers are unfortunately much longer to install.

[rarkins]

An idea I'd like to explore is if we can have a base image with multiple versions of Python installed using pyenv and then dynamically pick the right one at runtime before running poetry.

[rarkins]

Relevant: python-poetry/poetry#305

Concerning:

In poetry, it creates the virtual environment from the python against which poetry was installed, or uses a pre-made .venv

[rarkins]

This is a guide to pyenv and poetry together: https://blog.jayway.com/2019/12/28/pyenv-poetry-saviours-in-the-python-chaos/

What I'm hoping for is:

• Create an image with "all" versions of python 3 installed via pyenv
• Once done, install Poetry using the latest Python version (although it potentially doesn't matter which)
• At runtime in Renovate, execute some pyenv abc command prior to running python xyz

[itajaja]

I'd say the only potential caveat, if you are also supporting all patch versions, is that the image might be humongous to pull

[rarkins]

Yeah, we need to check that and then evaluate against how much data we'd ultimately pull anyway if we have a matrix of python X poetry versions and users spread out across many.

On a fairly fast mac on a fairly fast internet connection, poetry takes 38s to install:

❯ time curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python
Retrieving Poetry metadata

# Welcome to Poetry!

This will download and install the latest version of Poetry,
a dependency and package manager for Python.

It will add the `poetry` command to Poetry's bin directory, located at:

$HOME/.poetry/bin

This path will then be added to your `PATH` environment variable by
modifying the profile files located at:

$HOME/.profile
$HOME/.zprofile
$HOME/.bash_profile

You can uninstall at any time by executing this script with the --uninstall option,
and these changes will be reverted.

Installing version: 1.0.2
  - Downloading poetry-1.0.2-darwin.tar.gz (10.00MB)

Poetry (1.0.2) is installed now. Great!

To get started you need Poetry's bin directory ($HOME/.poetry/bin) in your `PATH`
environment variable. Next time you log in this will be done
automatically.

To configure your current shell run `source $HOME/.poetry/env`

curl -sSL   0.01s user 0.01s system 4% cpu 0.460 total
python  1.49s user 2.48s system 10% cpu 38.199 total

~ 38s

It's slow but manageable, if we have no other solution.

[itajaja]

any update on this? we unfortunately can't update to python 3.8 yet because of renovate

[rarkins]

@itajaja I wasn't able to get a definite answer on the feasibility of running multiple python versions etc. Any research/assistance you can do on that would be appreciated. e.g. could we:

• Use pyenv
• Install python 3.7 and 3.8
• Install poetry into python 3.8
• At runtime, set python to be the one needed and then run Poetry?

[itajaja]

hey @rarkins happy to help. I don't know the internals of renovate much, and these choices seem more like implementation details than user facing, but I might be misunderstanding.
I use pyenv locally, it's a good workflow for me, and would work in principle in the reonvate image, the only caveat as expressed before being image size, but if you limit to only a small number of latest minor versions (eg 3.6, 3.7, 3.8) should be good!

[rarkins]

This only slightly related to Renovate's internals, so don't worry there. Something we'd like to avoid is needing to build/publish a cross-matrix of every supported python version X every poetry version. So instead I'm trying to work out if we can have one published image per Poetry version that runs on a base that includes all python versions and switches them using pyenv.

What I don't know for example is:

• If I have three python versions, so I need to install poetry 3 separate times? Or can I install it into python 3.8 only, for example?
• What do I need to do in order for poetry to "find" the right poetry version each time?

[itajaja]

• If I have three python versions, so I need to install poetry 3 separate times?

no, no need. so you can first change the python version with pyenv and then run poetry

[rarkins]

Thanks! So then the Dockerfile would look something like this? (pseudocode):

• Install pyenv
• Install python 3.6 in pyenv
• Install python 3.7 in pyenv
• Install python 3.8 in pyenv and set to default
• Install poetry into python 3.8

And then at runtime, we'd do something like this (again.. pseudocode):

• pyenv select python 3.7
• poetry X .....

[itajaja]

yeah! except instead of "install poetry into python 3.8", the best way to install poetry is globally with vendorized deps. see https://python-poetry.org/docs/#installation.

3.[6-8] seems like a fair range to me, but it's up to you what versions you want to support :)

[rarkins]

I'm wondering if this helps us? python-poetry/poetry#1477

Is it "reasonable" that we expect users to follow this approach, or is it too limiting?

[rarkins]

I'm going to "bat signal" @sdispater here, hope he may have the time to look.

@sdispater here's what we want to do:

• Have a pre-built Docker image for each version of poetry that exists (1.0.0, 1.0.1, etc)
• Each pre-built image includes multiple Python runtimes, so that we can be compatible with the projects it's run on
• Either poetry magically selects the right Python at runtime itself, or we can run a "pre-command" prior to poetry that (for example) aliases the correct Python version globally

It's acceptable if we need to tell users "if you want this to work in all scenarios then configure your poetry like X" but ideally it would "just work" for anyone using Poetry 1.x.

[itajaja]

I think what that would allow you to do is to avoid introspecting the pytproject.py file to look for the python version. it's doable without it, but it's less work for you if that gets in

[viceice]

This should work with latest version of renovate 19.231.0