Improve discoverability of available images

[rhcarvalho]

Every time I decide to bump image versions it gets frustrating to discover what is the lastest Elixir + OTP + base image that is a valid tag.

Referring to a previous discussion:

These images are PITA to search for (Docker Hub being pretty slow and not that easy to use in my opinion) so if there is a way to make that easier that would help. Perhaps it's possible for a community-maintained example.com where FROM example.com/elixir:latest "redirects" to a hexpm elixir one?

Originally posted by @wojtekmach in #180 (comment)

Before I volunteer to publish the "latest" combination somewhere, is there a way to get it from build/job logs or somewhere that doesn't involve getting multiple pages from the Docker Hub API?

[ericmj]

Unfortunately not, Docker Hub is the authority of what images are available.

[rhcarvalho]

Hi Eric!

Unfortunately not, Docker Hub is the authority of what images are available.

Okay, thanks for the answer!

The job that publishes the images runs in some private infrastructure, right? Is the README up-to-date when it says that new images are built (possibly) every 15 minutes?

Docker Hub being the source of truth, I'll see what I can put together with their API.

[rhcarvalho]

I saw that there's already code in the repository that fetches the tags from Docker Hub (and even has a caching mechanism, seems battle-tested). Perhaps we just need to expose the list somewhere?

I'm trying the existing code locally and seems that there are a lot of pages, so constantly running into rate limits (fortunately with retries).

---

For later reference

iex(1)> Bob.Job.DockerChecker.expected_elixir_tags()
[warning] DockerHub https://hub.docker.com/v2/repositories/library/debian/tags?page=57&page_size=100 RATE LIMIT
[warning] DockerHub https://hub.docker.com/v2/repositories/library/debian/tags?page=59&page_size=100 RATE LIMIT
[warning] DockerHub https://hub.docker.com/v2/repositories/library/debian/tags?page=60&page_size=100 RATE LIMIT
[warning] DockerHub https://hub.docker.com/v2/repositories/library/debian/tags?page=61&page_size=100 RATE LIMIT
[error] GenServer ExAws.Config.AuthCache terminating
** (RuntimeError) Instance Meta Error: {:error, %{reason: :ehostunreach}}

You tried to access the AWS EC2 instance meta, but it could not be reached.
This happens most often when trying to access it from your local computer,
which happens when environment variables are not set correctly prompting
ExAws to fallback to the Instance Meta.

iex(2)> Bob.Job.DockerChecker.elixir_tags() |> Enum.into([])
[warning] DockerHub https://hub.docker.com/v2/repositories/hexpm/elixir-amd64/tags?page=182&page_size=100 RATE LIMIT
[warning] DockerHub https://hub.docker.com/v2/repositories/hexpm/elixir-amd64/tags?page=183&page_size=100 RATE LIMIT
[warning] DockerHub https://hub.docker.com/v2/repositories/hexpm/elixir-amd64/tags?page=184&page_size=100 RATE LIMIT
[warning] DockerHub https://hub.docker.com/v2/repositories/hexpm/elixir-amd64/tags?page=185&page_size=100 RATE LIMIT
[warning] DockerHub https://hub.docker.com/v2/repositories/hexpm/elixir-amd64/tags?page=186&page_size=100 RATE LIMIT
[warning] DockerHub https://hub.docker.com/v2/repositories/hexpm/elixir-amd64/tags?page=187&page_size=100 RATE LIMIT
[warning] DockerHub https://hub.docker.com/v2/repositories/hexpm/elixir-amd64/tags?page=188&page_size=100 RATE LIMIT
[warning] DockerHub https://hub.docker.com/v2/repositories/hexpm/elixir-amd64/tags?page=189&page_size=100 RATE LIMIT
(...)

[ericmj]

The job that publishes the images runs in some private infrastructure, right? Is the README up-to-date when it says that new images are built (possibly) every 15 minutes?

Yes and yes.

I saw that there's already code in the repository that fetches the tags from Docker Hub (and even has a caching mechanism, seems battle-tested). Perhaps we just need to expose the list somewhere?

Yes, we have a local cached copy of all tags in memory. With some logic to derive the latest version we can write the latest version to https://builds.hex.pm like we do for the elixir builds https://github.com/hexpm/bob/blob/main/priv/scripts/elixir/elixir.sh#L95 https://builds.hex.pm/builds/elixir/builds.txt. WDYT?

[rhcarvalho]

Yes, we have a local cached copy of all tags in memory.

Awesome, I was able to get one in memory on iex as well using the existing code 💜

With some logic to derive the latest version (...) builds.txt. WDYT?

Yes, I think it would solve the pain 👍

Even if some more parsing would be necessary, working with a file like https://builds.hex.pm/builds/elixir/builds.txt would be a much simpler and faster task than fetching data from Docker Hub.

I originally thought of contributing a GitHub job that automatically updates the README with a table, something like (but not exactly the same as):

https://github.com/rhcarvalho/cedict/blob/main/.github/workflows/update.yml

But a link in the README to a file with a list of latest versions for a given combo of Elixir/OTP major versions and base OS/arch would address the pain of figuring out updates. With a small enough list, ctrl+f would be enough.

---

To illustrate a bit the workflow that I'm trying to address, this is a recent update I did:

diff --git Dockerfile Dockerfile
index 5eb24e9..61ee2cc 100644
--- Dockerfile
+++ Dockerfile
@@ -11,9 +11,9 @@
 #   - https://pkgs.org/ - resource for finding needed packages
 #   - Ex: hexpm/elixir:1.15.7-erlang-26.1.2-debian-bullseye-20231009-slim
 #
-ARG ELIXIR_VERSION=1.15.7
-ARG OTP_VERSION=26.1.2
-ARG DEBIAN_VERSION=bookworm-20231009-slim
+ARG ELIXIR_VERSION=1.16.2
+ARG OTP_VERSION=26.2.3
+ARG DEBIAN_VERSION=bookworm-20240130-slim

 ARG BUILDER_IMAGE="docker.io/hexpm/elixir:${ELIXIR_VERSION}-erlang-${OTP_VERSION}-debian-${DEBIAN_VERSION}"
 ARG RUNNER_IMAGE="docker.io/debian:${DEBIAN_VERSION}"

Today when updating another project I realized there was a newer base image version bookworm-20240423 on the list of debian versions, but filtering the hexpm/elixir images on Docker Hub returns so many results that until I got the list in my local iex, I couldn't spot that I needed to bump the OTP version too:

diff --git Dockerfile Dockerfile
index 61ee2cc..702b2d8 100644
--- Dockerfile
+++ Dockerfile
@@ -12,8 +12,8 @@
 #   - Ex: hexpm/elixir:1.15.7-erlang-26.1.2-debian-bullseye-20231009-slim
 #
 ARG ELIXIR_VERSION=1.16.2
-ARG OTP_VERSION=26.2.3
-ARG DEBIAN_VERSION=bookworm-20240130-slim
+ARG OTP_VERSION=26.2.4
+ARG DEBIAN_VERSION=bookworm-20240423-slim

 ARG BUILDER_IMAGE="docker.io/hexpm/elixir:${ELIXIR_VERSION}-erlang-${OTP_VERSION}-debian-${DEBIAN_VERSION}"
 ARG RUNNER_IMAGE="docker.io/debian:${DEBIAN_VERSION}"

That's why I wanted a quick way to get the latest valid combo, in this case for Elixir 1.16, OTP 26, Debian bookworm, amd64.

[sheerlox]

I came up last month with a Renovate configuration that keeps hexpm/elixir up-to-date in my Dockerfile.

After stumbling upon this issue this morning, I created a minimum reproduction repository, and I just shared it on the forum: https://elixirforum.com/t/automatic-updates-of-elixir-docker-image-with-renovate/64170

Link to the repo: https://github.com/sheerlox/elixir_renovate_demo (you'll find extensive information in the README 😉)