Support docker engine 25.x

[sparrc]

Summary

Since the release of docker engine 25.x (Jan 19 2024), the docker engine no longer supports API versions less than 1.24. See deprecated API versions here: https://docs.docker.com/engine/api/#deprecated-api-versions

This issue would show up in the agent logs like:

Error response from daemon: client version 1.21 is too old. Minimum supported API version is 1.24, please upgrade your client to a newer version

1. This PR allows the agent to dynamically determine the minimum/default API version that it should use based on what versions the docker engine on the instance supports.
2. This PR also updates the logic around advertising capabilities to the ECS backend. We will now advertise all known API versions that we are compatible with, rather than using the response from the docker API. This is for the sake of task placement requiredAttributes, which are static strings like docker-remote-api.1.19. Since some tasks will have these attributes added to them, we need to advertise that we support this remote API version even though technically it is deprecated and we no longer support it. Although we don't support using the specific API version, we still should be able to support all functionality from that version in the minimum docker API version.
3. When the ECS backend specifies a docker API version in it's container payload (via the DockerConfig.Version field on the container struct) we might use a different API version to actually create and start the container if the specified version is less than the minimum API version. For example, if the ECS backend specifies using docker API version 1.19 when we are using Docker engine 25, we will create/start that container using API 1.24.

fixes aws/containers-roadmap#2267
fixes #4074

NOTE: Our minimum docker engine version supported is moving from 1.9.0 to 1.12.0, which maps to api version 1.24. Docker API versions prior to 1.24 are deprecated as of Jan 2024. See https://docs.docker.com/engine/api/#deprecated-api-versions.

Docker version 1.12 was released in 2016 and is not supported in Amazon Linux.

Details

After this change, we will check all versions of the Docker API on startup, and then set agent's minimum/default version to whatever is the minimum supported. So when starting up on Docker 25.x we see log messages like this appear:

deprecated api versions fail (info-level messages)

level=info time=2024-01-25T23:25:00Z msg="Unable to get Docker client for version 1.17: Error response from daemon: client version 1.17 is too old. Minimum supported API version is 1.24, please upgrade your client to a newer version" module=sdkclientfactory.go
level=info time=2024-01-25T23:25:00Z msg="Unable to get Docker client for version 1.18: Error response from daemon: client version 1.18 is too old. Minimum supported API version is 1.24, please upgrade your client to a newer version" module=sdkclientfactory.go
level=info time=2024-01-25T23:25:00Z msg="Unable to get Docker client for version 1.19: Error response from daemon: client version 1.19 is too old. Minimum supported API version is 1.24, please upgrade your client to a newer version" module=sdkclientfactory.go
level=info time=2024-01-25T23:25:00Z msg="Unable to get Docker client for version 1.20: Error response from daemon: client version 1.20 is too old. Minimum supported API version is 1.24, please upgrade your client to a newer version" module=sdkclientfactory.go
level=info time=2024-01-25T23:25:00Z msg="Unable to get Docker client for version 1.21: Error response from daemon: client version 1.21 is too old. Minimum supported API version is 1.24, please upgrade your client to a newer version" module=sdkclientfactory.go
level=info time=2024-01-25T23:25:00Z msg="Unable to get Docker client for version 1.22: Error response from daemon: client version 1.22 is too old. Minimum supported API version is 1.24, please upgrade your client to a newer version" module=sdkclientfactory.go
level=info time=2024-01-25T23:25:00Z msg="Unable to get Docker client for version 1.23: Error response from daemon: client version 1.23 is too old. Minimum supported API version is 1.24, please upgrade your client to a newer version" module=sdkclientfactory.go

minimum api version is bumped from 1.21 to 1.24:

level=info time=2024-01-25T23:25:00Z msg="Setting minimum docker API version" newMinAPIVersion=1.24 previousMinAPIVersion=1.21

while 1.17-1.23 are not directly supported, they are supported in "extended" versions in order to advertise to the ECS backend that we support features added in these versions (such as awslogs added in API version 1.19):

level=debug time=2024-01-25T23:25:00Z msg="Extended supported versions" supportedVersions=[1.17 1.18 1.19 1.20 1.21 1.22 1.23 1.24 1.25 1.26 1.27 1.28 1.29 1.30 1.31 1.32 1.33 1.34 1.35 1.36 1.37 1.38 1.39 1.40 1.41 1.42 1.43 1.44]

Testing

unit and functional tests

Description for the changelog

Bug: Fix ECS agent docker API failures on docker engine 25.x

Licensing

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[yinyic]

Nit - should probably update this

[sparrc]

This minimum version has actually not changed on Linux, so we should be good to leave this here

[raphael5178]

Any prediction on when a new version with this fix will be released?

[iforapsy]

@raphael5178 Please see comment 1906827813.

[sparrc]

@raphael5178 sorry missed updating this PR, this has now been released in agent version 1.81.0

[raphael5178]

@sparrc no problem my friend, thanks :)

[19h]

This "fix" broke the ECS agent for the entire cluster when using Flatcar Container Linux running Docker 24 on all stable, beta and alpha update trains; we have since downgraded the agent to an explicit 1.80.0 version until further notice.

Can you please lmk if there's a workaround to the following issue arising when running any ECS agent version gte 1.81.0:

level=info time=2024-06-05T13:16:22Z msg="Unable to get Docker client for version 1.44: Error response from daemon: client version 1.44 is too new. Maximum supported API version is 1.43" module=sdkclientfactory.go

I can replicate this on all build trains for flatcar; I don't think the container is respecting any API-version related env vars as we tried 1) configuring them on a systemd level, 2) on a docker level, 3) on the ecs.config level; neither worked and left the cluster in an unoperable state.

Please advise asap.