Incompatible with Docker v25 (API 1.44)

[shanet]

Summary

The ECS Agent is not compatible with the newest version of Docker.

Description

With Docker version 25, the minimum API version is now 1.24. The ECS Agent will fail to start with the following error on this Docker version:

level=error time=2024-01-19T19:43:31Z msg="Failed to load pause container: container load: failed to load container image: /images/amazon-ecs-pause.tar : Error response from daemon: client version 1.21 is too old. Minimum supported API version is 1.24, please upgrade your client to a newer version" module=agent.go

This is shown with the docker version output:

Server: Docker Engine - Community
 Engine:
  Version:          25.0.0
  API version:      1.44 (minimum version 1.24)

Our temporary fix was to downgrade Docker back to version 24.0.7 which has a minimum API version of 1.12. This resolves the issue and allows the ECS Agent to start for the time being.

Expected Behavior

The ECS Agent should start with Docker version 25.

Observed Behavior

The ECS agent crashes with the error above.

Environment Details

• Ubuntu 22.04
• ECS Agent v1.80.0
• Docker version 25 (API version 1.44)

[erankitcs]

I am having a similar problem within our stack.

Workaround Step for Ubuntu 20.04-

1. apt remove -y docker-ce docker-ce-cli containerd.io
2. VERSION_STRING=5:24.0.7-1ubuntu.20.04focal
3. apt install -y docker-ce=$VERSION_STRING docker-ce-cli=$VERSION_STRING containerd.io
4. Rerun ecs-anywhere-install command
   curl --proto "https" -o "/tmp/ecs-anywhere-install.sh" "https://amazon-ecs-agent.s3.amazonaws.com/ecs-anywhere-install-latest.sh" && bash /tmp/ecs-anywhere-install.sh --region "" --cluster "<CLUSTER_NAME>" --activation-id "<ID_HERE>" --activation-code "<CODE_HERE>"

Hopefully, the dependency issue will be fixed soon.

[shanet]

Looks like there's an open PR for this here #4075 so Docker 25 support should land soon.

[kyluca]

Another workaround that resolved the issue for us in the meantime was to follow the advice in Docker's deprecation notice from their new v25.0.0 release: https://docs.docker.com/engine/deprecated/#deprecate-legacy-api-versions

An environment variable (DOCKER_MIN_API_VERSION) is introduced that allows re-enabling older API versions in the daemon.

The summary is to add an environment variable to the Docker daemon environment lowering the minimum API requirement, to allow the ECS agent to continue using the older API until this issue is fixed and released. As they explained, this is only meant to be a temporary workaround for clients that we can't update ourselves such as this ECS agent.

We do not recommend depending on the DOCKER_MIN_API_VERSION environment variable other than for exceptional cases where it's not possible to update old clients, and those clients must be supported.

However, this was impacting our production stack and others I'm sure. So assuming you're using systemd:

1. Create the directory /etc/systemd/system/docker.service.d

   sudo mkdir -p /etc/systemd/system/docker.service.d
   

2. Create a file e.g. /etc/systemd/system/docker.service.d/min_api_version.conf with the contents

   [Service]
   Environment="DOCKER_MIN_API_VERSION=1.17"
   

3. Apply the changes

   sudo systemctl daemon-reload
   sudo systemctl restart docker
   

4. Verify the change is applied

   sudo systemctl show --property=Environment docker
   

We use SaltStack so applying that fix to our affected servers was relatively quick and easy, without needing to compromise and downgrade Docker itself.

[chienhanlin]

Close this issue as the fix is released with ECS Agent version 1.81.0. Thanks.
Release PR: #4083