ECS control plane not compatible with ECS-A and Docker v26 requirements for API version

[rClark-qubeyond]

Spawning this out to a new issue, rather than a comment on #4090

I think there are multiple problems here:
agent v1.80 had the incompatibility with Docker v25 when they forced the DOCKER_MIN_API_VERSION to be used, which was fixed in v1.81.
Now, with docker-ce v26, this setting no longer is accepted and the previous guidance of DOCKER_MIN_API_VERSION=1.17 pre ecs-agent 1.81.0 from #4074 (comment) prevents docker-ce v26 from starting all:

-- A start job for unit docker.service has begun execution.
--
-- The job identifier is 1779.
Mar 25 17:23:16 ubuntu20 dockerd[39706]: invalid DOCKER_MIN_API_VERSION: minimum supported API version is 1.24: 1.17
Mar 25 17:23:16 ubuntu20 systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
-- Subject: Unit process exited
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- An ExecStart= process belonging to unit docker.service has exited.
--
-- The process' exit code is 'exited' and its exit status is 1.
Mar 25 17:23:16 ubuntu20 systemd[1]: docker.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- The unit docker.service has entered the 'failed' state with result 'exit-code'.
Mar 25 17:23:16 ubuntu20 systemd[1]: Failed to start Docker Application Container Engine.
-- Subject: A start job for unit docker.service has failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A start job for unit docker.service has finished with a failure.
--
-- The job identifier is 1779 and the job result is failed.

This is a problem, because ECS itself is still requiring from task defs:

    "requiresAttributes": [
        {
            "name": "com.amazonaws.ecs.capability.ecr-auth"
        },
        {
            "name": "com.amazonaws.ecs.capability.docker-remote-api.1.21"
        },
        {
            "name": "ecs.capability.docker-plugin.local"
        },
        {
            "name": "ecs.capability.container-ordering"
        },
        {
            "name": "com.amazonaws.ecs.capability.docker-remote-api.1.25"
        },
        {
            "name": "com.amazonaws.ecs.capability.docker-remote-api.1.18"
        }
    ],

So it's just not possible to upgrade docker-ce at all past 25.0.5 and maintain compatibility with ECS

Originally posted by @rClark-qubeyond in #4090 (comment)

[danehlim]

Hello, could you clarify more details about your setup such as which version of ECS Agent you are using?

I tried to reproduce the issue but found that I was able to use ECS with docker-ce version 26.0.0.

Further Details:

I spun up an Ubuntu 22 instance for ECS-Anywhere using ECS Agent 1.82.1 and ensured docker-ce version 26.0.0 was installed.

$ apt list --installed | grep ecs

amazon-ecs-init/now 1.82.1-1 amd64 [installed,local]

$ docker --version
Docker version 26.0.0, build 2ae903e

$ apt list --installed | grep docker-ce/

docker-ce/jammy,now 5:26.0.0-1~ubuntu.22.04~jammy amd64 [installed]

I registered the instance with an empty ECS cluster, then started a task. The task transitioned to RUNNING and I ran the following on the instance to confirm ECS Agent and the container associated with the task were running as expected:

$ sudo docker ps
CONTAINER ID   IMAGE                            COMMAND                  CREATED          STATUS                    PORTS                               NAMES
614e5dfcd311   nginx                            "/docker-entrypoint.…"   2 minutes ago    Up 2 minutes              0.0.0.0:80->80/tcp, :::80->80/tcp   ecs-test-web-application-2-web2-e2a59de3e1fbbf866d00
e33a2b2425c8   amazon/amazon-ecs-agent:latest   "/agent"                 11 minutes ago   Up 11 minutes (healthy)                                       ecs-agent

I then stopped the task and ran the following on the instance to confirm ECS Agent was still running and the container associated with the task was no longer running as expected:

$ sudo docker ps
CONTAINER ID   IMAGE                            COMMAND    CREATED          STATUS                    PORTS     NAMES
e33a2b2425c8   amazon/amazon-ecs-agent:latest   "/agent"   11 minutes ago   Up 11 minutes (healthy)             ecs-agent

[rClark-qubeyond]

The issue is not Docker v26 or Agent 1.82.1, it's running both of those then assigning an ECS task from the control plane to the instance. These both run, but they won't accept any tasks configured in ECS.

The AWS ECS control plane for task definitions still assigns very old API versions as requiresAttributes, hence needing to stay on 25.0.5 and forcing the DOCKER_MIN_API_VERSION=1.17 but doing this prevents the docker.service from starting and hence the agent.

Literally a 3-body problem.

[danehlim]

The issue is not Docker v26 or Agent 1.82.1, it's running both of those then assigning an ECS task from the control plane to the instance. These both run, but they won't accept any tasks configured in ECS.

The AWS ECS control plane for task definitions still assigns very old API versions as requiresAttributes, hence needing to stay on 25.0.5 and forcing the DOCKER_MIN_API_VERSION=1.17 but doing this prevents the docker.service from starting and hence the agent.

Literally a 3-body problem.

Sorry, I'm not sure I am following here. I was able to assign an ECS task from control plane to the instance under the simple setup specified in my previous comment. Perhaps is there some special additional set up that you are doing when you observe this issue?

[danehlim]

This is a problem, because ECS itself is still requiring from task defs:

    "requiresAttributes": [
        {
            "name": "com.amazonaws.ecs.capability.ecr-auth"
        },
        {
            "name": "com.amazonaws.ecs.capability.docker-remote-api.1.21"
        },
        {
            "name": "ecs.capability.docker-plugin.local"
        },
        {
            "name": "ecs.capability.container-ordering"
        },
        {
            "name": "com.amazonaws.ecs.capability.docker-remote-api.1.25"
        },
        {
            "name": "com.amazonaws.ecs.capability.docker-remote-api.1.18"
        }
    ],

Could you clarify where exactly you are seeing the requiresAttributes for "ECS itself is still requiring from task defs"?

[rClark-qubeyond]

these requiresAttibutes are viewable within AWS CLI or within the console /ecs/v2/task-definitions/<task-def-name>/<task-def-rev>/json

There is nowhere that we are setting these, but was told that these are assigned automatically from ECS logic based on the requirements of the task definition. I've attempted creating new revisions but these remain. I have not attempted creating the task def again from scratch to see if these dependencies disappear in later newly created versions, since our rev 1 of this task was about 2 years old.

---

UPDATE: I may have answered my own question.... I'm guessing that ECS inspects the "DockerVersion" under which the image was created as it is assigned to the task definition, and may be holding older docker-remote-api versions based on image inspection.

[hozkaya2000]

Hi, are you still facing an issue regarding this? It seems like you have diagnosed the problem -- looking to confirm.