The CPU parameter in the task is not behaving as a hard limit and is not setting either cpu-quota or cpu

[FelipeGXavier]

Description

CPU limits in Amazon ECS can be configured at two levels: task level and container level. The task-level setting is intended to enforce a hard limit and is implemented using --cpu-quota, as indicated in this resolved issue GitHub Issue #1093.

However, when deploying a container and only setting the CPU limit at the task level, it does not seem to impact any CPU hard limit options when inspecting the container using docker inspect.

For example, the task definition specifies a CPU limit at the container level as 0 and at the task level as 262, supposedly representing a hard limit:

{
    "containerDefinitions": [
        {
            "name": "nginx",
            "image": "nginx:stable-alpine3.17-slim",
            "cpu": 0,
            "memory": 524,
            "memoryReservation": 524,
            "portMappings": [
                {
                    "name": "nginx-80-tcp",
                    "containerPort": 80,
                    "hostPort": 0,
                    "protocol": "tcp",
                    "appProtocol": "http"
                }
            ],
            "essential": true
        }
    ],
    "status": "ACTIVE",
    "compatibilities": [
        "EC2"
    ],
    "requiresCompatibilities": [
        "EC2"
    ],
    "cpu": "262",
    "memory": "524",
    "runtimePlatform": {
        "cpuArchitecture": "X86_64",
        "operatingSystemFamily": "LINUX"
    }
}

Expected Behavior

When running docker inspect <container_id> | grep -i cpu expect to change NanoCpus, CpusetCpus or CpuQuota to implement a hard limit.

Observed Behavior

"CpuShares": 2,
"NanoCpus": 0,
"CpuPeriod": 0,
"CpuQuota": 0,
"CpuRealtimePeriod": 0,
"CpuRealtimeRuntime": 0,
"CpusetCpus": "",
"CpusetMems": "",
"CpuCount": 0,
"CpuPercent": 0

Environment Details

Amazon ECS Agent - v1.70.0
Docker version 20.10.17, build 100c701

[3guboff]

Hi @FelipeGXavier, also if you try to set the CPU limit "cpu": 262 on the docker (container) level Amazon ECS agent will set "CpuShares": 262 instead "CpuShares": 2 (based on docker inspect) which is ridiculous as first look.

There is no hard limit based on docker documentation.

I don't understand the logic of how CPU limitation is working on ECS level, mb we need to try/stay/switch with cgroup v1 like other people do? Can someone explain it?

Environment Details

$ mount | grep group
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,seclabel,nsdelegate,memory_recursiveprot)

OS: Amazon Linux 2023.3.20240304
Amazon ECS Agent: v1.82.0 (bc3cb997)
Docker: Version: 20.10.25, API version: 1.41 (client and server)

[chienhanlin]

Hello @FelipeGXavier, @3guboff
Thanks for reaching out to ECS. Please find my response as follows.

• The "Observed Behavior" on this post is expected for both cgroup v1 and v2.
• cgroup v2 is used by default on Fedora (since 31), Debian GNU/Linux (since 11), Ubuntu (since 21.10) and Amazon Linux 2023 and ECS-optimized AL2023 AMI.
• For cgroup v1, ECS Agent creates a root cgroup /ecs at start-up. When an ECS task is placed on the container instance, ECS Agent creates a sub-cgroup under /ecs named with $task_id to apply task-level resource limits.
• When creating a docker container, the /ecs/$task_id cgroup will be passed to Docker as a cgroup parent. Docker further creates one sub-cgroup for each container using $container_id under the parent cgroup /ecs/$task_id parent cgroup, and apply container-level resource limits. As limits placed on a cgroup at a higher level in the hierarchy cannot be exceeded by descendant cgroups, each container will be covered by the hard limit set on the parent cgroup /ecs/$task_id.
• To review resources limits, users can go to cgroup parent of the container: /sys/fs/cgroup/cpu/ecs/$task_id for cgroup v1, and /sys/fs/cgroup/ecstasks.slice/ecstasks-$task_id.slice for cgroup v2.

Example:

$ docker inspect <container_id>
[
    {
            "CpuShares": 2,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "/ecs/bf1ddbfbbc8f45c9b1ac4f779368741e",
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Ulimits": [
                {
                    "Name": "nofile",
                    "Hard": 65536,
                    "Soft": 32768
                }
            ],
            "CpuCount": 0,
            "CpuPercent": 0,
            ...
    }
]

[root@ip-xxx cpu]# pwd
/sys/fs/cgroup/cpu
[root@ip-1xxx cpu]# ls
cgroup.clone_children  cpu.cfs_period_us  cpu.rt_runtime_us  cpuacct.stat       cpuacct.usage_percpu       cpuacct.usage_sys   ecs                system.slice
cgroup.procs           cpu.cfs_quota_us   cpu.shares         cpuacct.usage      cpuacct.usage_percpu_sys   cpuacct.usage_user  notify_on_release  tasks
cgroup.sane_behavior   cpu.rt_period_us   cpu.stat           cpuacct.usage_all  cpuacct.usage_percpu_user  docker              release_agent      user.slice

[root@ip-xxx ecs]# pwd
/sys/fs/cgroup/cpu/ecs
[root@ip-xxx ecs]# ls
${task_id}  cpu.cfs_period_us  cpu.rt_runtime_us  cpuacct.stat       cpuacct.usage_percpu       cpuacct.usage_sys   tasks
cgroup.clone_children             cpu.cfs_quota_us   cpu.shares         cpuacct.usage      cpuacct.usage_percpu_sys   cpuacct.usage_user
cgroup.procs                      cpu.rt_period_us   cpu.stat           cpuacct.usage_all  cpuacct.usage_percpu_user  notify_on_release

[root@ip-xxx ${task_id}]# pwd
/sys/fs/cgroup/cpu/ecs/${task_id}
[root@ip-xxx ${task_id}]# ls
${container_id}  cpu.cfs_period_us  cpu.rt_runtime_us  cpuacct.stat       cpuacct.usage_percpu       cpuacct.usage_sys   tasks
cgroup.clone_children                                             cpu.cfs_quota_us   cpu.shares         cpuacct.usage      cpuacct.usage_percpu_sys   cpuacct.usage_user
cgroup.procs                                                      cpu.rt_period_us   cpu.stat           cpuacct.usage_all  cpuacct.usage_percpu_user  notify_on_release

• CPU set on the container level is the number of cpu units that ECS Agent reserves for the container. On Linux, this parameter maps to CpuShares. In addition, null, zero, and CPU values of one are passed to Docker as 2 CPU shares since ECS Agent versions >= 1.2.0.

Reference:

1. Support Unified Cgroups (cgroups v2) #3127
2. https://docs.docker.com/config/containers/runmetrics/