[ECS] RPM installation inside amazonlinux containers running under Docker is extremely slow

[samuelkarp]

Platform I'm building on:
aws-ecs-1

What I expected to happen:
yum install -y tmux to complete in a reasonable timeframe

What actually happened:
Stalled on installing libevent-2.0.21-4.amzn2.0.3.x86_64

How to reproduce the problem:
Run a Docker container and try installing a package.

Past anecdata suggests that Amazon Linux's RPM is very slow when the nofile ulimit is high as it tries to iteratively close all possible file descriptors. I see a very high nofile limit:

bash-5.0# docker run -it --rm amazonlinux
bash-4.2# ulimit -a
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 3866
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1073741816
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) unlimited
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

This is likely inherited from here:

bottlerocket/packages/docker-engine/docker.service

Line 20 in f06359d

LimitNOFILE=infinity

[iliana]

Ugh, right... I seem to remember wanting to backport it in Amazon Linux's RPM, but this affects any distro with a sufficiently old RPM, too.

I'm not sure what the right thing for us to do here.

[samuelkarp]

At least for now, I think lowering nofile is probably a reasonable approach. Amazon Linux/Amazon Linux 2 both have non-infinite nofile for Docker.

[samuelkarp]

Fixed in v1.0.2.