Azure Linux Helix hosts should match containerd Limit configuration

[premun]

As per the discussion here: https://teams.microsoft.com/l/message/19:afba3d1545dd45d7b79f34c1821f6055@thread.skype/1748255195998?tenantId=72f988bf-86f1-41af-91ab-2d7cd011db47&groupId=4d73664c-9f2f-450d-82a5-c2f02756606d&parentMessageId=1748255195998&teamName=.NET%20Core%20Eng%20Services%20Partners&channelName=First%20Responders&createdTime=1748255195998

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Increase NOFILE ulimit in containers running on Azure Linux 3 hosts

[richlander]

Something is off. I validated that the Azure Linux VM image have unlimited ulimit set, via DTL. I didn't (cannot) do that with the actual deployed Helix VMs, however, internal mail suggested that this had been done. I worked directly with dnceng to make that happen.

Number 1 priority is forward progress so I'm happy with any change towards that end. However, we need to figure out what's up.

Context:

• Azure Linux 3 VMs need to be updated #5602 (comment)
• Update Linux versions in Libraries pipeline runtime#115756 (comment)

From internal mail ...

From: David ...
Sent: Wednesday, May 14, 2025 10:43 AM
To: .NET Core Engineering Partners
Subject: Heads-up: Engineering Services rollouts 2025-05-14

Feature changes/additions:
...
increased the nofile limit on azurelinux.3.amd64[*] queues. (#5602)

[premun]

@richlander this is about containers running on the Azure Linux hosts. The host itself has ulimit set well.
However, containers ran on these hosts have ulimit of 1024.
We're currently looking into setting the docker daemon properly.

I didn't (cannot) do that with the actual deployed Helix VMs

FYI there is a very simple way to validate Helix images:

1. Clone https://dev.azure.com/dnceng/internal/_git/dotnet-helix-machines
2. .\osob-cli.cmd exec -q azurelinux.3.amd64.open -P -c richlander 'ulimit -a'
   (It fails to show the log in console but it does produce a Helix job you can open in browser and find the console log there. Alternatively, try .\osob-cli.cmd exec -q azurelinux.3.amd64.open -P -c richlander --work-item-count 2 'ulimit -a')

[akoeplinger]

Another option is to use the helix-repro-vms Azure DTL to create a VM with the same image: https://dev.azure.com/dnceng/internal/_wiki/wikis/DNCEng%20Services%20Wiki/915/Investigating-Helix-VM-images

[premun]

Yeah, I am doing just that

[premun]

@jakobbotsch @akoeplinger I was able to increase the ulimit inside of a container:

I had to add the following to daemon.json:

"default-ulimits": {"nofile": {"Name": "nofile", "Soft": 16384, "Hard": 524288}}

What would you want me to set the values to?

[akoeplinger]

@premun can you also try whether adding this into containerd.service fixes it as well? it is what Ubuntu had

LimitNPROC=infinity
LimitCORE=infinity
LimitNOFILE=infinity

[premun]

@akoeplinger it worked. open files is now at 1073741816. What's the right way to set this though? Do I need to manipulate the config manually or is there a better way?

[premun]

Oh, I see, I need to pipe stuff into sudo systemctl edit containerd or write into /etc/systemd/system/containerd.service.d/override.conf

[akoeplinger]

yeah adding an override.conf sounds good. just to double-check, is it also 1073741816 when running a container on the ubuntu VM host?

[premun]

scheduling priority             (-e) 0
file size (blocks)              (-f) unlimited
pending signals                 (-i) 31749
max locked memory (kb)          (-l) 8192
max memory size (kb)            (-m) unlimited
open files                      (-n) 1048576
POSIX message queues (bytes)    (-q) 819200
real-time priority              (-r) 0
stack size (kb)                 (-s) 8192
cpu time (seconds)              (-t) unlimited
max user processes              (-u) unlimited
virtual memory (kb)             (-v) unlimited
file locks                      (-x) unlimited

[premun]

PR - https://dev.azure.com/dnceng/internal/_git/dotnet-helix-machines/pullrequest/50393

[richlander]

Two things:

• Can you explain how AL and Ubuntu are different here? Did we already configure Ubuntu correctly or it has a different configuration? Is this something that doesn't affect Ubuntu 22.04 but does with 24.04 (which we don't yet use as a host)? That's what Remove LimitNOFILE from containerd.service containerd/containerd#8924 suggests, particularly the timing.
• Can we document all the ulimit related configurations? We will surely hit this again.