allow setting cgroup memory limits via both nix.conf and pre-build hooks

[cleverca22]

allows setting resource limits with --option use-cgroups true --option memory-max $((1024*1024*1024*5))
allows a pre-build hook to set limits:

#!/bin/sh

if [[ -d $NIX_CGROUP ]]; then
  echo $((1024*1024*1024*1)) > $NIX_CGROUP/memory.max
fi

such a pre-build hook could fork out a daemon, that watches memory.events and dynamically raises memory.high while co-operating with a resource manager

note, currently requires that nix-daemon be ran with an extra dir in its cgroup path, such as /system.slice/nix-daemon.service/actual-daemon

[SuperSandro2000]

I am missing some documentation here in what relation to each other you should set memory-high and memory-max. I would assume something like memory-high 3GB and memory-max 4GB?

[cleverca22]

depends on how much a specific derivation needs, the goal is to keep track of how much memory each derivation is using, and not schedule too many on one machine
then use cgroups to enforce that it stays within the reserved amounts

[thufschmitt]

Could it make sense to have a more general cgroup-config parameter accepting a list of key-value pairs?

[edolstra]

This is very confusing naming. Doing dirOf means it isn't our cgroup, but the parent's. And it's not clear that we want to be putting stuff in the parent cgroup, since it causes nix to break out of whatever resource controls are in place for its own cgroup. It especially wouldn't be nice if Nix builds escape from system.slice/nix-daemon.service.

[cleverca22]

i had modified things so the daemon runs in system.slice/nix-daemon.service/actual-daemon

the problem is that you cant certain things on system.slice/nix-daemon.service if it contains any process, everything must be in a child c-group

i'm not sure what the best solution to that is, and this currently needs an extra script around nix-daemon to move it to a child cgroup

[edolstra]

Indentation is off here.

[edolstra]

This needs proper capitalization/spelling/Markdown formatting.

[edolstra]

This will fail if the cgroup does not have memory in cgroup.controllers, or if it's not a leaf cgroup or one that only has pids in its children (see the "no internal processes" rule in cgroups(7)). In particular, this may cause non-root chroot builds to fail (i.e. --store /path/to/my/nix as a non-root user) since we can't count on memory being enabled.

It's not clear what the best solution is, but some options:

• When nix-daemon starts, it can create a sub-cgroup (e.g. system.slice/nix-daemon.service/manager) for itself and move into it, and create sub-cgroups for the builds in the original cgroup (e.g. system.slice/nix-daemon.service/<build>). That way the original cgroup doesn't violate the "no internal processes" rule.
• That doesn't work for the non-root case, since the original cgroup will likely have other processes in it (e.g. user.slice/user-1000.slice/user@1000.service/app.slice/app-org.kde.konsole-7edbbf6bbf864ef9a259f86f31b0fd90.scope). So we can't easily convert it into a cgroup that doesn't violate the "no internal processes" rule. I guess we could search upwards for the nearest parent that has the memory controller enabled and has no internal processes (e.g. /sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service/app.slice in this example) but that's a bit naughty...
• Allow the parent cgroup for builds to be specified in nix.conf.

Whatever the solution, Nix shouldn't fail if writing to cgroup.subtree_control fails.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2022-12-09-nix-team-meeting-minutes-15/23951/1