nixos/networkd: wait for udev to settle before starting networkd

[xeji]

Motivation for this change

Fix #39069: Avoid a race condition between udevd renaming and networkd configuring interfaces when using networkd with predictable interface names.

Please backport to 18.03 because a resulting non-deterministic test failure has frequently delayed the nixos-18.03-small channel.

This is an interim fix to get rid of the race condition and resulting test failures. It does not break existing configs that use the old ethX interface names for stage 1 (initrd) networking, so it should be safe to backport to 18.03.

(#39329 proposes a better long-term solution: rename interfaces in stage 1 already, so interface names are consistently the same in initrd and fully booted system. But that change will break initrd networking in some existing configs, so we should make it carefully.)

Things done

• Tested using sandboxing (nix.useSandbox on NixOS, or option build-use-sandbox in nix.conf on non-NixOS)
• Built on platform(s)
  • NixOS
  • macOS
  • other Linux distributions
• Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
• Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
• Tested execution of all binary files (usually in ./result/bin/)
• Fits CONTRIBUTING.md.

---

[Mic92]

Upstream systemd-networkd now renames network interfaces itself, which seems like a good idea, no?
That would prevent race conditions, but require us to have different udev rules, when networkd is enabled vs. disabled.

[Mic92]

We would need udev rules from here:

$ find $buildInputs | grep 80-net-setup-link.rules
/nix/store/64xpvhqlvvl9vvvvbih0b767lfpbg74y-systemd-238/lib/udev/rules.d/80-net-setup-link.rules

• I made this in pull request: nixos/udev: fix outdated udev rules for network devices #33617

and also this pull request, which was reverted: #29768

Back then I made the mistake merging/reverting both features independently.
However the combination of both should make renaming interfaces to work properly

[xeji]

Upstream systemd-networkd now renames network interfaces itself, which seems like a good idea, no?

it's still udev, not networkd. The .link files control the net_setup_link builtin of udev, see systemd.link(5).

That would prevent race conditions,

Unfortunately no. The race condition occurs because nixos brings up interfaces in stage 1 without renaming them, we only add the renaming rule in stage 2. networkd just tries to configure all interfaces that it finds, hence a race condition with udev trying to rename them in stage 2.

The best solution is to do renaming when the interfaces are first created in stage 1, see #39329.

but require us to have different udev rules, when networkd is enabled vs. disabled.

Actually, no if we do the renaming in stage 1.

Back then I made the mistake merging/reverting both features independently. However the combination of both should make renaming interfaces to work properly

Thanks for the references, I had not seen your previous PRs. Yes, switching back to upstream rules and .link files is a good idea, we should integrate this into #39329.

There's a small drawback: the builtin mechanism is controlled by the net.ifnames kernel boot parameter, so changing predictableInterfaceNames will require a reboot. I don't mind at all but there was a discussion about this a long time ago, f756369 and 1def5ba.

[Mic92]

Personally I have:

systemd.services.systemd-udev-settle.serviceConfig.ExecStart = ["" "${pkgs.coreutils}/bin/true"];

in my configuration because systemd-udev-settle timeouts on my machine sometimes.
That's why I would be glad if we could get rid of that service sometimes.

[xeji]

I agree this is not ideal but the best we can do in stage 2 to prevent the race condition. We can revert this as soon as we do the renaming in stage 1.

[xeji]

Instead of using systemd-udev-settle, we could add
serviceConfig.ExecStartPre="${systemd.package}/bin/udevadm settle -t 10";
directly to systemd-networkd to use a shorter timeout than the default 120s. I guess 10s will be enough to get interface renaming done... ( on a real system, but on hydra you never know).

[oxij]

LGTM