networking.networkd.privacy test broken by systemd 247 upgrade

[andersk]

Describe the bug

The channel-blocking networking.networkd.privacy test is failing. The test hangs for 15 minutes and times out. I bisected this failure to commit ef050bc “systemd: 246.6 -> 247-rc2” (#102355), then bisected systemd and found systemd/systemd@5e0534f (systemd/systemd#17240) to be the first bad commit.

To Reproduce
nix build -Lf ~/nixpkgs nixosTests.networking.networkd.privacy

Notify maintainers
@Mic92 @andir @edolstra @flokli @kloenk

Metadata

• system: "x86_64-linux"
• host os: Linux 5.4.81, NixOS, 21.03.git.b7c48f3ea5c (Okapi)
• multi-user?: yes
• sandbox: yes
• version: nix-env (Nix) 2.3.9
• channels(root): "nixos-21.03pre257780.e9158eca70a"
• nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixos

Maintainer information:

# a list of nixpkgs attributes affected by the problem
attribute: [systemd]
# a list of nixos modules affected by the problem
module: []

[kloenk]

This could be fixed with #106834?

(Currently can't test it)

[andersk]

I tested with current staging (commit 39b7603), which includes #106834, but it still fails.

[matthewmazzanti]

Looks like the test is failing on this section:
https://github.com/NixOS/nixpkgs/blob/master/nixos/tests/networking.nix#L591

Interestingly the test its running is the only one with a '!' negation. Removing the negation causes the test to not hang, however I think that also changes the semantics of the test.

[ghost]

The archlinux folks have cherry-picked some commits, maybe that helps? https://github.com/archlinux/svntogit-packages/commits/packages/systemd/trunk

To be honest I'm kind of glad that this means systemd 247 hasn't made it to the nixos-unstable channel yet, because that contained some other really annoying breakages (which are fixed in 247.1), oh my...

[rnhmjoj]

Interestingly the test its running is the only one with a '!' negation. Removing the negation causes the test to not hang, however I think that also changes the semantics of the test.

I wrote the privacy test so let me clarify: this subtest is checking whether the default address for outbound connections is temporary (RFC 4941) on the client with tempAddress = "default". It does so by looking for a permanent EUI-64 address (based on the interface MAC address) and negating the result of grep.

On the other client, tempAddress = "enabled" instead, so temporary addresses are assigned but not used by default.

Since the scripted test is passing, it looks like networkd made a breaking change or introduced a bug regarding temporary addresses. A breaking change in ip-route or the kernel would have affected both tests.

[andersk]

I spent the day doing a systemd bisection and found systemd/systemd@5e0534f (systemd/systemd#17240) to be the first bad commit.

[mohe2015]

Offtopic but are there any docs about bisecting a specific package in nix (not nixpkgs itself) or how did you do it?

[andersk]

@mohe2015 I did it by checking out nixpkgs at the parent commit of ef050bc, and manually making changes like this:

--- a/pkgs/os-specific/linux/systemd/default.nix
+++ b/pkgs/os-specific/linux/systemd/default.nix
@@ -110,7 +110,7 @@ assert withCryptsetup ->
 let
   wantCurl = withRemote || withImportd;
 
-  version = "246.6";
+  version = "246-1196-g5e0534f1c1";
 in
 stdenv.mkDerivation {
   inherit version pname;
@@ -121,7 +121,7 @@ stdenv.mkDerivation {
     owner = "systemd";
     repo = "systemd-stable";
     rev = "v${version}";
-    sha256 = "1yhj2jlighqqpw1xk9q52f3pncjn47ipi224k35d6syb94q2b988";
+    sha256 = "1wpqw9cwn54q12ivxkq3caw19pdh71c9db4sgkhzkq54s9vyyp8d";
   };
 
   # If these need to be regenerated, `git am path/to/00*.patch` them into a

while running git bisect in a separate checkout of systemd that served no purpose except to tell me which commits to try next. The version is formatted using git describe, which works because this derivation passes rev = "v${version}" to fetchFromGitHub. Other derivations may require different strategies.

(The reason it took a day is that changing systemd forces hundreds of packages used in the test’s closure to be rebuilt in each iteration, including spidermonkey(?!!).)

[flokli]

@andersk Thanks for doing the bisect, much appreciated! Did you file an upstream issue about this, linking to here?

[flokli]

(The reason it took a day is that changing systemd forces hundreds of packages used in the test’s closure to be rebuilt in each iteration, including spidermonkey(?!!).)

Here you go: 😆

[arianvp]

I swear we had an issue tracking removing libfido2 from opensshMinimal so that not everything pulls in systemd but I can not find it. Anyhow; we should still do that =)

[flokli]

I swear we had an issue tracking removing libfido2 from opensshMinimal so that not everything pulls in systemd but I can not find it. Anyhow; we should still do that =)

There's no opensshMinimal, but feel free to open an issue about it :-)

[FRidh]

Friday evening I intend to revert the systemd upgrade unless there is a tested fix before then.

[flokli]

Fix in #107061

[flokli]

The "Fix" does work it around, by setting IPv6PrivacyExtensions=true/prefer-public/false, in addition to setting the values in sysctl.

This is still an upstream issue, I opened systemd/systemd#18003 there.

[flokli]

I sent a fix for IPv6PrivacyExtensions=kernel upstream. However, I think NixOS should keep setting IPv6PrivacyExtensions=true/false/prefer-public as it does currently - it's closer to the rest of the network config, and more visible than a sysctl setting set deep in the module system.

[rnhmjoj]

Uhm.. I'd prefer for it to be reverted once fixed: it's a small addition but it's easier to overlook in case tempAddress would have to be updated, say in future kernel update.