Run `nixosTests` on darwin by roberth · Pull Request #193336 · NixOS/nixpkgs

roberth · 2022-09-28T13:47:21Z

Description of changes

I had to add lazyFunction in order to half-support make-test-python.nix calls which don't pass hostPkgs along.
All tests should be migrated to use the simpler runTest IMO.

Draft because this needs testing and validation, but I don't have much time to spend on this. Figured I'd dump my knowledge in the form of a draft PR (and because I've changed things #191540).

TODO

Eval on hydra against eval regressions
- in existing nixos.test.*
- future breakage in the macos tests (but only nixos.tests.simple and test-framework testing tests?)
Docs
Add tests to (TBD) nixosTests.nixos-test-runner suite (categorize tests with recurseIntoAttrs)
Consider moving more of the logic into nixos/lib/testing. The nixosTests system logic is also useful there.
Get lib.lazyFunction: init #194514 done and rebase

Things done

YorikSar

Thanks a lot for tackling this! I couldn't find all ends where hostPkgs needs to be pulled through.

I've tried running nix build -L .#nixosTests.doas with --system aarch64-darwin and --system x86_64-darwin. It worked fine with aarch64 and expectedly broke on x86_64. I'm on M1 Mac and QEMU command line currently doesn't support running non-native architecture (needs some tweaks, if that's possible). Would it be possible to warn user about this somehow?

I've also tried running boot.uefiUsb, but Nix segfaults on evaluation on my machine for some reason.

Would it be possible to add some of these tests on Hydra, so that we can be sure that this will not get broken in the future?

nixos/lib/testing/run.nix

YorikSar · 2022-10-04T19:53:58Z

nixos/lib/testing/run.nix

I don't think we need additional flag for this. hvf is always available on Darwin, I don't think there's any hardware platform left that doesn't support it.
On the other hand, it would be good to allow to run tests even in plain emulation. It will be slow, but will still allow to test locally.

I know some people virtualize macOS on apple hardware in order to make their build farm setup reproducible.
It seems unlikely that nested virtualization would be available on such VMs.

roberth · 2022-10-04T22:38:10Z

nixos/tests/all-tests.nix

      else val;
  handleTest = path: args:
-    discoverTests (import path ({ inherit system pkgs; } // args));
+    discoverTests (lazyFunction (import path) ({ inherit system pkgs hostPkgs; } // args));


This silently ignores make-test-python.nix-based runners that don't pass hostPkgs through, which has the effect of running the test on a linux remote builder.

roberth · 2022-10-04T23:08:53Z

I've removed the cross compilation changes, which I wasn't confident about. However I did leave the preliminary support for running NixOS tests using cross-compiled pkgs in place.

roberth · 2022-10-04T23:12:32Z

I'm on M1 Mac and QEMU command line currently doesn't support running non-native architecture (needs some tweaks, if that's possible). Would it be possible to warn user about this somehow?

I don't know about what can be tweaked and I have no ambition to research this (sorry). A warning could be added if we're confident that this will never be supported, but until we're confident I don't want to discourage anyone from picking up where I/we left off.

Would it be possible to add some of these tests on Hydra, so that we can be sure that this will not get broken in the future?

I would think so and this is on the todo list. 👍 EDIT: elaborated that a bit

Gabriella439 · 2022-11-11T15:16:10Z

Is it possible to use nixosTest with this branch? I tried the following:

{ inputs.nixpkgs.url =
    "github:NixOS/nixpkgs/e39a5efc4504099194032dfabdf60a0c4c78f181";

  outputs = { nixpkgs, ... }: {
    checks.aarch64-darwin.default =
      nixpkgs.legacyPackages.aarch64-darwin.nixosTest {
        name = "test";

        nodes.machine.virtualisation.host.pkgs =
          nixpkgs.legacyPackages.aarch64-darwin;

        testScript = "";
      };
  };
}

… but that fails with:

error: assertion '(stdenv).isLinux' failed

       at /nix/store/3zj9cv9pj7q2y4g0q1bfxzmv4wq2i4zj-source/pkgs/os-specific/linux/kernel/generic.nix:71:1:

           70|
           71| assert stdenv.isLinux;
             | ^
           72|
(use '--show-trace' to show detailed location information)

roberth · 2022-11-11T15:39:23Z

pkgs/top-level/all-packages.nix

+    systemArgs =
+      if stdenv.hostPlatform.isLinux && (stdenv.buildPlatform.isLinux || stdenv.buildPlatform.isDarwin)
+      then {
+        hostPkgs = pkgs.buildPackages;
+        pkgs = pkgs;
+        system = stdenv.hostPlatform.system;
+      }
+      else if stdenv.hostPlatform.isDarwin && stdenv.buildPlatform.isDarwin
+      then
+        # In the context of a normal darwin compile, repurpose nixosTests as a
+        # non-cross build. This requires a "remote" builder such as a local
+        # Linux VM, but all VM hosting happens on darwin.
+        let linuxSystem = "${stdenv.buildPlatform.parsed.cpu.name}-linux";
+        in {
+          hostPkgs = pkgs;
+          pkgs = import ../.. { inherit config overlays; system = linuxSystem; };
+          system = linuxSystem;
+        }
+      else throw "Don't know how to configure NixOS for VM host ${stdenv.buildPlatform.system} and/or VM guest ${stdenv.hostPlatform.system} built on ${stdenv.buildPlatform.system}";


This logic is useful in other places, so it should be factored or moved to a common location, such as the nixos/lib/testing/pkgs.nix module.

How about pkgsLinux (in pkgs)

roberth · 2022-11-11T15:47:31Z

Is it possible to use nixosTest with this branch?

It doesn't have a good reason not to, but the immediate problem is that doesn't let you set anything related to either pkgs or hostPkgs.

It should probably the logic I've highlighted in my previous comment in one way or another.

In your example, perhaps you might get away with setting nodes.machine.nixpkgs.pkgs to a linux one by hand?
Really it should be automated though, like nixosTests plural.

Gabriella439 · 2022-11-11T15:51:18Z

That works! The following modified version built for me:

{ inputs.nixpkgs.url =
    "github:NixOS/nixpkgs/e39a5efc4504099194032dfabdf60a0c4c78f181";

  outputs = { nixpkgs, ... }: {
    checks.aarch64-darwin.default =
      nixpkgs.legacyPackages.aarch64-darwin.nixosTest {
        name = "test";

        nodes.machine = {
          nixpkgs.pkgs = nixpkgs.legacyPackages.aarch64-linux.pkgs;

          virtualisation.host.pkgs =
            nixpkgs.legacyPackages.aarch64-darwin;
        };

        testScript = "";
      };
  };
}

roberth · 2022-11-11T15:51:22Z

Oh and nixosTest should be superseded by a new function that takes a module argument instead of behaving like callPackage. Sadly those behaviors are mutually exclusive.
I did add nixpkgsFlake.lib.nixos.runTest, which takes a module, but leaves the pkgs and hostPkgs for you to set. Not exactly the answer to your question, but perhaps a solution?

goodlyrottenapple · 2023-01-17T15:53:49Z

@Gabriella439 I tried building your example flake on an M1 but getting the following error:

nix build .#checks.aarch64-darwin.default                               
error: a 'aarch64-darwin' with features {hvf, nixos-test} is required to build '/nix/store/icqc6g8lf1f1lnl1dzipfaw5rlmp4vsx-vm-test-run-test.drv', but I am a 'aarch64-darwin' with features {benchmark, big-parallel, nixos-test}

(currently on macOS 12.4)

Is there any other setup necessary to get hvf working?

Gabriella439 · 2023-01-17T16:01:56Z

Oh yeah, I had to also add this to my nix.conf:

extra-system-features = hvf

goodlyrottenapple · 2023-01-17T16:08:36Z

Oh yeah, I had to also add this to my nix.conf:
extra-system-features = hvf

Ah thanks so much! I did try this before, but the option was being ignored when put in ~/.config/nix/nix.conf. After I moved it to /etc/nix/nix.conf, it built successfully.

roberth · 2023-01-17T19:58:46Z

Created Automatically add apple virtualization to system-features nix#7619 to mirror the solution we have for kvm on linux.

@roberth

Closes NixOS#193336 Closes NixOS#261694 Related to 108984 This pull request is a composite of the changes from NixOS#193336 and also NixOS#261694, with some changes of my own that I made. The goal here was to get the following flake to build and run on `aarch64-darwin`: ```nix { inputs.nixpkgs.url = <this branch>; outputs = { nixpkgs, ... }: { checks.aarch64-darwin.default = nixpkgs.legacyPackages.aarch64-darwin.nixosTest { name = "test"; nodes.machine = { }; testScript = ""; }; }; } ``` … and after this change it does. There's no longer a need for the user to set `nodes.*.nixpkgs.pkgs` or `nodes.*.virtualisation.host.pkgs` as the correct values are inferred from the host system. This change is spiritually much closer to the approach in NixOS#261694 than the approach in NixOS#193336. However, I still made a few changes compared to NixOS#261694: - I didn't include the change to increase `ulimit` I think this change was questionable because I feel like a script provided by Nixpkgs shouldn't be tinkering with the user's ambient `ulimit` settings. - I named the required system feature `hvf` instead of `apple-virt` I preferred the `hvf` system feature name chosen by NixOS#193336. - I didn't use the `node.pkgs` setting to set `nodes.*.nixpkgs.pkgs` That does not work, based on my testing, but setting the same thing in `nixos/lib/testing/nodes.nix` does work. - I created a `pkgsLocal` helper This is based on the feedback from @roberth here: https://github.com/NixOS/nixpkgs/pull/261694/files#r1365443955 … and is similar in spirit to `pkgsCross`.

ofborg bot added the 6.topic: darwin Running or building packages on Darwin label Sep 28, 2022

github-actions bot added the 6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS label Sep 28, 2022

roberth mentioned this pull request Sep 28, 2022

VM tests should not depend on any specific VM technology #5241

Open

ofborg bot added 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin. 10.rebuild-linux: 1-10 This PR causes between 1 and 10 packages to rebuild on Linux. labels Sep 28, 2022

YorikSar suggested changes Oct 4, 2022

View reviewed changes

roberth added 4 commits October 5, 2022 00:22

lib.lazyFunction: init

8cfb5bc

lib/systems: Clarify on which system we can't emulate

c6e0356

nixosTests: Wire up hostPkgs

dc28942

nixosTests: Add hvf to requiredSystemFeatures on darwin

e39a5ef

roberth force-pushed the nixosTests-darwin-and-cross branch from 2c210d3 to e39a5ef Compare October 4, 2022 22:33

roberth commented Oct 4, 2022

View reviewed changes

roberth mentioned this pull request Oct 4, 2022

lib.lazyFunction: init #194514

Open

13 tasks

roberth changed the title ~~Run nixosTests on darwin and cross~~ Run nixosTests on darwin Oct 4, 2022

YorikSar mentioned this pull request Oct 31, 2022

Allowing NixOS VM's to be run on macOS #108984

Closed

roberth commented Nov 11, 2022

View reviewed changes

roberth mentioned this pull request Jan 17, 2023

Automatically add apple virtualization to system-features NixOS/nix#7619

Closed

cleeyv mentioned this pull request Jul 27, 2023

CI builds for darwin ngi-nix/ngipkgs#24

Closed

luetge added a commit to luetge/nixpkgs that referenced this pull request Sep 25, 2023

Patches for running on darwin (NixOS#193336)

e199caa

roberth mentioned this pull request Oct 19, 2023

NixOS Integration Tests on macOS: Remove some obstacles (ulimit, requiredSystemFeatures) #261694

Closed

12 tasks

Gabriella439 mentioned this pull request Jan 20, 2024

macOS support for NixOS tests #282401

Merged

3 tasks

Gabriella439 closed this in #282401 Mar 2, 2024

Gabriella439 closed this in b8698cd Mar 2, 2024

Uh oh!

Comments

Conversation

roberth commented Sep 28, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of changes

Things done

Uh oh!

YorikSar left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

YorikSar Oct 4, 2022

Choose a reason for hiding this comment

Uh oh!

roberth Oct 4, 2022

Choose a reason for hiding this comment

Uh oh!

roberth Oct 4, 2022

Choose a reason for hiding this comment

Uh oh!

roberth commented Oct 4, 2022

Uh oh!

roberth commented Oct 4, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Gabriella439 commented Nov 11, 2022

Uh oh!

roberth Nov 11, 2022

Choose a reason for hiding this comment

Uh oh!

roberth Jun 14, 2023

Choose a reason for hiding this comment

Uh oh!

roberth commented Nov 11, 2022

Uh oh!

Gabriella439 commented Nov 11, 2022

Uh oh!

roberth commented Nov 11, 2022

Uh oh!

goodlyrottenapple commented Jan 17, 2023

Uh oh!

Gabriella439 commented Jan 17, 2023

Uh oh!

goodlyrottenapple commented Jan 17, 2023

Uh oh!

roberth commented Jan 17, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

roberth commented Sep 28, 2022 •

edited

Loading

roberth commented Oct 4, 2022 •

edited

Loading