ZTS: init zfs test suite nixos tests (WIP)

[Mindavi]

Motivation for this change

#108649

Using the official test suite is nice. Currently some random tests are enabled, the hardest part of this whole story seems to be: which tests should be enabled?

Note that running the 'sanity' tests already takes over 40 minutes, which is unacceptable. @adisbladis and me had a quick chat over on IRC (https://logs.nix.samueldr.com/nixos/2021-01-10, search for Mindavi and adisbladis), and we determined that ~10 mins is a good amount of time to shoot for (this is quite arbitrary, and based on the fact that ZFS is a fundamental part of the system).

There are some things I'd like feedback on:

• Do we want to combine the zfs.nix and this one (zfs-tests.nix currently)?
• Is there a way to conditionally not remove the test suite files? I've commented out the line and tried some things with a variable as function argument, but couldn't figure out how to combine override and overrideAttrs (they're both needed).
• Is it acceptable that ZFS is rebuilt? This is currently done because the build scripts are changed (due to changes required to the test scripts).

All the currently enabled tests are passing, but (as can be seen in the test script) some of the tests fail. I'm not sure why, and the failing tests may be contributing to the long sanity test time as well.

This one in particular seems to be a culprit:

file_wait_event exceeded 120.289 seconds
ERROR: file_wait_event /var/tmp/zed/zed.debug.log sysevent.fs.zfs.scrub_start exited 1

It can be seen multiple times in a failure log.

I think the first step for this should be that we decide on where to put this and how to handle the ZFS conditional building / inclusion of the test suite. After this is merged, it's possible to look into what tests are best to run. This set will at least do something in the good direction. If you have suggestions, please let me know. I don't want to be bikeshedding about which ones to run exactly now though. I'll trust any good advice on this, as long as the runtime stays acceptable.

Things done

• Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
• Built on platform(s)
  • NixOS
  • macOS
  • other Linux distributions
• Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
• Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
• Tested execution of all binary files (usually in ./result/bin/)
• Determined the impact on package closure size (by running nix path-info -S before and after)
• Ensured that relevant documentation is up to date
• Fits CONTRIBUTING.md.

@adisbladis
@Atemu

Attached is a test log of some of the failing tests, with the data of the logfiles that the ZTS generates included.

zfs-tests-failing-tests-dump.txt

[hmenke]

It looks like there is a test set linux-fast upstream. Maybe that would be suitable?
https://github.com/openzfs/zfs/blob/f4f50a70488ebad3fa96c2fe4142924215733113/scripts/zfs-tests.sh#L338-L339

[Mindavi]

It does mention a linux-fast test, but it doesn't exist. A linux test does exist though, but I've started it an it has been running for 1830 seconds already now -- so that's too long.

[stale]

I marked this as stale due to inactivity. → More info

[Atemu]

They're also quite large, multiple dozen MiB.

[Atemu]

Is there no better way to provide these dependencies? How do upstream testers declare these binaries? It'd probably be best to just get them from $PATH which is hermetic anyways.

[Mindavi]

Hmm, the PATH is being modified when these tests are run, which is slighly hocus-pocus to me.

[Mindavi]

@Atemu do you have a tip on how to run those tests in parallel on my machine? Is there some way to run them all instead of having to nix-build -A <testname> for each of them?

[Atemu]

Usually, you can build a set of drvs with nix-build but the tests evaluate to lambdas which nix-build is somehow able to eval to a drv? I don't understand this magic well enough.

An easy way of doing this is to generate a list instead of a set; just map mkTest tests.

[Mindavi]

I did some experiments with making multiple outputs and by doing that properly, there was no loop in the references anymore. I should look into that some time (this PR isn't high-priority for me, just a nice to have).

[hmenke]

It would be amazing if we had the full ZTS as a multiple output derivation. This would allow us to run the full ZTS as a single-shot test when bumping the ZFS version. With zfsUnstable there are often demands to “unlock” it for kernel version that have not been marked as compatible by upstream. If such a configuration passes the ZTS I would be much more comfortable approving such a merge, see also #145458

[Mindavi]

Rebased on top of latest master, but the deadman test now crashes the kernel and hangs indefinitely.

I think this is starting to be ready to go though. The tests are now put in a separate 'zfs_tests' output, to ensure they're properly separated.

Only thing that'd still be nice is if we could remove all hardcoded paths to the tools, but since the ZTS does a little dance for setting up the PATH that I don't fully understand, I'm not 100% sure how to approach that properly.

[Mindavi]

@ofborg test zfs

[Mindavi]

@ofborg build zfs

[Mindavi]

Hmm, so I should add it to passthru and to the toplevel tests file still, I think.

[Atemu]

Two things:

• The test isn't discoverable in nixosTests
• We need to be able to test a matrix of zfs and kernel versions.

[hmenke]

Oh boy, look at the time! It's time to resurrect this PR yet again :)

I think these scripts could use makeWrapper instead of awkwardly patching them. Most variables use the pattern LOSETUP=${LOSETUP:-/sbin/losetup}, so they can be overridden using makeWrapper ... --set LOSETUP "${util-linux}/bin/losetup".

Also what's up with all the ksh stuff? I've read through zfs-tests.sh and to me it looks like ksh is only used on FreeBSD.

[Mindavi]

I agree! Hopefully I can find some time soon to maybe clean this up :). Of course, feel free to contribute if you feel like it.

[Mindavi]

I rebased this on top of master, didn't change anything else for now (other than solve a minor merge conflict). Seems to be still working in this form.

[Mindavi]

I'm not really interested in getting this working anymore. I'm not that big of a ZFS user and I did this at some point to scratch an itch. It seems like the tests are now broken and there were some features I couldn't figure out how to implement (test all zfs versions). Hopefully someone can pick up from where I left, if someone's still interested in this.