nixos/test-driver: provide a global timeout

[RaitoBezarius]

Description of changes

Since the debut of the test-driver, we didn't obtain a race timer with the test execution to ensure that tests doesn't run beyond a certain amount of time.

This is particularly important when you are running into hanging tests which cannot be detected by current facilities (requires more pvpanic wiring up, QMP API stuff, etc.).

Two easy examples:

• Some QEMU tests may get stuck in some situation and run for more than 24 hours → we default to 24 hours max.
• Some QEMU tests may panic in the wrong place, e.g. UEFI firmware or worse → end users can set a "reasonable" amount of time

And then, we should let the retry logic retest them until they succeed and adjust their global timeouts.

Of course, this does not help with the fact that the timeout may need to be a function of the actual busyness of the machine running the tests. This is only one step towards increased reliability.

Self tests are offered via testBuildFailure.

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
  • sandbox = relaxed
  • sandbox = true
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 23.11 Release Notes (or backporting 23.05 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

[roberth]

👍 this is more robust than meta.timeout, e.g. when tests are accumulated with symlinkJoin or some reporting derivation.
Would love to use this in Nix itself to report which tests have run in which test environments. That's currently far too opaque.

TODO: self tests?

You could write a test with a short timeout and a long sleep somewhere, then transform it with testers.testBuildFailure, which produces a log in its $out.
You can then grep the log from a runCommand to make sure it failed in the right way (ie not due to an unrelated error).

[tfc]

General timeouts are useful and generally an improvement for CI throughput.

I have two thoughts about this change:

For how many people is a test without a timeout useful at all? In interactive mode (which could run for days for playing with), the timer is not started, which makes sense.
As much as i like "maybe" values, we could get rid of this complexity by simply always having a timeout. And people who want no timeout can still simulate it by setting it to a week or so (and interactive users don't need to touch it at all).

The default value of 24 hours is already very relaxed. What is our knowledge about the longest legitimately running test? It might make sense to already use this CI-throughput-improving functionality by setting a useful halfway strict global timeout (as it still can be set to longer times for tests which really need it).

[RaitoBezarius]

General timeouts are useful and generally an improvement for CI throughput.

I have two thoughts about this change:

For how many people is a test without a timeout useful at all? In interactive mode (which could run for days for playing with), the timer is not started, which makes sense.
As much as i like "maybe" values, we could get rid of this complexity by simply always having a timeout. And people who want no timeout can still simulate it by setting it to a week or so (and interactive users don't need to touch it at all).

Sounds reasonable!

The default value of 24 hours is already very relaxed. What is our knowledge about the longest legitimately running test? It might make sense to already use this CI-throughput-improving functionality by setting a useful halfway strict global timeout (as it still can be set to longer times for tests which really need it).

Discussed it on infra and 1 hour seems better.

[nikstur]

Very very cool

[blitz]

I love it! ❤️

[RaitoBezarius]

For self testing, I am reviving #157161 and plan to use it as a base for the self test here.

[RaitoBezarius]

@roberth Done for everything in this scope. I would suggest to get this first probably. :)