nixos/tests/keymap: improve keymap tests

[xeji]

Motivation for this change

Fix #39330. Simplify tests, prevent timeouts and non-deterministic hydra failures.
Hopefully this will stabilize the tests on hydra - let's try and see.

Please backport to 18.03 because failing keymap tests have delayed the channels there as well.

Things done

• Simplify and redesign test logic: a single reader script now waits for all test characters to be typed (previously, a new reader script was started on the guest for each character)
• Improve host-guest synchronization to reduce waiting and timeout risks
• Xorg test: wait for xterm window to open (waitForWindow())
• Xorg test: use xdotool to make sure xterm has focus (from nixos/tests/keymap: Workaround xterm not in focus #33719, thanks @srhb)
• Add some logging to help future debugging
• Verify that all 6 keymap tests pass locally, even when run in parallel with high CPU load (since I don't own a hydra simulator 😀)

Things that could be done later

• Flatten test data sets - I chose to keep test data unchanged for now and have nix flatten it
• Add a standard synchronization mechanism to the test framework; having to sync by creating and waiting for marker files is a pain and error-prone, and everyone rolls their own...

---

[srhb]

@GrahamcOfBorg test keymap.colemak

[GrahamcOfBorg]

Success on x86_64-linux (full log)

Attempted: tests.keymap.colemak

Partial log (click to expand)

syncing
machine: running command: sync
machine: exit status 0
2 out of 2 tests succeeded
test script finished in 18.36s
cleaning up
killing machine (pid 593)
vde_switch: EOF on stdin, cleaning up and exiting
vde_switch: Could not remove ctl dir '/build/vde1.ctl': Directory not empty
/nix/store/a3j8c8clkqdvklalzjzmk9d5bkxw2dhi-vm-test-run-keymap-colemak

[adisbladis]

Works great here even under quite extreme load (load average 18) on my laptop using a https://ark.intel.com/products/95451.

Great work! 😃

[srhb]

Looks excellent! Thanks!

[GrahamcOfBorg]

Failure on aarch64-linux (full log)

Attempted: tests.keymap.colemak

Partial log (click to expand)

cannot build derivation '/nix/store/6fs5pcs38yk83zshbwpd19h067w4llrx-etc.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/4r9i25dqkzfyjflqy9xh7p569jnvi61f-stage-1-init.sh.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/z1hpnas8mh2jhz6z8qy7715hpv971l6i-initrd.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/7d870isrnxysjiryzdfx3z63xh7b7xp0-nixos-system-machine-18.09.git.dd4d1a4.drv': 5 dependencies couldn't be built
cannot build derivation '/nix/store/srdj9y73bdxf0m12gcirwa0jjsa9vd6f-closure-info.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/kqam08i23b5cjblaynj6apxvg344dhiq-run-nixos-vm.drv': 2 dependencies couldn't be built
cannot build derivation '/nix/store/2nsagj1fqjkn0nb2cp7ywcdav1ymsb0r-nixos-vm.drv': 2 dependencies couldn't be built
cannot build derivation '/nix/store/cl4kasd95x05d38rzl2vrqsk7ap2mp52-nixos-test-driver-keymap-colemak.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/m26z62v6hi9qvvmq25sqy7q29fg7rsnd-vm-test-run-keymap-colemak.drv': 1 dependencies couldn't be built
�[31;1merror:�[0m build of '/nix/store/m26z62v6hi9qvvmq25sqy7q29fg7rsnd-vm-test-run-keymap-colemak.drv' failed

[srhb]

People of the future, this is the first Hydra evaluation of nixos:release-18.03 with this commit:

https://hydra.nixos.org/eval/1451135

[srhb]

https://hydra.nixos.org/build/73238407/nixlog/13

Unfortunately, no dice. Shall I revert?

[xeji]

Oops. Please revert on 18.03 while I check it out. Looks like an issue with xdotool and the window manager, will probably have to modify a little for 18.03.

[xeji]

Thanks @srhb for catching this.

Strangely, the same test succeeded in the same eval for i686: https://hydra.nixos.org/build/73238415
Plus I cannot reproduce the above failure (which definitely looks like a deterministic error) on the same git revision.
The test different derivation hashes on the same commit differ between hydra and my local machine (this seems to be the case for other tests as well, whereas normal packages have identical hashes).
Do we have an impurity in the test framework? 😕
Appreciate any ideas how to analyze this...

[srhb]

The tests are quite often impure. And uh, did someone rerun the tests or did I completely misread that there was an error? My old look shows it's fine now. Am I dreaming?

[srhb]

And I'm not sure it looked to me like a deterministic error. Could it be that the property in question does not exist if we ask too fast?

[xeji]

You're not dreaming, I saw the error too ... But I doubt someone restarted that job. No idea what happened.

And I'm not sure it looked to me like a deterministic error. Could it be that the property in question does not exist if we ask too fast?

Maybe. The only thing I found in xdotool doc is that windowactivate may not work with all window managers.

[srhb]

@xeji OK, good! And yes, I saw that too, and I think what's going on is that it doesn't work yet during whatever startup is happening. I can't imagine that the VM would be different between our systems and Hydras on that point.

[xeji]

Since I do waitForWindow before xdotool, the window is already there. But maybe the WM hasn't decorated and set all of the window's properties yet. I could wrap the xdotool call in a waitUntilSucceeds() loop. Or use windowfocus alternative to windowactivate.

[srhb]

If windowfocus works regardless, that seems like the way to go!

[srhb]

(But if we can find a step to actually waitUntilSucceeds that makes this work, I think that's more than acceptable!)

[xeji]

Well the manpage says about windowfocus:

Uses XSetInputFocus which may be ignored by some window managers or programs.

[srhb]

That should be testable in the by spawning another window with focus and switching, right?

[xeji]

Good idea, I'll do that.

I haven't seen any case where the old test failed because of losing focus after the second xterm window was removed. Now there's only one fullscreen window. So I wonder if the xdotool is just an extra failsafe that creates more trouble than it prevents...

[srhb]

Oh, good point. I forgot about that. In that case, feel free to nuke it! :-) I thought it would help us find a safe spot to start the test, but that may indeed not be the case at all.

[xeji]

Let's wait and see how the tests work run on master. I'll do some more experiments and then make an updated version.

[xeji]

Thanks again for your help!