Reduce unnecessary reboots: part 1, AUD, MNE

[philipanda]

implementing changes for #748

[philipanda]

Logs showing:

1. A normal test case over serial that enters the setup menu works as before
   custom-boot-menu-key_log.html

2. AUD test suite, which had the platform power state control turned on by overriding a variable in *** Variables *** section and removing forced Power On calls. By doing so, the platform is rebooted only once per OS, to boot into the OS, which saves about 50% of execution time in this case: audio-subsystem_log.html

[philipanda]

Splitting the suites into multiple files per OS was supposed to be a way to achieve less reboots during tests. Turns out, that this approach is essentially what was used in a PR to solve #1099 in #1130. The approach where we first gather all the data required for all tests (for a given OS) and then we only interpret the collected data in test cases is essentially what the solution for concurrent execution is. Splitting into multiple files, one per OS, allows to run data gathering on every OS independently.

Just monitoring the power state and reducing reboots if not needed can work without this change. Just ${PLATFORM_STATE_CONTROL}= ${TRUE} and removing forced power ons (Power On Ex force_reboot=${True}) is enough to benefit from reduced reboots. That makes the tests much more tidy though, so I'd stand for keeping the splits from this PR.

Logs after splitting the AUD test suite into files per OS. audio-subsystem_log.html

[macpijan]

Would it make sense to have some tests for these new keywords on QEMU under self-tests to ensure they work properly?

[philipanda]

I guess the best way to test them would be to just do the same setup menu tests as now, just with the power state control ON and Power On Ex force_reboot=${TRUE} swapped to Power On Ex force_reboot=${FALSE}. Alternatively a separate test case could be added that just moves between setup/bootmenu/OS repeatedly a couple times.

[macpijan]

Ex stand for what exactly?

[philipanda]

Extended/Expanded, something to differentiate from Power On for which Power On Ex is a wrapper

[macpijan]

should we explain why do we need to handle this case?

[philipanda]

I think the commit message should show more or less what's the intention: Login To Linux: Handle already logged in via serial. It's added so that if the OS is already booted and the user (or root) is already logged in, we can just continue instead of waiting for the login prompt (which will never appear)

[macpijan]

Do you have some metrics - how long it took to run modified test suite - before and after the changes?

[philipanda]

Do you have some metrics - how long it took to run modified test suite - before and after the changes?

Sure @macpijan , here's a direct comparison from logs:

• AUD originally - 7:17 audio-subsystem_log.html
• AUD with automatic reboot reduction - 4:08 audio-subsystem_log.html
• AUD with automatic reboot reduction and split into files - 3:36 audio-subsystem_log.html

That's 44% time reduction for just the automatic reboot reduction. Suprisingly after splitting into files-per-OS it gets even better at 51% time reduction. I'm nut quite sure what has changed, maybe it's just variation.