tests/rspec: re-factor cl version

[squat]

Make the Container Linux version getter resistant to races against CLUO.
By reading /var/lib/update_engine/prefs/aleph-version, we can be sure
that no matter if CLUO runs before or after the SSH command, we will
always know the version of the OS at installation time.

This commit also adds rescues to the SSH commands to ensure that the
executions will be retried if the node is rebooted by CLUO during the
test.

cc @alexsomesan

[alexsomesan]

It would be cool to use this ssh_exec wrapper everywhere where ssh is needed by tests.
But to do that we need to make the retry behaviour configurable as some actions might not be retriable or might lead to unexpected results when inadvertently retried.
Also, unbounded retries are a very efficient way of shooting one's foot off.
How about we add a retry count here?

[squat]

yeah sounds good. I did not want to overengineer this wrapper but it definitely makes sense if we want to generalize its use. I had already ported the wait_for_bootstrap check to use this method in a previous PR. I can port the azure_vpn's use of ssh to this method in a follow up, since it is unrelated.

[alexsomesan]

Thanks a lot for porting the rest of the use cases to the wrapper. It certainly helps fighting tech debt as we improve the test code.
Even if we were to not use it in a more generalised setting, the unbounded retry is dangerous, especially when put into the perspective of flaky networking to the node instances.

[squat]

Yes that's a good point; I didn't want to overengineer this method but it makes sense especially if we want to generalize its use. I already had ported the wait_for_bootstrap method to use this wrapper in a previous PR. I'll port the azure_vpn use of SSH to this wrapper as well but in a followup since it is unrelated to this race condition.

[squat]

PTAL again @alexsomesan.

Regarding the retries, yes that's a good point; I didn't want to overengineer this wrapper but it makes sense especially if we want to generalize its use. I already had ported the wait_for_bootstrap method to use this wrapper in a previous PR. I'll port the azure_vpn use of SSH to this wrapper as well but in a followup since it is unrelated to this race condition.

[cpanato]

retest this please

[squat]

ok to test

[squat]

Got the following Azure error:

2 error(s) occurred:

* module.masters.azurerm_virtual_machine.tectonic_master: 1 error(s) occurred:

* azurerm_virtual_machine.tectonic_master: compute.VirtualMachinesClient#CreateOrUpdate: Failure responding to request: StatusCode=400 -- Original Error: autorest/azure: Service returned an error. Status=400 Code="SubscriptionNotAuthorizedForImage" Message="The subscription is not authorized."
* module.workers.azurerm_virtual_machine.tectonic_worker: 1 error(s) occurred:

* azurerm_virtual_machine.tectonic_worker: compute.VirtualMachinesClient#CreateOrUpdate: Failure responding to request: StatusCode=400 -- Original Error: autorest/azure: Service returned an error. Status=400 Code="SubscriptionNotAuthorizedForImage" Message="The subscription is not authorized."

[squat]

@cpanato all the AWS tests passed this time. Some azure flakes. Going to run again

[squat]

ok to test

[cpanato]

created a Jira ticket for the flaky issue

[cpanato]

retest this please

[cpanato]

👍