Fix Instance network not working on server reboot

Related Jira ticket : ALEPH-549

Problem:
When the whole server rebooted, internet was not working inside instance
and they were unreachable on the network. As the network interface were not properly recreated

Analysis:

At the instance automatic restart once the serevr rebooted, the network interface is not existant, thus QEMU recreating some kind of default network  that is
not setup as needed (tunneling, ip address,  nat, ipv6, etc.._ and thus the network inside the instance was non functional.)

Solution:
In the controller, wait for the network interface to be recreated by the supervisor
In the supevisor recreate the network interface and set it up if it is not existing
This is done this way as it is not possible to add tuntap after the
interface is created

How to test:
* Stop the supervisor
* Stop the instance controller (via systemctl stop aleph-vm-controller@ )
* Delete the network interface : ip link del vmtapX (e.g. vmtap4 if it is the one the interface is using)
* Start the instance again (systemctl start aleph-vm-controler@... )
* Check that it is waiting for the network interface (journalctl -u aleph-vm-controller@... -f )
* Start the supervisor again, in the log it should create the network interface
* The instance controler should detect it and launch QEMU
* Check that the interface exist and has an ip: ip a
* SSH into the instance , check that the network is working as intented

Author: olethanh

src/aleph/vm/controllers/__main__.py

@@ -6,6 +6,7 @@
 import sys
 from asyncio.subprocess import Process
 from pathlib import Path
+from time import sleep
 
 from aleph.vm.hypervisors.firecracker.microvm import MicroVM
 from aleph.vm.hypervisors.qemu.qemuvm import QemuVM
@@ -127,21 +128,27 @@ def main():
 
     config.settings.check()
 
-    if args.initialize_network_settings:
-        network = Network(
-            vm_ipv4_address_pool_range=config.settings.IPV4_ADDRESS_POOL,
-            vm_network_size=config.settings.IPV4_NETWORK_PREFIX_LENGTH,
-            external_interface=config.settings.NETWORK_INTERFACE,
-            ipv6_allocator=make_ipv6_allocator(
-                allocation_policy=config.settings.IPV6_ALLOCATION_POLICY,
-                address_pool=config.settings.IPV6_ADDRESS_POOL,
-                subnet_prefix=config.settings.IPV6_SUBNET_PREFIX,
-            ),
-            use_ndp_proxy=config.settings.USE_NDP_PROXY,
-            ipv6_forwarding_enabled=config.settings.IPV6_FORWARDING_ENABLED,
-        )
+    network = Network(
+        vm_ipv4_address_pool_range=config.settings.IPV4_ADDRESS_POOL,
+        vm_network_size=config.settings.IPV4_NETWORK_PREFIX_LENGTH,
+        external_interface=config.settings.NETWORK_INTERFACE,
+        ipv6_allocator=make_ipv6_allocator(
+            allocation_policy=config.settings.IPV6_ALLOCATION_POLICY,
+            address_pool=config.settings.IPV6_ADDRESS_POOL,
+            subnet_prefix=config.settings.IPV6_SUBNET_PREFIX,
+        ),
+        use_ndp_proxy=config.settings.USE_NDP_PROXY,
+        ipv6_forwarding_enabled=config.settings.IPV6_FORWARDING_ENABLED,
+    )
 
+    if args.initialize_network_settings:
         network.setup()
+    else:
+        # Wait for the network interface to be by the supervisor
+        # Otherwise QEMU will create a new one, and it won't be properly setup as a tuntap as we expect
+        while not network.interface_exists(config.vm_id):
+            logger.info("Waiting for network interface to be created...")
+            sleep(1)
 
     asyncio.run(run_persistent_vm(config))
 

src/aleph/vm/pool.py

@@ -299,6 +299,9 @@ async def load_persistent_executions(self):
                 if self.network:
                     vm_type = VmType.from_message_content(execution.message)
                     tap_interface = await self.network.prepare_tap(vm_id, vm_hash, vm_type)
+                    if not self.network.interface_exists(vm_id):
+                        # In case of a reboot, the network is not automatically created
+                        await self.network.create_tap(vm_id, tap_interface)
 
                     # Activate ndp_proxy for existing interfaces if needed
                     if self.network.ndp_proxy and self.network.interface_exists(vm_id):