Problem: QEMUVM killed before shutdown command (#698)

* Problem: QEMUVM killed before shutdown command

When running the controller as a systemd service (the normal usecase in prod) and stopping the service
the QEMU process was always killed BEFORE the shutdown command could be sent
so the VM could not properly clean up

As an additional symptom this error appeared and confused dev and user
```
Sep 05 12:24:03 testing-hetzner python3[2468548]: 2024-09-05 12:24:03,187 | ERROR | Task exception was never retrieved
Sep 05 12:24:03 testing-hetzner python3[2468548]: future: <Task finished name='Task-3' coro=<QemuVM.stop() done, defined at /opt/aleph-vm/aleph/vm/hypervisors/qemu/qemuvm.py:149> exception=QMPCapabilitiesError()>
Sep 05 12:24:03 testing-hetzner python3[2468548]: Traceback (most recent call last):
Sep 05 12:24:03 testing-hetzner python3[2468548]:   File "/opt/aleph-vm/aleph/vm/hypervisors/qemu/qemuvm.py", line 151, in stop
Sep 05 12:24:03 testing-hetzner python3[2468548]:     self.send_shutdown_message()
Sep 05 12:24:03 testing-hetzner python3[2468548]:   File "/opt/aleph-vm/aleph/vm/hypervisors/qemu/qemuvm.py", line 141, in send_shutdown_message
Sep 05 12:24:03 testing-hetzner python3[2468548]:     client = self._get_qmpclient()
Sep 05 12:24:03 testing-hetzner python3[2468548]:              ^^^^^^^^^^^^^^^^^^^^^
Sep 05 12:24:03 testing-hetzner python3[2468548]:   File "/opt/aleph-vm/aleph/vm/hypervisors/qemu/qemuvm.py", line 136, in _get_qmpclient
Sep 05 12:24:03 testing-hetzner python3[2468548]:     client.connect()
Sep 05 12:24:03 testing-hetzner python3[2468548]:   File "/opt/aleph-vm/qmp.py", line 162, in connect
Sep 05 12:24:03 testing-hetzner python3[2468548]:     return self.__negotiate_capabilities()
Sep 05 12:24:03 testing-hetzner python3[2468548]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Sep 05 12:24:03 testing-hetzner python3[2468548]:   File "/opt/aleph-vm/qmp.py", line 88, in __negotiate_capabilities
Sep 05 12:24:03 testing-hetzner python3[2468548]:     raise QMPCapabilitiesError
Sep 05 12:24:03 testing-hetzner python3[2468548]: qmp.QMPCapabilitiesError
Sep 05 12:24:03 testing-hetzner python3[2468548]: 2024-09-05 12:24:03,285 | WARNING | Process terminated with 0
```

Solution: Use mixed kill mode in Systemd, which will at first only send
the term signal to the main process, and give the VM time to properly
cleanup and shutdown.

Note that some time the "shutdown" error is not acted upon so stoping
the process is still necessary. It seems to happend when the boot is not
completed yet.

So a fallback kill is done after a timeout.

---------

Co-authored-by: Hugo Herter <[email protected]>

Author: olethanh

packaging/aleph-vm/etc/systemd/system/[email protected]

@@ -11,6 +11,12 @@ WorkingDirectory=/opt/aleph-vm
 Environment=PYTHONPATH=/opt/aleph-vm/:$PYTHONPATH
 ExecStart=/usr/bin/python3 -m aleph.vm.controllers --config=/var/lib/aleph/vm/%i-controller.json
 Restart=on-failure
+# KillMode=Mixed is used so initially only the Python controller process receives the SIGTERM signal.
+# The controller catches it and sends a QEMU command to shut down the Guest VM, allowing it to clean up
+# properly and avoid disk corruption.
+# After 30s (TimeoutStopSec), if the process is still running, both the controller and subprocesses receive SIGKILL.
+KillMode=mixed
+TimeoutStopSec=30
 
 [Install]
 WantedBy=multi-user.target

src/aleph/vm/controllers/__main__.py

@@ -90,6 +90,7 @@ async def handle_persistent_vm(config: Configuration, execution: MicroVM | QemuV
 
     def callback():
         """Callback for the signal handler to stop the VM and cleanup properly on SIGTERM."""
+        logger.debug("Received SIGTERM")
         loop.create_task(execution.stop())
 
     loop.add_signal_handler(signal.SIGTERM, callback)