Clarification needed regarding building/running Virtualbox-KVM

[eugenesan]

I am working on packaging virtualbox-kvm for Debian/Ubuntu and I have a few questions:

0. I was able to build and run virtualbox-kvm but VMs won't start due to missing vboxdrv. Is there a special step I am missing to utilize KVM?
1. Do we need to setup KVM service/access for a regular user?
2. readme mentions that vboxdrv is not required but build still exports out/bin/src/vboxdrv. Should I just ignore it?
3. Network modules netflt and netadp are exluded from the build. Could you provide a brief guide on how network support should be enabled/configured? Also, do we need to start anything on boot?
4. Is there support for guest extensions under KVM? Should we just replace virtualbox-guest-additions-iso with https://github.com/virtio-win/virtio-win-pkg-scripts/blob/master/README.md ?

This is the configure command my package utilizes:

./configure \
    --ose \
    --odir=/home/username/Develop/ppa/virtualbox-7.0.14-dfsg-1/out \
    --with-linux="/usr" \
    --disable-kmods \
    --enable-webservice \
    --enable-vde \
    --enable-vnc \
    --with-kvm

I tried to remove "--ose" but it made no difference and I guess we can ignore missing --disable-hardening since it's not an issues with distro packages.

[parthy]

I am working on packaging virtualbox-kvm for Debian/Ubuntu

That's awesome! I will try my best to assist with this effort.

0. I was able to build and run virtualbox-kvm but VMs won't start due to missing vboxdrv. Is there a special step I am missing to utilize KVM?

That's odd. I have a faint memory of a case where the VirtualBox binary was the one I built, but somehow the other binaries (e.g., VirtualBoxVM) were used from the system package. I had to run PATH=./out/linux.amd64/release/bin:$PATH ./out/linux.amd64/release/bin/VirtualBox to work around that. But I haven't experienced that in a while now.. Not sure if that applies to this context, but is there a way you can make sure that the newly built binaries are actually the ones executed? And could you try and obtain some logs of the attempt to start the VM?

Also, if there is a way to give us access to the packaging scripts or a finished package, we could also take a closer look into what's happening.

1. Do we need to setup KVM service/access for a regular user?

Generally speaking, KVM must be available to the VMM. I would imagine this needs to be done similar to when Qemu is used as the VMM. It seems like sometimes, suid is used (#6), but so far, "just running the VM" worked fine for me when I tested this on Ubuntu.

2. readme mentions that vboxdrv is not required but build still exports out/bin/src/vboxdrv. Should I just ignore it?

Yes, that can be ignored. I'd have to take a closer look if there is a way to prevent the kernel part build from being prepared, but for the package, it should not matter.

3. Network modules netflt and netadp are exluded from the build. Could you provide a brief guide on how network support should be enabled/configured? Also, do we need to start anything on boot?

Yes, thanks to #3 we noticed that we did not include any information about the networking limitations. We will provide those in the next update. For now, just using standard NAT should work fine.

4. Is there support for guest extensions under KVM? Should we just replace virtualbox-guest-additions-iso with https://github.com/virtio-win/virtio-win-pkg-scripts/blob/master/README.md ?

There should be no change to the guest extensions. The relevant emulated devices are still all provided by VirtualBox and so the standard guest additions available through VirtualBox should work just fine. Anything regarding virtio would only be needed when VirtIO devices are selected in the config. But all this should be identical behavior between the KVM variant and stock VirtualBox.

[antermin]

@parthy

It seems that several distros, such as Debian's virtualbox-qt, Arch Linux's virtualbox, and openSUSE's virtualbox, apply setuid to the binaries.

[VanNostrand]

It works so far, using the current Gentoo ebuild. All I did was unloading the kernel modules, uninstalling the virtualbox and virtualbox-modules packages, then install the new virtualbox-kvm package.

But the warnings created by Virtualbox are kind of an issue, as I use virtualbox with vagrant, which seems to scan the Virtualbox output and then exits with the following error message:

The provider 'virtualbox' that was requested to back the machine
'default' is reporting that it isn't usable on this system. The
reason is shown below:

VirtualBox is complaining that the kernel module is not loaded. Please
run `VBoxManage --version` or open the VirtualBox GUI to see the error
message which should contain instructions on how to fix this error.

But the VM I created before switching to virtualbox-kvm is running so far if I start it manually. I reconfigured it to use virtio-scsi, virtio-net and KVM paravirtualization interface and change a bridged network to NAT though, as the network configuration prevented the VM to be started at first.
Not sure if "vagrant ssh" will work without the bridged network though, but I cannot test in the current state and I am not sure if that is more a vagrant or more a virtualbox-kvm issue, but I guess as virtualbox-kvm does not require the modules it should not throw/print errors.

[eugenesan]

@parthy
Thank you for your reply.

1. I did ensure that the only VirtualBox on the system is the one I've built since I replaced the one installed on the system previously. I am still getting the following dialog:

Kernel driver not installed (rc=-1908)

The VirtualBox Linux kernel driver is either not loaded or not set up correctly. Please reinstall virtualbox-dkms package and load the kernel module by executing

'modprobe vboxdrv'

as root.

If your system has EFI Secure Boot enabled you may also need to sign the kernel modules (vboxdrv, vboxnetflt, vboxnetadp, vboxpci) before you can load them. Please see your Linux system's documentation for more information.

where: suplibOsInit what: 3 VERR_VM_DRIVER_NOT_INSTALLED (-1908) - The support driver is not installed. On linux, open returned ENOENT. 

Machine log is not created and VBoxSVC.log contains the following relevant lines:

...
Executable: /usr/lib/virtualbox/VBoxSVC
...
SUP: In driverless mode.
...
00:10:33.252358 nspr-5   ERROR [COM]: aRC=VBOX_E_INVALID_VM_STATE (0x80bb0002) aIID={300763af-5d6b-46e6-aa96-273eac15538a} aComponent={MachineWrap} aText={The machine is not mutable (state i
s PoweredOff)}, preserve=false aResultDetail=0
00:10:34.890480 nspr-6   Launched VM: 2013270944 pid: 322786 (0x4ece2) frontend: GUI/Qt name: Ubuntu-23.10
00:10:34.947318 nspr-5   ERROR [COM]: aRC=VBOX_E_INVALID_VM_STATE (0x80bb0002) aIID={300763af-5d6b-46e6-aa96-273eac15538a} aComponent={MachineWrap} aText={The machine is not mutable (state i
s PoweredOff)}, preserve=false aResultDetail=0
00:10:35.391389 Watcher  ERROR [COM]: aRC=NS_ERROR_FAILURE (0x80004005) aIID={300763af-5d6b-46e6-aa96-273eac15538a} aComponent={MachineWrap} aText={The virtual machine 'Ubuntu-23.10' has ter
minated unexpectedly during startup with exit code 1 (0x1)}, preserve=false aResultDetail=0
00:10:35.570302 nspr-2   ERROR [COM]: aRC=VBOX_E_INVALID_VM_STATE (0x80bb0002) aIID={300763af-5d6b-46e6-aa96-273eac15538a} aComponent={MachineWrap} aText={The machine is not mutable (state i
s PoweredOff)}, preserve=false aResultDetail=0

I wonder if KVM code fails for some reason and it just shows Virtualbox's default error msg regarding VboxDRV or KVM code is not active at all?

3. I wonder if my failure to run has anything to do with the fact that I am using Debian's source which is sanitized from useless junk (like non-free and non-linux stuff). FYI, I did compare all 3 trees (Oracle, Debian and KVM) and could not find any functional difference and KVM patches (git format-patch baa237b..0396965) applied cleanly but I might have missed something.
4. I've uploaded experimental package to: https://launchpad.net/~eugenesan/+archive/ubuntu/boiler/+packages?field.name_filter=virtualbox.

[eugenesan]

It works so far, using the current Gentoo ebuild. All I did was unloading the kernel modules, uninstalling the virtualbox and virtualbox-modules packages, then install the new virtualbox-kvm package.

But the warnings created by Virtualbox are kind of an issue, as I use virtualbox with vagrant, which seems to scan the Virtualbox output and then exits with the following error message:

The provider 'virtualbox' that was requested to back the machine
'default' is reporting that it isn't usable on this system. The
reason is shown below:

VirtualBox is complaining that the kernel module is not loaded. Please
run `VBoxManage --version` or open the VirtualBox GUI to see the error
message which should contain instructions on how to fix this error.

But the VM I created before switching to virtualbox-kvm is running so far if I start it manually. I reconfigured it to use virtio-scsi, virtio-net and KVM paravirtualization interface and change a bridged network to NAT though, as the network configuration prevented the VM to be started at first. Not sure if "vagrant ssh" will work without the bridged network though, but I cannot test in the current state and I am not sure if that is more a vagrant or more a virtualbox-kvm issue, but I guess as virtualbox-kvm does not require the modules it should not throw/print errors.

Could you share your ebuild for me to study?

[VanNostrand]

Could you share your ebuild for me to study?

The ebuild is part of the gentoo repository since today (but I did not create it), here is the github PR: gentoo/gentoo#35258

[thesamesam]

cc @ceamac

[parthy]

@eugenesan With your package, I was able to run the VM with /usr/lib/VirtualBoxVM --driverless --startvm <VM name>. I'm not too experienced in packaging VirtualBox for Ubuntu, so I don't immediately see why this is happening. The functionality seems to be there, but somehow not used by default as it is in the case where the locally built binaries are used.

[eugenesan]

@eugenesan With your package, I was able to run the VM with /usr/lib/VirtualBoxVM --driverless --startvm <VM name>. I'm not too experienced in packaging VirtualBox for Ubuntu, so I don't immediately see why this is happening. The functionality seems to be there, but somehow not used by default as it is in the case where the locally built binaries are used.

Thank you for testing.
Now I have a direction to investigate.
I'll report back once the issue resolved.

[lgbaldoni]

Chiming in, currently adapting the openSUSE build script for the regular Virtualbox to this and I have a few doubts:

• is USB passthrough now handled solely by kvm?
• Which guest drivers/tools are still necessary or still functional? I'm thinking of networking and filesharing in particular.
• Are official extensions still usable?

So far the package builds (still finding my way through the numerous patches added over the years), will let you know how it works.

EDIT:

• Can I safely do away with anything related to host drivers (vboxdrv, vboxdrvu, vboxnetctl)?

[parthy]

Thank you for your interest and work on packaging!

Is USB passthrough now handled solely by kvm?

As far as I know, the pass-through functionality has always been handled by the user-space part of VirtualBox. This remains unchanged when using KVM as the hypervisor. Together with the udev rules that get deployed with distributed packages, this should work as expected. We are also currently updating the readme to include some information about this topic.

Which guest drivers/tools are still necessary or still functional? I'm thinking of networking and filesharing in particular.

As mentioned in #3, there are limitations to the networking capabilities (which we will also include in the readme with the next update). Other than that, all convenience features (e.g., file sharing, USB, clipboard) should work as usual, at least we have not observed and restrictions. If there is anything odd, feel free to create an issue and we can look into it.

Are official extensions still usable?

Guest additions and host extensions should work fine as well, of course provided the license agreement conditions are observed. We might not have covered all aspects out there, so if there is anything in particular you are referring to or encounter problems with, we are eager to hear about it as well.

Can I safely do away with anything related to host drivers (vboxdrv, vboxdrvu, vboxnetctl)?

The goal is to operate VirtualBox without any additional kernel modules. So yes, none of the drivers you mentioned are needed and would probably in fact interfere or not work anyway.

I hope this helps, if there is anything else, don't hesitate to reach out again here or in a separate issue, if that's appropriate!

[lgbaldoni]

Thank you for your interest and work on packaging!

And thank you for your work on KVM, it definitely fills a gap.

Is USB passthrough now handled solely by kvm?

As far as I know, the pass-through functionality has always been handled by the user-space part of VirtualBox. This remains unchanged when using KVM as the hypervisor. Together with the udev rules that get deployed with distributed packages, this should work as expected. We are also currently updating the readme to include some information about this topic.

Then what I saw in virt-manager must have been managed by qemu-kvm. It's a bit of a disappointment, since it appears to work better in that regard and it doesn't require closed-source extensions for USB 3.0 support.
But Rome wasn't built in a day, I suppose.

Which guest drivers/tools are still necessary or still functional? I'm thinking of networking and filesharing in particular.

As mentioned in #3, there are limitations to the networking capabilities (which we will also include in the readme with the next update).

Will I still need to ship VBoxNetNAT, VBoxNetDHCP and VBoxNetAdpCtl with the host package? Perhaps it's obvious, but I haven't tested the whole setup, yet.

I hope this helps, if there is anything else, don't hesitate to reach out again here or in a separate issue, if that's appropriate!

Well then, since I'm here I'll ask a broader question: do you plan on following upstream Virtualbox in the long term and re-integrate your changes for each new release or fork it outright?

[tpressure]

Then what I saw in virt-manager must have been managed by qemu-kvm. It's a bit of a disappointment, since it appears to work better in that regard and it doesn't require closed-source extensions for USB 3.0 support.
But Rome wasn't built in a day, I suppose.

Just to clarify: since VirtualBox 7.x, no closed source extensions are required for USB 3.0 support. It's now part of the open source release. See: https://github.com/cyberus-technology/virtualbox-kvm/blob/dev/src/VBox/Devices/USB/DevXHCI.cpp

[snue]

Just an observation, we do set VBOX_WITH_DRIVERLESS_FORCED=1 when building the KVM (NEM) backend, but we do it too late to force disable the hardening with it.

  # No need for hardening when not using the driver.

When using the KVM backend the binaries should probably not be installed with setuid. There might be some other things governed by the hardening that I currently missed, but they seem to mostly concern the sane invocation of the privileged binaries.

[parthy]

Well then, since I'm here I'll ask a broader question: do you plan on following upstream Virtualbox in the long term and re-integrate your changes for each new release or fork it outright?

We are not planning to fork and diverge from upstream VirtualBox. Speaking about long-term plans is always a bit of a gamble, of course, but for now we are expecting to rebase our changes onto new versions as they are released.

[parthy]

Just to let everyone know, we have now included more information in the readme. Feel free to have a look and let us know if that clarifies things!

[xvim]

I have a question about using the VirtualBox KVM Backend. What are the differences and advantages of using it compared to using qemu/kvm directly (or the virt-manager graphical frontend)?

[parthy]

I have a question about using the VirtualBox KVM Backend. What are the differences and advantages of using it compared to using qemu/kvm directly (or the virt-manager graphical frontend)?

Thanks for your interest! The goal of this project is not to convince people to use VirtualBox over qemu/kvm (or any other virtualization solution, for that matter). What we provide is an alternative backend for existing VirtualBox users that allows them to leverage KVM hypervisor functionality (with the benefits mentioned in the readme, such as running qemu and VirtualBox side by side), while maintaining the majority of the convenience aspects provided by VirtualBox.

Analysing particular requirements and recommending a fitting virtualization solution is outside the scope of this backend project. But feel free to reach out to us directly if you have any further questions!

[dionorgua]

I was able to add KVM patches to latest version 7.0.14 on Debian Unstable and get binary packages. It's pretty easy.
Btw it looks like adding --with-kvm conflicts somehow with building of virtualbox-dkms and virtualbox-source package. So it was required to disable them.

My ./configure args are followed:

        ./configure \
            --ose \
            --odir=$(abspath out) \
            --with-linux="/usr" \
            --with-kvm \
            --disable-kmods \
            --enable-webservice \
            --enable-vde \
            --enable-vnc

VirtualBox GUI still complains about missing vboxdrv kernel module and can't start VM by default.
So there is no way to 'Start' VM from UI.

At the same time I can start VM from CLI:

VirtualBoxVM --driverless --startvm VM_NAME

So I'm 100% sure that I'm using patched VirtualBox and it works.

Is something important missing here? Should it be possible to start VM from usual VirtualBox UI?
Any way to make --driverless default? Surely I can add it to VBox.sh but I don't think it's expected behavior.

Another question: Can this patched VirtualBox binary run using standard vboxdrv.ko module? From distro point of view it'll much easier to have single binary package that can choose kvm vs vboxdrv at runtime

Thanks for this really good stuff!

[tpressure]

@dionorgua can you try building it with --disable-hardening and check if this changes the behavior?

[dionorgua]

@tpressure it works with --disable-hardening. It was also needed to remove SUID bit from VirtualBoxVM binary

[dionorgua]

@tpressure So should we assume that --disable-hardening is required right now to use KVM backend?

[tpressure]

@dionorgua this is correct. The KVM-Backend only works with --disable-hardening. The hardening feature is mainly thre to prevent side-loading of VBox kernel code, which is no longer present with the KVM backend.

[dionorgua]

Ok thanks.

For me it feels like just some minor bug somewhere. As I said I was able to run VM with hardening enabled. Just not from UI because I need to pass --driverless to VirtualBoxVM process.

[tpressure]

@dionorgua I agree. Maybe you could open another ticket for this specific issue so we can track this correctly?

[dionorgua]

@tpressure #23