vz: add support for nested virtualization #2530

abiosoft · 2024-08-05T14:12:18Z

This PR adds support for nested virtualization for the VZ driver via the nestedVirtualization yaml config.

~~It is dependent on upstream PR Code-Hex/vz#159 getting merged.~~ VZ has been forked lima-vm/vz#1.

Nested virtualization is currently limited to m3 devices running macOS 15 or newer.

abiosoft · 2024-09-07T08:01:04Z

Testing Instructions

⚠️ This only works on macOS 15 ~~(currently beta)~~ with VZ driver and on M3 devices.

For those willing to try this out, these are the steps.

1. Clone the source

Clone my fork of Lima and checkout vz-nested-virtualization branch.

git clone -b vz-nested-virtualization github.com/abiosoft/lima

2. Build Lima

From the root directory of Lima, perform a build.

PS: You might need to unlink Brew version if previously installed with Brew. brew unlink lima

make && sudo make install

3. Start Lima with nested virtualization enabled

Use VZ driver and enabled nested virtualization.

The lima.yaml config file should contain the following.

vmType: vz
nestedVirtualization: true

4. Validate nested virtualization in Lima VM

Enter the Lima VM with lima (or similar command) and validate the existence of /dev/kvm.

$ ls -lh /dev/kvm
crw-rw---- 1 user kvm 10, 232 Aug 29 09:06 /dev/kvm

If the VM is running Ubuntu, you can install cpu-checker package and run kvm-ok to further ascertain.

$ kvm-ok
INFO: /dev/kvm exists
KVM acceleration can be used

nirs · 2024-09-17T14:25:26Z

pkg/vz/vm_darwin.go

+
+		if macOSProductVersion.LessThan(*semver.New("15.0.0")) {
+			return fmt.Errorf("nested virtualization requires macOS 15 or newer")
+		}


It will be little nicer to add a helper like isNestedVirtualiztionAvailable() with these checks. This can also be implemented for linux, so it can be a common interface for all platforms.

asauber · 2024-09-18T17:07:31Z

I was able to create a Lima VM with KVM support with your testing instructions.

$ limactl shell ubuntunested
WARN[0000] treating lima version "bc39aed.m" from "/Users/asauber/.lima/ubuntunested/lima-version" as very latest release
asauber@lima-ubuntunested:/Users/asauber$ kvm-ok
INFO: /dev/kvm exists
KVM acceleration can be used

Very cool. Thank you for working on this.

AkihiroSuda · 2024-09-19T05:25:19Z

CI is failing https://github.com/lima-vm/lima/actions/runs/10903645853/job/30258186834?pr=2530

# github.com/lima-vm/lima/pkg/vz
Error: pkg/vz/vm_darwin.go:256:18: platformConfig.SetNestedVirtualizationEnabled undefined (type *"github.com/Code-Hex/vz/v3".GenericPlatformConfiguration has no field or method SetNestedVirtualizationEnabled)
make[1]: *** [_output/bin/limactl] Error 1

abiosoft · 2024-09-19T05:33:10Z

CI is failing https://github.com/lima-vm/lima/actions/runs/10903645853/job/30258186834?pr=2530

Yes, that is because the upstream PR is not yet merged and that is why I made it a draft PR.

AkihiroSuda · 2024-10-04T07:35:53Z

Code-Hex/vz#159 remains still unreviewed, maybe the repo should be forked?

abiosoft · 2024-10-04T09:37:21Z

Code-Hex/vz#159 remains still unreviewed, maybe the repo should be forked?

lima-vm/vz#1

Signed-off-by: Abiola Ibrahim <[email protected]>

AkihiroSuda

Thanks

abiosoft · 2024-10-08T13:35:43Z

The failing test does not appear related to the code change and it appears to run Lima successfully https://github.com/lima-vm/lima/actions/runs/11231643603/job/31221517661?pr=2530#step:10:1133. 😕

jandubois · 2024-10-08T15:40:04Z

The failing test does not appear related to the code change and it appears to run Lima successfully

This unfortunately happens all the time, and we are retriggering failed test many times each day. 😞

AkihiroSuda · 2024-10-09T04:20:55Z

CI is green on the master, but this PR seems failing 4 times.

Maybe the failure is caused by another recent change in vz?

AkihiroSuda · 2024-10-09T07:31:14Z

CI is now green, let's merge it

⚠️ **CAUTION: this is a major update, indicating a breaking change!** ⚠️ This MR contains the following updates: | Package | Update | Change | |---|---|---| | [lima-vm/lima](https://github.com/lima-vm/lima) | major | `v0.23.2` -> `v1.0.2` | MR created with the help of [el-capitano/tools/renovate-bot](https://gitlab.com/el-capitano/tools/renovate-bot). **Proposed changes to behavior should be submitted there as MRs.** --- ### Release Notes <details> <summary>lima-vm/lima (lima-vm/lima)</summary> ### [`v1.0.2`](https://github.com/lima-vm/lima/releases/tag/v1.0.2) [Compare Source](lima-vm/lima@v1.0.1...v1.0.2) #### Changes - DNS: - Fixed the host resolver regression in v1.0.0 [#2939](lima-vm/lima#2939) ([#2964](lima-vm/lima#2964)) - `limactl create`: - Fixed races during parallel downloads ([#2903](lima-vm/lima#2903), thanks to [@nirs](https://github.com/nirs)) - Optimized qcow2-to-raw conversion for vz mode ([#2933](lima-vm/lima#2933), thanks to [@nirs](https://github.com/nirs)) - `limactl start-at-login`: - Fixed the support for Linux hosts (systemd) ([#2943](lima-vm/lima#2943), thanks to [@kachick](https://github.com/kachick)) - nerdctl: - Updated to [v2.0.1](https://github.com/containerd/nerdctl/releases/tag/v2.0.1) ([#2966](lima-vm/lima#2966)) - Templates: - Updated to the latest revisions ([#2936](lima-vm/lima#2936) [#2953](lima-vm/lima#2953), thanks to [@tcooper](https://github.com/tcooper)) - Web site: - Added an example of running Lima on GitHub Actions to run commands on non-Ubuntu ([#2954](lima-vm/lima#2954)): https://lima-vm.io/docs/examples/gha/ - Project: - Invite Nir Soffer ([@nirs](https://github.com/nirs)) as a Reviewer ([#2916](lima-vm/lima#2916), thanks to [@jandubois](https://github.com/jandubois)) Full changes: https://github.com/lima-vm/lima/milestone/51?closed=1 Thanks to [@SpiffyEight77](https://github.com/SpiffyEight77) [@alexandear](https://github.com/alexandear) [@jandubois](https://github.com/jandubois) [@kachick](https://github.com/kachick) [@nirs](https://github.com/nirs) [@norio-nomura](https://github.com/norio-nomura) [@tamird](https://github.com/tamird) [@tcooper](https://github.com/tcooper) #### Usage ```console [macOS]$ limactl create [macOS]$ limactl start ... INFO[0029] READY. Run `lima` to open the shell. [macOS]$ lima uname Linux ``` *** The binaries were built automatically on GitHub Actions. The build log is available for 90 days: https://github.com/lima-vm/lima/actions/runs/12134682585 The sha256sum of the SHA256SUMS file itself is `02ef78494c498ca4180915ba78d5e2fc471ed401f63dfb2b5864c3711f3c0fb2` . *** Release manager: [@AkihiroSuda](https://github.com/AkihiroSuda) ### [`v1.0.1`](https://github.com/lima-vm/lima/releases/tag/v1.0.1) [Compare Source](lima-vm/lima@v1.0.0...v1.0.1) Reverted the default port forwarder from gRPC to SSH for the stability reason ([#2864](lima-vm/lima#2864)). This reversion fixes several regressions related to `docker run -p` in Lima v1.0.0 ([#2859](lima-vm/lima#2859)). Although the gRPC forwarder is faster and has an advanced feature (UDP support), it turned out to be still immature. Set `LIMA_SSH_PORT_FORWARDER=false` to opt-in to the gRPC forwarder. See <https://lima-vm.io/docs/config/port/>. Full changes: https://github.com/lima-vm/lima/milestone/50?closed=1 Thanks to [@alexandear](https://github.com/alexandear) [@jandubois](https://github.com/jandubois) [@norio-nomura](https://github.com/norio-nomura) #### Usage ```console [macOS]$ limactl create [macOS]$ limactl start ... INFO[0029] READY. Run `lima` to open the shell. [macOS]$ lima uname Linux ``` *** The binaries were built automatically on GitHub Actions. The build log is available for 90 days: https://github.com/lima-vm/lima/actions/runs/11735352652 The sha256sum of the SHA256SUMS file itself is `f5c12d003e25dc46291803a8acae9e9d325a45eca0c1f9f40bd6852ec8ed9be1` . *** Release manager: [@AkihiroSuda](https://github.com/AkihiroSuda) ### [`v1.0.0`](https://github.com/lima-vm/lima/releases/tag/v1.0.0) [Compare Source](lima-vm/lima@v0.23.2...v1.0.0) With the support from 110+ contributors in 3+ years, the Lima project has finally reached v1.0. 🎉 This release introduces several breaking changes, such as switching the default machine driver from QEMU to VZ for better filesystem performance. The `limactl` CLI is designed to print hints when the user hits those breaking changes. e.g., `limactl create template://experimental/vz` now fails with a hint that suggests using `limactl create --vm-type=vz template://default` instead. 🔴 = Major breaking changes 🟡 = Minor breaking changes - VZ: - Graduate VZ machine driver from experimental ([#2758](lima-vm/lima#2758)) - 🔴 Use VZ by default for new instances on macOS >= 13.5 ([#1951](lima-vm/lima#1951)) - Support nested virtualization on M3 ([#2530](lima-vm/lima#2530), thanks to [@abiosoft](https://github.com/abiosoft)) - Optimize qcow2-to-raw image conversion (lima-vm/go-qcow2reader@v0.1.2...v0.4.0 , thanks to [@nirs](https://github.com/nirs)) - Support specifying a custom kernel ([#2562](lima-vm/lima#2562), thanks to [@norio-nomura](https://github.com/norio-nomura)) - QEMU: - Graduate 9p mount driver from experimental ([#2758](lima-vm/lima#2758)) - 🔴 Use 9p by default for most templates ([#1953](lima-vm/lima#1953), [#2822](lima-vm/lima#2822)) - riscv64: switch from u-boot to EDK2 ([#2592](lima-vm/lima#2592)) - Network: - Graduate user-v2 network driver from experimental ([#2758](lima-vm/lima#2758)) - Support UDP port forwarding ([#2411](lima-vm/lima#2411), thanks to [@balajiv113](https://github.com/balajiv113)) - 🔴 Strictly require `socket_vmnet` binary to be owned by root ([#2734](lima-vm/lima#2734)) - SSH: - 🟡 Disable `ssh.loadDotSSHPubKeys` by default ([#2706](lima-vm/lima#2706)) - YAML: - Support generating jsonschema ([#2306](lima-vm/lima#2306), thanks to [@afbjorklund](https://github.com/afbjorklund)) - Support specifying `param` for provisioning scripts ([#2570](lima-vm/lima#2570), thanks to [@jandubois](https://github.com/jandubois)) - Support specifying `minimumLimaVersion` and `vmOpts.qemu.minimumVersion` ([#2659](lima-vm/lima#2659), thanks to [@jandubois](https://github.com/jandubois)) - Support template expansion in mounts ([#2588](lima-vm/lima#2588), thanks to [@norio-nomura](https://github.com/norio-nomura)) - `limactl` CLI: - Add `limactl tunnel` command so as to allow the host to join the guest network ([#2710](lima-vm/lima#2710)) - Add `--log-format=json` ([#2584](lima-vm/lima#2584), thanks to [@nirs](https://github.com/nirs)) - `limactl prune`: Add `--keep-referred` ([#2569](lima-vm/lima#2569), thanks to [@norio-nomura](https://github.com/norio-nomura)) - nerdctl: - Updated to [v2.0.0](https://github.com/containerd/nerdctl/releases/tag/v2.0.0) ([#2178](lima-vm/lima#2178)) - rootless: allocate 1G subuids from 524288 (0x80000) for new users ([#2725](lima-vm/lima#2725)) - Templates: - 🔴 `experimental/vz`: Merged into the `default` template ([#2730](lima-vm/lima#2730), [#2736](lima-vm/lima#2736)) - 🟡 `experimental/{riscv64, armv7l}`: Merged into the `default` template ([#2730](lima-vm/lima#2730), [#2736](lima-vm/lima#2736)) - 🔴 `vmnet`: Removed in favor of `limactl create --network=lima:shared template://default` ([#2736](lima-vm/lima#2736)) - 🟡 `experimental/net-user-v2`: Removed in favor of `limactl create --network=lima:user-v2 template://default` ([#2736](lima-vm/lima#2736)) - 🔴 `experimental/9p`: Removed in favor of `limactl create --mount-type=9p template://default` ([#2736](lima-vm/lima#2736)) - 🟡 `experimental/virtiofs-linux`: Removed in favor of `limactl create --mount-type=virtiofs template://default` ([#2736](lima-vm/lima#2736)) - 🔴 `alpine`: Renamed to `alpine-iso` ([#2704](lima-vm/lima#2704)) - 🔴 `alpine-image`: Renamed to `alpine` ([#2704](lima-vm/lima#2704)) - `archlinux`: Demoted from Tier 1 to Tier 2 ([#2717](lima-vm/lima#2717), [#2823](lima-vm/lima#2823)) - `default`, `ubuntu`, ...: Updated to Ubuntu 24.10. The older versions are available as `ubuntu-20.04`, `ubuntu-22.04`, and `ubuntu-24.04` ([#2755](lima-vm/lima#2755), [#2795](lima-vm/lima#2795)) - `fedora`: Updated to Fedora 41 ([#2821](lima-vm/lima#2821), [#2822](lima-vm/lima#2822), thanks to [@subpop](https://github.com/subpop)) - `opensuse`: Renamed to `opensuse-leap`. Still aliased as `opensuse` ([#2612](lima-vm/lima#2612), thanks to [@afbjorklund](https://github.com/afbjorklund)) - `experimental/opensuse-tumbleweed`: Support aarch64 ([#2613](lima-vm/lima#2613), thanks to [@afbjorklund](https://github.com/afbjorklund)) - `hack/update-template.sh` is added for automating updates ([#1347](lima-vm/lima#1347), thanks to [@norio-nomura](https://github.com/norio-nomura)) - Project: - Invite Norio Nomura ([@norio-nomura](https://github.com/norio-nomura)) as a Reviewer ([#2567](lima-vm/lima#2567)) Full changes: https://github.com/lima-vm/lima/milestone/47?closed=1 Thanks to [@AdamKorcz](https://github.com/AdamKorcz) [@Mr-Sunglasses](https://github.com/Mr-Sunglasses) [@SmartManoj](https://github.com/SmartManoj) [@YorikSar](https://github.com/YorikSar) [@abiosoft](https://github.com/abiosoft) [@afbjorklund](https://github.com/afbjorklund) [@alexandear](https://github.com/alexandear) [@balajiv113](https://github.com/balajiv113) [@hasan4791](https://github.com/hasan4791) [@jandubois](https://github.com/jandubois) [@nirs](https://github.com/nirs) [@norio-nomura](https://github.com/norio-nomura) [@pvdvreede](https://github.com/pvdvreede) [@subpop](https://github.com/subpop) [@tsukasaI](https://github.com/tsukasaI) #### Usage ```console [macOS]$ limactl create [macOS]$ limactl start ... INFO[0029] READY. Run `lima` to open the shell. [macOS]$ lima uname Linux ``` *** The binaries were built automatically on GitHub Actions. The build log is available for 90 days: https://github.com/lima-vm/lima/actions/runs/11695321667 The sha256sum of the SHA256SUMS file itself is `4bd200a163111fe78c6f3e6de405113d416053802fe1507597f9a42f89a98c90` . *** Release manager: [@AkihiroSuda](https://github.com/AkihiroSuda) </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever MR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this MR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this MR, check this box --- This MR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).

AnthoBalitrand · 2025-01-05T20:59:25Z

Hi all,
Thanks for this add. Still not available on the brew version ?

AkihiroSuda · 2025-01-05T21:05:07Z

Already available since v1.0.0 (last November)

abiosoft marked this pull request as draft August 5, 2024 14:12

abiosoft force-pushed the vz-nested-virtualization branch from 1fd7d72 to b946be0 Compare August 6, 2024 04:09

abiosoft force-pushed the vz-nested-virtualization branch from b946be0 to 25bd2b7 Compare September 7, 2024 07:03

abiosoft mentioned this pull request Sep 7, 2024

add support for nested virtualization Code-Hex/vz#159

Merged

abiosoft force-pushed the vz-nested-virtualization branch from 25bd2b7 to bc39aed Compare September 17, 2024 13:03

nirs reviewed Sep 17, 2024

View reviewed changes

abiosoft force-pushed the vz-nested-virtualization branch 4 times, most recently from b5f9998 to eb784b8 Compare September 21, 2024 13:30

abiosoft force-pushed the vz-nested-virtualization branch from eb784b8 to ddefc80 Compare October 1, 2024 09:03

abiosoft force-pushed the vz-nested-virtualization branch 2 times, most recently from c574b0b to d0a8131 Compare October 8, 2024 08:25

abiosoft marked this pull request as ready for review October 8, 2024 08:30

vz: add support for nested virtualization

10a60fe

Signed-off-by: Abiola Ibrahim <[email protected]>

abiosoft force-pushed the vz-nested-virtualization branch from d0a8131 to 10a60fe Compare October 8, 2024 08:33

AkihiroSuda approved these changes Oct 8, 2024

View reviewed changes

AkihiroSuda added this to the v1.0 milestone Oct 8, 2024

AkihiroSuda added the component/vz label Oct 8, 2024

AkihiroSuda merged commit 7ab02dc into lima-vm:master Oct 9, 2024
27 checks passed

AkihiroSuda added the impact/changelog label Oct 9, 2024

abiosoft deleted the vz-nested-virtualization branch October 9, 2024 07:45

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

vz: add support for nested virtualization #2530

vz: add support for nested virtualization #2530

abiosoft commented Aug 5, 2024 •

edited

Loading

abiosoft commented Sep 7, 2024 •

edited

Loading

nirs Sep 17, 2024

asauber commented Sep 18, 2024

AkihiroSuda commented Sep 19, 2024

abiosoft commented Sep 19, 2024

AkihiroSuda commented Oct 4, 2024

abiosoft commented Oct 4, 2024

AkihiroSuda left a comment

abiosoft commented Oct 8, 2024

jandubois commented Oct 8, 2024

AkihiroSuda commented Oct 9, 2024

AkihiroSuda commented Oct 9, 2024

AnthoBalitrand commented Jan 5, 2025

AkihiroSuda commented Jan 5, 2025

vz: add support for nested virtualization #2530

vz: add support for nested virtualization #2530

Conversation

abiosoft commented Aug 5, 2024 • edited Loading

abiosoft commented Sep 7, 2024 • edited Loading

Testing Instructions

1. Clone the source

2. Build Lima

3. Start Lima with nested virtualization enabled

4. Validate nested virtualization in Lima VM

nirs Sep 17, 2024

Choose a reason for hiding this comment

asauber commented Sep 18, 2024

AkihiroSuda commented Sep 19, 2024

abiosoft commented Sep 19, 2024

AkihiroSuda commented Oct 4, 2024

abiosoft commented Oct 4, 2024

AkihiroSuda left a comment

Choose a reason for hiding this comment

abiosoft commented Oct 8, 2024

jandubois commented Oct 8, 2024

AkihiroSuda commented Oct 9, 2024

AkihiroSuda commented Oct 9, 2024

AnthoBalitrand commented Jan 5, 2025

AkihiroSuda commented Jan 5, 2025

abiosoft commented Aug 5, 2024 •

edited

Loading

abiosoft commented Sep 7, 2024 •

edited

Loading