Apple macOS Intel host -> macOS Intel guest VM support / repo naming

[jerrymarino]

Good morning, thank you for making this tool. The API and OCI innovation here is setting this apart from all the rest of them.

I'm looking to spin up VMs in general on macOS hosts. Apples low level tooling should work on x86 as well from my understanding, but the repo naming and title is oriented around Apple Silicon. I'm about to embark on testing this out on intel -> intel VMs, and I was wondering if there were any major blockers and if supporting this in the mainline is a reasonable workstream.

[jerrymarino]

Also I was reading through #67 but wasn't sure if it was around intel -> intel or arm64 -> intel!

[fkorotkov]

#67 was about intel -> intel but for macOS which doesn't seem will be possible at all. But for Linux VMs it seems more feasible with the recent updates to Virtualization.Framework as part of the new macOS Ventura. As part of #62 we'll try to run on Intel and see how it goes (plus we'll need to evaluate engineering cost of Intel maintenance given the fact there will be no new Intel Macs). If that will work we'll update repo description but for now Tart only runs on Apple Silicon.

What kind of VMs are you planning to run? Could you please elaborate you use case a bit?

[jerrymarino]

#67 was about intel -> intel but for macOS which doesn't seem will be possible at all.

Yeah concretely, I'm looking to run macOS intel hosts -> macOS intel guests under something like tart. Perhaps if this doesn't work in their higher level framework that tart uses e.g. Virtualizaton.framework, and you're not looking support the lower level hypervisor constructs, then it is ruled out!

[fkorotkov]

Yeah, then it's a duplicate of #67. Veertu's Anka 2 is still supporting Intel since they implemented the low level stuff. But they also moved to Virtualization.Framework for the Anka 3. Seems all the virtualization tools are slowly abandoning Intel Macs... Apple Silicon is amazing though!

[jerrymarino]

Ah ok well let me close it out then, sorry for the duplication! I updated the title to make this concrete for the next person.

Apple Silicon is amazing though!

I'd be curious to know about this more if you have a few cycles to spare 😄

1. In general have you found the new para virt integration fast enough to run concurrent iOS simulators? I saw the 4g memory limitation in this project, so couldn't help but wonder about concurrency: do you find it better to run concurrent VMs or concurrent tests within 1 VM?

2. Do you maintain a decent host uptime with their new VM frameworks or do the hosts require frequent reboots because of leaky resources?

3. Also overall with the modern offerings for iOS virtualization, I can't help but wonder if would make sense to virtualize macOS to run simulators. e.g. this may add a potentially redundant layer ( macOS kernel ) in the mix if we can reasonably spawn the tests against a virtualized iOS device.

[fkorotkov]

Sure thing;

1. Virtualization.Framework provides nearly native performance in syntactic tests when I created a VM with the same resources as the host. The 2CPUs / 4Gb is just the default minimal settings. For our cloud offering at Cirrus CI we use VMs with 4CPUs / 8Gb to run two of those on a single M1 Mac Mini. But I guess you can run as many simulators as you can run on a bare metal M1 Mac Mini in a VM that takes up all resources.
2. We've been running hosts for weeks with dozens VM runs a day on each one of them without any issues.
3. Apple did some incredible job with Virtualization.Framework. Seems there are no performance implications. We only saw some issue with paravirtualized GPU when Metal APIs are used intensively. VM's GPU just crashed. But I had a call with Apple folks about the problem during WWDC and they told me Ventura has a lot of improvements in that department. And indeed I tried to reproduce the crashing GPU issue in a Ventura VM running on a Monterey host and everything worked without crashing.

[jerrymarino]

Cool stuff, thanks for sharing your wisdom on this! I'm also impressed by these para virt capabilities and this polished end to end integration. It's good to know you were able to successfully mix and match up a Montery host with the virtualized devices in Ventura as well, and are having reasonable uptime!

[astlock]

Hi @fkorotkov. Thanks for sharing your thoughts. I also see random issues with GPU restarts in Monterey VMs and try to catch similar behavior before everything went into a frozen state but no luck. Could you share how manually reproduce the crashing GPU issue/restarts? Any hints would be useful!

[fkorotkov]

Here are the steps to reproduce from my feedback assistant issue:

brew install cirruslabs/cli/tart 
tart clone ghcr.io/cirruslabs/macos-monterey-xcode:12.4 monterey-xcode
tart run --vnc monterey-xcode 

Then inside the VM do:

git clone --branch cirrus-m1 https://github.com/fkorotkov/plugins
cd plugins 
git fetch origin 
xcrun simctl create Flutter-iPhone com.apple.CoreSimulator.SimDeviceType.iPhone-11 com.apple.CoreSimulator.SimRuntime.iOS-15-5 | xargs xcrun simctl boot
./script/tool_runner.sh build-examples --ios 
./script/tool_runner.sh drive-examples --ios --exclude=script/configs/exclude_integration_ios.yaml

This was crashing the GPU constantly.

@astlock, why do you want to reproduce the crash? It seems the fix is to use Ventura VMs even on Monterey hosts.

[astlock]

@fkorotkov Thanks a lot! The fact is that our product cannot use Xcode beta, and there is no other in Ventura. At the same time, problems with the GPU in our case are rather random, therefore, I am looking for a way to somehow check the virtual machine in advance to see if it will go into these restarts or not.