bug: macOS fails to start ('Haswell-v4-x86_64-cpu.ept' not found)

[garrett]

Expected behavior

The VM should start.

Actual behavior

Error message:

qemu-system-x86_64: Property 'Haswell-v4-x86_64-cpu.ept' not found

Steps to reproduce the behavior

Attempt to start a macOS VM with quickemu. Windows 11 works as expected.

Additional context

This is on a Fedora 40 system. I've tried it in a Fedora distrobox, a Debian Sid distrobox, a Fedora toolbox, and by overlaying packages on my Fedora 40 Silverblue system.

Quickemu output

Run quickemu or quickemu and include the output of the failure below:

Quickemu output

./quickemu --vm macos-sonoma.conf
Quickemu 4.9.4 using /usr/bin/qemu-system-x86_64 v8.2.2
- Host:     Fedora Linux 40.20240512.0 (Silverblue) running Linux 6.8 (drought)
- CPU:      11th Gen Intel(R) Core(TM) i7-11850H @ 2.50GHz
- CPU VM:   1 Socket(s), 4 Core(s), 2 Thread(s), 16G RAM
- BOOT:     EFI (macOS), OVMF (OVMF_CODE.fd), SecureBoot (off).
- Disk:     macos-sonoma/disk.qcow2 (96G)
           Looks unused, booting from macos-sonoma/RecoveryImage.img
- Recovery: macos-sonoma/RecoveryImage.img
- Display:  SDL, VGA, GL (on), VirGL (off) @ (1280 x 800)
- Sound:    intel-hda
- ssh:      On host:  ssh user@localhost -p 22220
- 9P:       On guest: sudo mount_9p Public-garrett
- 9P:       On host:  chmod 777 /var/home/garrett/Public
           Required for macOS integration 👆
- Network:  User (virtio-net)
- Monitor:  On host:  nc -U "macos-sonoma/macos-sonoma-monitor.socket"
           or     :  socat -,echo=0,icanon=0 unix-connect:macos-sonoma/macos-sonoma-monitor.socket
- Serial:   On host:  nc -U "macos-sonoma/macos-sonoma-serial.socket"
           or     :  socat -,echo=0,icanon=0 unix-connect:macos-sonoma/macos-sonoma-serial.socket
- Process:  ERROR! Failed to start macos-sonoma.conf as macos-sonoma

qemu-system-x86_64: Property 'Haswell-v4-x86_64-cpu.ept' not found

System Information

Quickreport output (doesn't work)

----------------------------------
      Quickemu missing!
----------------------------------

(I think it looks for quickemu on the path, instead of ./quickemu. It's similar for ./quickget — it suggests running quickemu instead of ./quickemu.)

I ran bash (as bash isn't my default shell) and then the base command as suggested:

Bash command output, like quick report

Architecture:                         x86_64
CPU op-mode(s):                       32-bit, 64-bit
Address sizes:                        39 bits physical, 48 bits virtual
Byte Order:                           Little Endian
CPU(s):                               16
On-line CPU(s) list:                  0-15
Vendor ID:                            GenuineIntel
Model name:                           11th Gen Intel(R) Core(TM) i7-11850H @ 2.50GHz
CPU family:                           6
Model:                                141
Thread(s) per core:                   2
Core(s) per socket:                   8
Socket(s):                            1
Stepping:                             1
CPU(s) scaling MHz:                   45%
CPU max MHz:                          4800.0000
CPU min MHz:                          800.0000
BogoMIPS:                             4992.00
Flags:                                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l2 cdp_l2 ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_pt avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves split_lock_detect user_shstk dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp hwp_pkg_req vnmi avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg tme avx512_vpopcntdq rdpid movdiri movdir64b fsrm avx512_vp2intersect md_clear ibt flush_l1d arch_capabilities
Virtualization:                       VT-x
L1d cache:                            384 KiB (8 instances)
L1i cache:                            256 KiB (8 instances)
L2 cache:                             10 MiB (8 instances)
L3 cache:                             24 MiB (1 instance)
NUMA node(s):                         1
NUMA node0 CPU(s):                    0-15
Vulnerability Gather data sampling:   Mitigation; Microcode
Vulnerability Itlb multihit:          Not affected
Vulnerability L1tf:                   Not affected
Vulnerability Mds:                    Not affected
Vulnerability Meltdown:               Not affected
Vulnerability Mmio stale data:        Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed:               Not affected
Vulnerability Spec rstack overflow:   Not affected
Vulnerability Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:             Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:             Mitigation; Enhanced / Automatic IBRS; IBPB conditional; RSB filling; PBRSB-eIBRS SW sequence; BHI SW loop, KVM SW loop
Vulnerability Srbds:                  Not affected
Vulnerability Tsx async abort:        Not affected

[garrett]

I checked out a prior version of the quickemu file and it started the VM (although macOS never boots; it shows a black screen):

git checkout 9a09bad quickemu

So a commit after 9a09bad might be a problem on at least my system, when running macOS.

Just to note: Windows 11 works fine with the latest code in the main branch, so this seems to be something mac-specific on my hardware?

[garrett]

I did a git bisect and it suggests git commit 5e507e6 is where things broke on my system.

I guess Haswell-noTSX-IBRS is supported, but Haswell-v4 is not?

[flexiondotorg]

I was literally just looking into this 😄

Please can you test this commit and let me know if it works for you: 2d1319e

[japetrus]

In my case removing ept from the flags or replacing it with ept_1gb allowed the system to boot, but changing the cpu model did not.

[flexiondotorg]

@japetrus @garrett Please can you test commit 97e2493 and let me know if this works for you.

[japetrus]

Yes, that works fine for me. I should add that I don't know if ept_1gb is a reasonable substitute for ept beyond the name being similar!

[garrett]

That commit also works for me! Thanks!

[TuxVinyards]

There were/are quite few things wrong. And there is still quite a bit more work to do.

Sorry if this is short and incomplete I was working on things but have finished work for today, now.

Ept never was a qemu parameter, only ever vmx-ept which is intel only.

Also invtsc is an almost obsolete 486 instruction

https://github.com/search?q=repo%3Aqemu%2Fqemu%20invtsc&type=code

which has been replaced by rdtscp even from haswell onwards.

https://www.amd.com/content/dam/amd/en/documents/processor-tech-docs/programmer-references/40332.pdf

Page 422

"The TSC is a model-specific register that can also be read using one of the special read time-stamp
counter instructions, RDTSC (Read Time-Stamp Counter) or RDTSCP (Read Time-Stamp Counter
and Processor ID)"

I haven't yet tried building but I am imaging that tsc does probably account for the hangs.

SSSE3 > Only the super set of 'supplementary sse3' exists.
sse3 oesn't exist as a qemu parameter.

sse4.1 should also be included for backward compatibility.

change invtsc to rdtscp

drop non-existent vmware parameters

qemu-system-x86_64 -cpu help | grep -A 100 flags | tr ' ' '\n' | grep vmware is null

There are a lot of instructions yet to add, possibly. Also PDPE1GB is a skylake flag. I was looking at using skylake rather than haswell last week ...

Skylake is officially Sonoma supported. Haswell isn't.

@flexiondotorg you seem to be working Pacific Coast hours at present ... 🤣

https://github.com/search?q=repo%3Aqemu%2Fqemu+path%3A%2F%5Etarget%5C%2Fi386%5C%2F%2F++skylake&type=code

[TuxVinyards]

As an update, this morning, I added in all the non intel, non VMX flags that Qemu list as available.

I tried Skylake but couldn't get it to boot. No idea on that. Same as in #1114 It would only use Haswell ...

The Haswell boot was more successful. On my first attempt it flew past the one minute point but then failed the rest. On my second try, with more flags enabled by that point, it only needed one reboot and install restart but then stuck at the one minute.

Of further note. Penryn doesn't have either FMA or RDTSCP and is in fact very limited.

{

  .name = "Penryn",
    .level = 10,
    .vendor = CPUID_VENDOR_INTEL,
    .family = 6,
    .model = 23,
    .stepping = 3,
    .features[FEAT_1_EDX] =
        CPUID_VME | CPUID_SSE2 | CPUID_SSE | CPUID_FXSR | CPUID_MMX |
        CPUID_CLFLUSH | CPUID_PSE36 | CPUID_PAT | CPUID_CMOV | CPUID_MCA |
        CPUID_PGE | CPUID_MTRR | CPUID_SEP | CPUID_APIC | CPUID_CX8 |
        CPUID_MCE | CPUID_PAE | CPUID_MSR | CPUID_TSC | CPUID_PSE |
        CPUID_DE | CPUID_FP87,
    .features[FEAT_1_ECX] =
        CPUID_EXT_SSE41 | CPUID_EXT_CX16 | CPUID_EXT_SSSE3 |
        CPUID_EXT_SSE3,
    .features[FEAT_8000_0001_EDX] =
        CPUID_EXT2_LM | CPUID_EXT2_NX | CPUID_EXT2_SYSCALL,
    .features[FEAT_8000_0001_ECX] =
        CPUID_EXT3_LAHF_LM,
    .features[FEAT_VMX_BASIC] = MSR_VMX_BASIC_INS_OUTS,
    .features[FEAT_VMX_ENTRY_CTLS] = VMX_VM_ENTRY_IA32E_MODE |
         VMX_VM_ENTRY_LOAD_IA32_PERF_GLOBAL_CTRL,
    .features[FEAT_VMX_EXIT_CTLS] = VMX_VM_EXIT_ACK_INTR_ON_EXIT |
         VMX_VM_EXIT_LOAD_IA32_PERF_GLOBAL_CTRL,
    .features[FEAT_VMX_MISC] = MSR_VMX_MISC_ACTIVITY_HLT,
    .features[FEAT_VMX_PINBASED_CTLS] = VMX_PIN_BASED_EXT_INTR_MASK |
         VMX_PIN_BASED_NMI_EXITING | VMX_PIN_BASED_VIRTUAL_NMIS,
    .features[FEAT_VMX_PROCBASED_CTLS] = VMX_CPU_BASED_VIRTUAL_INTR_PENDING |
         VMX_CPU_BASED_USE_TSC_OFFSETING | VMX_CPU_BASED_HLT_EXITING |
         VMX_CPU_BASED_INVLPG_EXITING | VMX_CPU_BASED_MWAIT_EXITING |
         VMX_CPU_BASED_RDPMC_EXITING | VMX_CPU_BASED_RDTSC_EXITING |
         VMX_CPU_BASED_CR8_LOAD_EXITING | VMX_CPU_BASED_CR8_STORE_EXITING |
         VMX_CPU_BASED_TPR_SHADOW | VMX_CPU_BASED_MOV_DR_EXITING |
         VMX_CPU_BASED_UNCOND_IO_EXITING | VMX_CPU_BASED_USE_IO_BITMAPS |
         VMX_CPU_BASED_MONITOR_EXITING | VMX_CPU_BASED_PAUSE_EXITING |
         VMX_CPU_BASED_VIRTUAL_NMI_PENDING | VMX_CPU_BASED_USE_MSR_BITMAPS |
         VMX_CPU_BASED_ACTIVATE_SECONDARY_CONTROLS,
    .features[FEAT_VMX_SECONDARY_CTLS] =
         VMX_SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES |
         VMX_SECONDARY_EXEC_WBINVD_EXITING,
    .xlevel = 0x80000008,
    .model_id = "Intel Core 2 Duo P9xxx (Penryn Class Core 2)",
},

This is how my current state of play looks:

            # TODO: Investigate if hosts with an Intel CPU can just use `-cpu host` or VMX intel extensions should be enabled
            case ${macos_release} in
                ventura|sonoma)
                    if check_cpu_flag sse4_2 && check_cpu_flag avx2; then
                        CPU="-cpu Haswell-v4,kvm=on,vendor=GenuineIntel,+avx,+avx2,+sse,+sse2,+ssse3,+sse4.1,+sse4.2,rdtscp"
                    else
                        echo "ERROR! macOS ${macos_release} requires a CPU with SSE 4.2 and AVX2 support."
                        exit 1
                    fi;;
                catalina|big-sur|monterey)
                    if check_cpu_flag sse4_2; then
                        CPU="-cpu Haswell-v4,kvm=on,vendor=GenuineIntel,+avx,+sse,+sse2,+ssse3,+sse4.1,+sse4.2,rdtscp"
                    else
                        echo "ERROR! macOS ${macos_release} requires a CPU with SSE 4.2 support."
                        exit 1
                    fi;;
                *)
                    if check_cpu_flag sse4_1; then
                        CPU="-cpu Penryn,kvm=on,vendor=GenuineIntel,+avx,+sse,+sse2,+ssse3,+sse4.1,tsc"
                    else
                        echo "ERROR! macOS ${macos_release} requires a CPU with SSE 4.1 support."
                        exit 1
                    fi;;
            esac

            local FLAG=""

            if [[ $CPU == *'Penryn'* ]]; then
                for FLAG in fxsr mmx clflush pse36 pat cmov mca pge mtrr sep apic cx8 \
                            mce pae msr tsc pse de fp87 cx16 lm nx syscall ; do
                    if check_cpu_flag "${FLAG}"; then
                        CPU+=",+${FLAG}"
                    fi
                done
                if check_cpu_flag lahf_lm ; then CPU+=",+lahf-lm" ; fi
            else
                if ! check_cpu_flag fma && ! check_cpu_flag rdtscp; then
                    echo "ERROR! macOS requires a CPU with FMA and TSC support."
                    exit 1
                fi
                for FLAG in abm adx aes amd-ssbd bmi1 bmi2 cx8 eist f16c fma \
                            mmx movbe mpx popcnt smep vaes vbmi2 vpclmulqdq \
                            fxsr clflush pse36 pat cmov mca pge mtrr sep apic cx8 \
                            mce pae msr tsc pse de fp87 x2apic cx16 \
                            pcid f16c rdrand lm nx syscall \
                            fsgsbase hle erms invpcid rtm arat \
                            xgetbv1 xsave xsaveopt; do
                    if check_cpu_flag "${FLAG}"; then
                        CPU+=",+${FLAG}"
                    fi
                done
                if check_cpu_flag tsc_adjust ; then CPU+=",+tsc-adjust" ; fi
                if check_cpu_flag lahf_lm ; then CPU+=",+lahf-lm" ; fi
            fi

[flexiondotorg]

FMA is required for Metal.

[TuxVinyards]

FMA is required for Metal.

If the flags don't exist, wishful thinking won't make them appear.

[TuxVinyards]

And closing complicated issues as 'completed' when they are clearly not will solve issues either ...

[flexiondotorg]

I have tested every macOS release supported by Quickemu and they all work without any issue. And...

[TuxVinyards]

Hi @flexiondotorg and thanks for replying back. Not closing this issue, in other words ....

I sympathise with the concept of "If it's not broken, don't fix it" but it's broken for me.

I would be interested to see the lspcu's of your test machines. Your 'solution' is still not working properly for me, on the Series 11 Intel that I am using.

I tested again this morning using your 'macos-host' branch to see if it possessed some magic properties that perhaps circumvented an error in Qemu source code. No such.

There was one other new thing I did notice though. You have an error in your FMA INVTSC test. It should be || not && if want both to be present. If you 'fix' this to be as you want things, however, I think you find things failing for at least 90% of people. As in the earlier details above ....

I remain unhappy with my solution but apart from your errored test that I copied over it is technically correct. I would like to get it to running. As in full install, right up to the bit where you get asked for name, location and preferences. And without restarts and hangs.

I think it is only luck, not judgement, that is making your solution "work" on your machines. But I would like to know why it is working and what it is about your machines that makes that happen.

The only reason that the ept_1gb flag fixes things, is that this flag doesn't exist, so lscpu can't find it and the non-existent 'ept-1gb` never gets added:

qemu-system-x86_64 -cpu help | grep -A 100 flags | tr ' ' '\n' | grep ept | cat -s 

I am not sure why FMA is critical. Do you have a reference? If this is the case, then Penryn needs to be removed.

[TuxVinyards]

SUCCESS 🥳

            case ${macos_release} in
                ventura|sonoma)
                    if check_cpu_flag sse4_2 && check_cpu_flag avx2; then
                        CPU="-cpu Haswell-v2,kvm=on,vendor=GenuineIntel,+avx,+avx2,+sse,+sse2,+ssse3,+sse4.1,+sse4.2,+rdtscp"
                        #CPU="-cpu Skylake-Server-v3,kvm=on,vendor=GenuineIntel,+avx,+avx2,+sse,+sse2,+ssse3,+sse4.1,+sse4.2,+rdtscp"
                        #if check_cpu_flag pdpe1gb ; then CPU+=",+pdpe1gb" ; fi
                        #if check_cpu_flag smap ; then CPU+=",+smap" ; fi
                        #if check_cpu_flag clwb ; then CPU+=",+clwb" ; fi
                        #if check_cpu_flag adx ; then CPU+=",+adx" ; fi
                    else
                        echo "ERROR! macOS ${macos_release} requires a CPU with SSE 4.2 and AVX2 support."
                        exit 1
                    fi ;;
                catalina|big-sur|monterey)
                    if check_cpu_flag sse4_2; then
                        CPU="-cpu Haswell-v2,kvm=on,vendor=GenuineIntel,+avx,+sse,+sse2,+ssse3,+sse4.1,+sse4.2,+rdtscp"
                    else
                        echo "ERROR! macOS ${macos_release} requires a CPU with SSE 4.2 support."
                        exit 1
                    fi ;;
                *)
                    if check_cpu_flag sse4_1; then
                        CPU="-cpu Penryn,kvm=on,vendor=GenuineIntel,+sse,+sse2,+ssse3,+sse4.1"
                    else
                        echo "ERROR! macOS ${macos_release} requires a CPU with SSE 4.1 support."
                        exit 1
                    fi ;;
            esac

            local FLAG=""

            if [[ $CPU == *'Penryn'* ]]; then
                for FLAG in tsc vme fxsr mmx clflush pse36 pat cmov mca pge mtrr sep apic cx8 \
                            mce pae msr pse de fp87 cx16 lm nx syscall ; do
                    if check_cpu_flag "${FLAG}"; then CPU+=",+${FLAG}" ; fi
                done
                if check_cpu_flag lahf_lm ; then CPU+=",+lahf-lm" ; fi
            else
                if ! check_cpu_flag fma || ! check_cpu_flag rdtscp; then
                    echo "ERROR! macOS requires a CPU with FMA and TSC support."
                    exit 1
                fi
                for FLAG in vmx abm aes bmi1 bmi2 cx8 eist f16c fma \
                            mmx movbe mpx popcnt smep vaes vbmi2 vpclmulqdq \
                            fxsr clflush pse36 pat cmov mca pge mtrr sep apic cx8 \
                            mce pae msr tsc pse de fp87 x2apic cx16 \
                            pcid f16c rdrand lm nx syscall fsgsbase erms invpcid arat \
                            xgetbv1 xsave xsaveopt ; do
                    if check_cpu_flag "${FLAG}"; then CPU+=",+${FLAG}" ; fi
                done
                if check_cpu_flag tsc_adjust ; then CPU+=",+tsc-adjust" ; fi
                if check_cpu_flag lahf_lm ; then CPU+=",+lahf-lm" ; fi
                if check_cpu_flag amd_ssbd ; then CPU+=",+amd-ssbd" ; fi
            fi

Tested with FULL complete installs of Sonoma, Catalina and Mojave. Full setup with login and preferences set. Shutdown and re-login as working.

Much faster than previously with 4.9.2 through 4.9.4 etc. Fully working, no hangs, no stalls.

Of new, now requires the installer icon to be pressed on the restarts. This now happens a couple of times. But this all flows smoothly.

CAVEAT: not yet tested on Ryzen. Volunteers please ...

Additionally note the Skylake version is also working. This is commented out but has been tested on Sonoma. cpu_cores="8" should be placed in the .conf before running.

A conf selector for skylake should be considered, or at least noted and archived in case of future problems.

[tdcolvin]

This worked for me with Sonoma

[dhobsd]

Worked for me as well, though I'd note that I was also able to just use "-cpu host". (I saw there was a TODO about that.)