Skip to content
This repository was archived by the owner on Jan 28, 2023. It is now read-only.

Cannot resume after STI when debugging with HAX_DEBUG_STEP enabled #193

Open
leecher1337 opened this issue Mar 10, 2019 · 8 comments
Open

Cannot resume after STI when debugging with HAX_DEBUG_STEP enabled #193

leecher1337 opened this issue Mar 10, 2019 · 8 comments

Comments

@leecher1337
Copy link
Contributor

Describe the Bug

Summary: When enabling single step debugging (HAX_DEBUG_ENABLE | HAX_DEBUG_STEP), the emulator doesn't continue after STI instruction, but crashes with VMX_EXIT_FAILED_VMENTER_GS.

Host Environment

  • HAXM version: c072ad9
  • Host OS version: Windows 10 pro 1803
  • Host OS architecture: x86_64
  • Host CPU model: Xeon E31235
  • Host RAM size: 12 GB

To Reproduce

__declspec (naked) static void the_code(void)
{
__asm
{
inc ax // 0
cli // 2
inc ax // 3
sti // 5
dec ax // 6
dec ax // 8
int 3 // A
}
}

haxdbg.control = HAX_DEBUG_ENABLE | HAX_DEBUG_STEP;
DeviceIoControl(hVCPU, HAX_IOCTL_VCPU_DEBUG, &haxdbg, sizeof(haxdbg), NULL, 0, &bytes, NULL);

After executing the STI instruction, the next instruction cannot be executed and emulator refuses to resume with VMX_EXIT_FAILED_VMENTER_GS.

exitStatus = 0000000B @ 2
HAX_EXIT_DEBUG
exitStatus = 0000000B @ 3
HAX_EXIT_DEBUG
exitStatus = 0000000B @ 5
HAX_EXIT_DEBUG
exitStatus = 0000000B @ 6
HAX_EXIT_DEBUG
exitStatus = 00000007 @ 6
HAX_EXIT_STATECHANGE
VCPU shutdown request, cs=0040, ip=6

Expected Behavior

Expectation: Emulator should execute next instruction.

Reproducibility

100%

Diagnostic Information

HAXM log:
haxm_debug:vcpu begin to run....
haxm_debug:vcpu begin to run....in PE
haxm_debug:vm entry!
haxm_debug:fs 53
haxm_debug:....exit_reason.raw 80000021, cpu 0 0
haxm_debug:vmread_cr cr0 30, cr0_mask e0000020, state->_cr0 10
haxm_debug:vmread_cr, state->_cr0 10
haxm_panic:vcpu->tr:8b
haxm_warning:4000 VMX_PIN_CONTROLS: 1f
haxm_warning:4002 VMX_PRIMARY_PROCESSOR_CONTROLS: 969861fa
haxm_warning:401e VMX_SECONDARY_PROCESSOR_CONTROLS: aa
haxm_warning:4004 VMX_EXCEPTION_BITMAP: 40002
haxm_warning:4006 VMX_PAGE_FAULT_ERROR_CODE_MASK: 0
haxm_warning:4008 VMX_PAGE_FAULT_ERROR_CODE_MATCH: 0
haxm_warning:400c VMX_EXIT_CONTROLS: 236fff
haxm_warning:400e VMX_EXIT_MSR_STORE_COUNT: 0
haxm_warning:4010 VMX_EXIT_MSR_LOAD_COUNT: 0
haxm_warning:4012 VMX_ENTRY_CONTROLS: 11ff
haxm_warning:4014 VMX_ENTRY_MSR_LOAD_COUNT: 0
haxm_warning:4016 VMX_ENTRY_INTERRUPT_INFO: 0
haxm_warning:4018 VMX_ENTRY_EXCEPTION_ERROR_CODE: 0
haxm_warning:401a VMX_ENTRY_INSTRUCTION_LENGTH: 0
haxm_warning:401c VMX_TPR_THRESHOLD: 0
haxm_warning:6000 VMX_CR0_MASK: ffffffffe0000020
haxm_warning:6002 VMX_CR4_MASK: fffffffffff9f860
haxm_warning:6004 VMX_CR0_READ_SHADOW: 0
haxm_warning:6006 VMX_CR4_READ_SHADOW: 2000
haxm_warning:400a VMX_CR3_TARGET_COUNT: 0
haxm_warning:6008 VMX_CR3_TARGET_VAL_BASE: 0
haxm_warning:0000 VMX_VPID: 1
haxm_warning:2000 VMX_IO_BITMAP_A: a6fe7000
haxm_warning:2002 VMX_IO_BITMAP_B: a6daa000
haxm_warning:2004 VMX_MSR_BITMAP: a6da9000
haxm_warning:2006 VMX_EXIT_MSR_STORE_ADDRESS: 0
haxm_warning:2008 VMX_EXIT_MSR_LOAD_ADDRESS: 0
haxm_warning:200a VMX_ENTRY_MSR_LOAD_ADDRESS: 0
haxm_warning:2010 VMX_TSC_OFFSET: fffd5ed7883379b3
haxm_warning:2012 VMX_VAPIC_PAGE: 0
haxm_warning:2014 VMX_APIC_ACCESS_PAGE: 0
haxm_warning:201a VMX_EPTP: 9bf7701e
haxm_warning:482e VMX_PREEMPTION_TIMER: 0
haxm_warning:4400 VMX_INSTRUCTION_ERROR_CODE: c
haxm_warning:4402 VM_EXIT_INFO_REASON: 80000021
haxm_warning:4404 VM_EXIT_INFO_INTERRUPT_INFO: 80000301
haxm_warning:4406 VM_EXIT_INFO_EXCEPTION_ERROR_CODE: 0
haxm_warning:4408 VM_EXIT_INFO_IDT_VECTORING: 0
haxm_warning:440a VM_EXIT_INFO_IDT_VECTORING_ERROR_CODE: 0
haxm_warning:440c VM_EXIT_INFO_INSTRUCTION_LENGTH: 0
haxm_warning:440e VM_EXIT_INFO_INSTRUCTION_INFO: 0
haxm_warning:6400 VM_EXIT_INFO_QUALIFICATION: 0
haxm_warning:6402 VM_EXIT_INFO_IO_ECX: 0
haxm_warning:6404 VM_EXIT_INFO_IO_ESI: 0
haxm_warning:6406 VM_EXIT_INFO_IO_EDI: 0
haxm_warning:6408 VM_EXIT_INFO_IO_EIP: 0
haxm_warning:640a VM_EXIT_INFO_GUEST_LINEAR_ADDRESS: 0
haxm_warning:2400 VM_EXIT_INFO_GUEST_PHYSICAL_ADDRESS: 0
haxm_warning:6c16 HOST_RIP: fffff8002d1a1a63
haxm_warning:6c14 HOST_RSP: ffffa709c7f10520
haxm_warning:6c00 HOST_CR0: 80050031
haxm_warning:6c02 HOST_CR3: 6500000
haxm_warning:6c04 HOST_CR4: 426f8
haxm_warning:0c02 HOST_CS_SELECTOR: 10
haxm_warning:0c06 HOST_DS_SELECTOR: 28
haxm_warning:0c00 HOST_ES_SELECTOR: 28
haxm_warning:0c08 HOST_FS_SELECTOR: 0
haxm_warning:0c0a HOST_GS_SELECTOR: 0
haxm_warning:0c04 HOST_SS_SELECTOR: 18
haxm_warning:0c0c HOST_TR_SELECTOR: 40
haxm_warning:6c06 HOST_FS_BASE: 9eb000
haxm_warning:6c08 HOST_GS_BASE: fffff8002da79000
haxm_warning:6c0a HOST_TR_BASE: fffff8002fe53000
haxm_warning:6c0c HOST_GDTR_BASE: fffff8002fe54fb0
haxm_warning:6c0e HOST_IDTR_BASE: fffff8002fe52000
haxm_warning:4c00 HOST_SYSENTER_CS: 0
haxm_warning:6c10 HOST_SYSENTER_ESP: 0
haxm_warning:6c12 HOST_SYSENTER_EIP: 0
haxm_warning:681e GUEST_RIP: 6
haxm_warning:6820 GUEST_RFLAGS: 302
haxm_warning:681c GUEST_RSP: 0
haxm_warning:6800 GUEST_CR0: 30
haxm_warning:6802 GUEST_CR3: 0
haxm_warning:6804 GUEST_CR4: 2040
haxm_warning:0800 GUEST_ES_SELECTOR: 0
haxm_warning:0802 GUEST_CS_SELECTOR: 40
haxm_warning:0804 GUEST_SS_SELECTOR: 0
haxm_warning:0806 GUEST_DS_SELECTOR: 0
haxm_warning:0808 GUEST_FS_SELECTOR: 0
haxm_warning:080a GUEST_GS_SELECTOR: 0
haxm_warning:080c GUEST_LDTR_SELECTOR: 0
haxm_warning:080e GUEST_TR_SELECTOR: 0
haxm_warning:4814 GUEST_ES_AR: 93
haxm_warning:4816 GUEST_CS_AR: 9b
haxm_warning:4818 GUEST_SS_AR: 93
haxm_warning:481a GUEST_DS_AR: 93
haxm_warning:481c GUEST_FS_AR: 93
haxm_warning:481e GUEST_GS_AR: 93
haxm_warning:4820 GUEST_LDTR_AR: 10000
haxm_warning:4822 GUEST_TR_AR: 8b
haxm_warning:6806 GUEST_ES_BASE: 0
haxm_warning:6808 GUEST_CS_BASE: 400
haxm_warning:680a GUEST_SS_BASE: 0
haxm_warning:680c GUEST_DS_BASE: 0
haxm_warning:680e GUEST_FS_BASE: 0
haxm_warning:6810 GUEST_GS_BASE: 0
haxm_warning:6812 GUEST_LDTR_BASE: 0
haxm_warning:6814 GUEST_TR_BASE: 0
haxm_warning:6816 GUEST_GDTR_BASE: 0
haxm_warning:6818 GUEST_IDTR_BASE: 0
haxm_warning:4800 GUEST_ES_LIMIT: ffff
haxm_warning:4802 GUEST_CS_LIMIT: ffff
haxm_warning:4804 GUEST_SS_LIMIT: ffff
haxm_warning:4806 GUEST_DS_LIMIT: ffff
haxm_warning:4808 GUEST_FS_LIMIT: ffff
haxm_warning:480a GUEST_GS_LIMIT: ffff
haxm_warning:480c GUEST_LDTR_LIMIT: 0
haxm_warning:480e GUEST_TR_LIMIT: 0
haxm_warning:4810 GUEST_GDTR_LIMIT: 0
haxm_warning:4812 GUEST_IDTR_LIMIT: 0
haxm_warning:2800 GUEST_VMCS_LINK_PTR: ffffffffffffffff
haxm_warning:2802 GUEST_DEBUGCTL: 0
haxm_warning:2804 GUEST_PAT: 0
haxm_warning:2806 GUEST_EFER: 0
haxm_warning:2808 GUEST_PERF_GLOBAL_CTRL: 0
haxm_warning:280a GUEST_PDPTE0: 0
haxm_warning:280c GUEST_PDPTE1: 0
haxm_warning:280e GUEST_PDPTE2: 0
haxm_warning:2810 GUEST_PDPTE3: 0
haxm_warning:681a GUEST_DR7: 400
haxm_warning:6822 GUEST_PENDING_DBE: 0
haxm_warning:482a GUEST_SYSENTER_CS: 0
haxm_warning:6824 GUEST_SYSENTER_ESP: 0
haxm_warning:6826 GUEST_SYSENTER_EIP: 0
haxm_warning:4828 GUEST_SMBASE: 0
haxm_warning:4824 GUEST_INTERRUPTIBILITY: 1
haxm_warning:4826 GUEST_ACTIVITY_STATE: 0
haxm_error:vcpu has panicked, id:0
haxm_error:log_host_cr4_vmxe: 0
haxm_error:log_host_cr4 0
haxm_error:log_vmxon_res 0
haxm_error:log_vmxon_addr a6da8000
haxm_error:log_vmxon_err_type1 0
haxm_error:log_vmxon_err_type2 0
haxm_error:log_vmxon_err_type3 0
haxm_error:log_vmclear_err 0
haxm_error:log_vmptrld_err 0
haxm_error:log_vmoff_no 0
haxm_error:log_vmxoff_res 0
haxm_error:vcpu has panicked, id:0
haxm_error:log_host_cr4_vmxe: 0
haxm_error:log_host_cr4 0
haxm_error:log_vmxon_res 0
haxm_error:log_vmxon_addr a6da8000
haxm_error:log_vmxon_err_type1 0
haxm_error:log_vmxon_err_type2 0
haxm_error:log_vmxon_err_type3 0
haxm_error:log_vmclear_err 0
haxm_error:log_vmptrld_err 0
haxm_error:log_vmoff_no 0
haxm_error:log_vmxoff_res 0
haxm_info:HaxClose device 3 at process FFFF818C26E2E440
haxm_info:vcpu_vpid_free: Clearing bit: 0x0, vpid_seed: 0x1. vcpu_id: 0, vm_id: 0.
haxm_info:vcpu 0 is teardown.
haxm_info:HaxClose device 2 at process FFFF818C26E2E440
haxm_info:Close VM 0
haxm_info:ept_tree_free: Total 4 EPT page(s) freed
haxm_debug:ramblock_deref: Reset RAM block (FFFF818C24C16040): base_uva = 0xbc0000, size = 0x10f000, ref_count = 0
haxm_info:ramblock_free_chunks: Freeing <= 1 chunks, bitmap:
haxm_info:ramblock_free_chunks: [0]=0x01
haxm_info:ramblock_free_chunks: [1]=0x00
haxm_info:ramblock_free_chunks: [2]=0x00
haxm_info:ramblock_free_chunks: [3]=0x00
haxm_info:ramblock_free_chunks: [4]=0x00
haxm_info:ramblock_free_chunks: [5]=0x00
haxm_info:ramblock_free_chunks: [6]=0x00
haxm_info:ramblock_free_chunks: [7]=0x00
haxm_info:ramblock_free_chunks: [8]=0x00
haxm_info:ramblock_free_chunks: All chunks freed: 1084KB total, 1084KB used
haxm_info:ramblock_free_list
haxm_info:ramblock_free_list: Freeing RAM block: uva: 0xbc0000, size: 0x10f000, ref_count: 0
haxm_info:ramblock_free_chunks: Freeing <= 1 chunks, bitmap:
haxm_info:ramblock_free_chunks: [0]=0x00
haxm_info:ramblock_free_chunks: [1]=0x00
haxm_info:ramblock_free_chunks: [2]=0x00
haxm_info:ramblock_free_chunks: [3]=0x00
haxm_info:ramblock_free_chunks: [4]=0x00
haxm_info:ramblock_free_chunks: [5]=0x00
haxm_info:ramblock_free_chunks: [6]=0x00
haxm_info:ramblock_free_chunks: [7]=0x00
haxm_info:ramblock_free_chunks: [8]=0x00
haxm_info:ramblock_free_chunks: All chunks freed: 1084KB total, 0KB used
haxm_error:
...........hax_teardown_vm
haxm_info:HaxClose device 1 at process FFFF818C26E2E440

Additional context

Maybe I don't understand the debug interface correctly and there is something I need to take care of which I don't?
@raphaelning
Copy link
Contributor

Thanks for the detailed report. I'm glad you're already using the guest debugging API, which is quite new and may have limitations.

VMX_EXIT_FAILED_VMENTER_GS is basic VM exit reason 33:

VM-entry failure due to invalid guest state. A VM entry failed one of the checks identified in Section 26.3.1.

With some patience, one should be able to manually go over the checks specified in Intel SDM Vol. 3C, Section 26.3.1 (Checks on the Guest State Area), and find out which GUEST_xxx value in the VMCS dump above is invalid. Unfortunately, I haven't got the time to do that myself...

I wonder if you get the same error when you run the same guest code outside single-stepping mode. If not, maybe you can try to dump the VMCS before and after the STI instruction, and then compare the dumps with the one above. I think you just need to manually trigger a fault-like VM exit that doesn't alter the guest state, and one way to do that (maybe not the best) is to insert a CPUID instruction into the guest code as well as a dump_vmcs() call into exit_cpuid(). Could you try that?

It would also help if we could reproduce the issue on our side. Is it possible to reproduce it with QEMU (after applying the necessary QEMU-side patches)?

@leecher1337
Copy link
Contributor Author

Hi,

Thank you for taking a look!
Here are the VMCS that changed on execution:

4402 VM_EXIT_INFO_REASON: a --> 80000021
4404 VM_EXIT_INFO_INTERRUPT_INFO: 0 --> 80000301 (That is Type EXCEPTION, VECTOR_DB?)
440c VM_EXIT_INFO_INSTRUCTION_LENGTH: 2 --> 0
681e GUEST_RIP: 5 --> 8
6820 GUEST_RFLAGS: 102 --> 302
4824 GUEST_INTERRUPTIBILITY: 0 --> 1

The code runs through properly when Debugging is disabled.
I haven't found violations to the Checks in Intel SDM yet, but I'll have a look at it again. Maybe the VM_EXIT_INFO_INTERRUPT_INFO is a hint?
Would it be enough for you if I just give you a simple test application (it's just the forked haxm-test repository where I placed my code in it for testing such things as this bug)?

@leecher1337
Copy link
Contributor Author

26.3.1.5
The following checks are performed if any of the following holds:
(1) the interruptibility-state field indicates blocking by STI (bit 0 in that field is 1); (2) the interruptibility-state field indicates blocking by MOV SS (bit 1 in that field is 1); or (3) the activity-state field indicates HLT:
• Bit 14 (BS) must be 1 if the TF flag (bit 8) in the RFLAGS field is 1 and the BTF flag (bit 1) in the IA32_DEBUGCTL field is 0.
• Bit 14 (BS) must be 0 if the TF flag (bit 8) in the RFLAGS field is 0 or the BTF flag (bit 1) in the IA32_DEBUGCTL field is 1

@leecher1337
Copy link
Contributor Author

From KVM sourcecode:

		/* When single-stepping over STI and MOV SS, we must clear the
		 * corresponding interruptibility bits in the guest state. Otherwise
		 * vmentry fails as it then expects bit 14 (BS) in pending debug
		 * exceptions being set, but that's not correct for the guest debugging
		 * case. */
		state._interruptibility_state.raw &= ~3;

haxm does this in advance_rip:

static void advance_rip(struct vcpu_t *vcpu)
{
struct vcpu_state_t *state = vcpu->state;
uint32_t interruptibility = vmx(vcpu, interruptibility_state).raw;

if (interruptibility & 3u) {
    interruptibility &= ~3u;
    vmx(vcpu, interruptibility_state).raw = interruptibility;
    vcpu->interruptibility_dirty = 1;
}

state->_rip += vmx(vcpu, exit_instr_length);
vcpu->rip_dirty = 1;

}

When Debug vector is called, there is no advance_rip call. The debugger also cannot do it, as HAX_VCPU_SET_REGS doesn't update interruptibility_state.

@leecher1337
Copy link
Contributor Author

Here is a patch proposal to fix this bug:
leecher1337@06ab1fe

@raphaelning
Copy link
Contributor

Thanks! Apparently, we overlooked the part of the Intel SDM about "pending debug exceptions" when we added guest debugging support. I'm still not quite sure why VT-x had to introduce "pending debug exceptions" in the first place--how could STI interfere with single-stepping (#DB exception) anyway? Also, why does KVM consider the other solution, which is to set bit 14 in GUEST_PENDING_DBE, incorrect?

@raphaelning
Copy link
Contributor

Would it be enough for you if I just give you a simple test application (it's just the forked haxm-test repository where I placed my code in it for testing such things as this bug)?

Sorry for the delay. Could you point me to this haxm-test repository? It sounds interesting. And yes, I'd like to try your test application.

@leecher1337
Copy link
Contributor Author

Hi,

Here you are, just forked it form Nukem9 and added some test code, as I was too lazy to write my own test application:
https://github.com/leecher1337/Haxm-1/tree/master/Project/HaxmTest

I first tried the other way round to set bit 14, it entered the VM, but I was stuck at the same instruction then and EIP didn't advance.
Maybe setting these bits has to do with the gap of one instruction after STI, until its results become effective (see SDM for details).

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@raphaelning @leecher1337