Dirty page logging

[JayFoxRox]

(Feature request)

KVM allows dirty page logging through the KVM_GET_DIRTY_LOG vm ioctl.
See https://www.kernel.org/doc/Documentation/virtual/kvm/api.txt for details.
To my knowledge this is currently not implemented in HAXM.

Reasoning

A similar feature for HAXM would allow speedup in emulation of certain systems.

If the guest system is sharing memory between the GPU and CPU, the emulation host could know which resources used in the GPU have to be updated by inspecting the dirty page log.
Without such logging, the emulator or virtualization environment has to rely on either guessing (which might break graphics), hashing (which is considerable amount of additional work for the host) or more invasive techniques such as using the guest pagetable (which depends on knowledge about the guest code).

[raphaelning]

You are right. This feature is in our plans, and I'd like to work on it soon.

[mborgerson]

Hi @raphaelning, are there any updates on this feature request since last year? I'm willing to help implement parts of this (time permitting), but I don't want to interfere if there is existing work in this area and it is planned.

I have some thoughts about how to go about implementing this feature. Based on a cursory inspection of how KVM accomplishes this, generally speaking there are two methods:

• The first is a less efficient, but more compatible method in which we write-protect a range of memory, intercept writes through page faults and set a bit corresponding to the page of the GPA in a dirty bitmap, then either let the write go through and quickly return control to the VM, or handle the page-fault if it is actually a fault. User code can then fetch and clear this dirty bitmap through a new IOCTL (which QEMU code would do through the hax_log_sync function in hax-mem.c).
• The second is a more efficient method leveraging a feature added in Broadwell timeframe called Page Modification Logging (PML) which enables hardware-assisted logging and removes the need to write-protect the region and handle all of the page faults. ~4%-5% performance improvement over WP according to the KVM patchset. Fetch and clear works the same as before.

I think we will probably want to have support for both methods in HAXM, but perhaps we can start with the first one to support older generations of processors.

Thanks!

[raphaelning]

I'm willing to help implement parts of this (time permitting), but I don't want to interfere if there is existing work in this area and it is planned.

Thanks in advance! We've been prioritizing other work over this feature, so I'm really glad that you have offered to help :-) Speaking of potential conflicts, I think the first method may interfere a little bit with the GPA protection API which we want to improve in Q4.

Earlier this year, we merged 0a40260 to enable protection of a GPA range, and the only protection modes implemented are "full" (no access) and "none" (RWX access). This PROTECT_RAM ioctl is now used by Android Emulator (but not by QEMU) to implement lazy RAM snapshot loading. The semantics of this protection is that any access violation on a protected GPA is forwarded to QEMU, where there's a user-mode page fault handler. In Q4, we'd like to extend that ioctl to enable fine-grained protection (e.g. write-protect) and fix some of the known issues with that patch (as described in the commit log).

If both userfault and dirty page tracking features are based on write-protecting a GPA range, there must be some overlap between the two tasks. My advice would be to put the first method on standby until after we are done with the userfault API enhancement--hopefully by that time the first method will be easier to implement. Admittedly, the second method does not benefit pre-2015 CPUs, but I hope it's still useful for your purpose.

In addition, I'm working on a patch set to enhance the VMX feature detection API for HAXM. So if you decide to go for the second method, please don't worry about detecting host support for the EPT Accessed/Dirty bits; you can just assume PML is usable.

[mborgerson]

@raphaelning When I wrote my comment above, I wasn't aware of the PROTECT_RAM ioctl. After some thought, I agree that it is a much better way to go! This ioctl will not only enable dirty page tracking, but will also provide a lot more flexibility to QEMU to intercept reads as well and handle them accordingly.