Time Drifting Issues in Virtualized Environment

When running a time-critical service in a virtualized environment, sometimes you may discover that the time is skipping a few seconds. This may happen because the virtual machine don't have access to the machine's physical clock IC and therefore it has to calculate the clock based on unreliable sources. The clock drifts even more when the system is under high load.

Some hypervisors recognize this, and when the clock drift exceeds a given amount, they may synchronize the clock in the guest VM. For example Hyper-V resyncs the clock when the drift exceeds 5 seconds. From the application's point of view, it seems that the world has stopped for 5 seconds.

When the application is time-critical, like a distributed database, this may lead to unexpected failover-like situations. Specifically with ScalienDB this may lead to Controller Master lease loss or Shard Server Primary lease loss. If you have other healthy servers in the cluster, they will become elected as Master or Primary, but until the lease is not expired, the system cannot accept schema modifications or write operations. These intervals are chosen in the seconds range (usually 5-7 secs), so the system will not be affected severely because the client library handles these cases gratefully.

Still, if you want to avoid such issues, we recommend to use some form of Time Synchronization service on your virtualized servers, and readjust the time frequently, like in every 15 minutes. The rest of this document is a guide how to do that in the case of Hyper-V virtualization.

Time Synchronization Settings in Hyper-V Windows Guests

First you need an administrator console. Start the Windows Time Service if it is not started already:

net start w32time

Copy the following registry entries to a file and set them in the registry.

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\W32Time\TimeProviders\NtpClient]
"SpecialPollInterval"=dword:000384

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\W32Time\Parameters]
"NtpServer"="time.windows.com,0x1"

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\W32Time\Config]
"UpdateInterval"=dword:0003e8
"MinPollInterval"=dword:00000008

These settings change the frequency of time syncronization. You can learn more about these settings here.

Next, update the time from the given timeserver. Right-click on the clock on the Taskbar, and choose "Adjust date/time". Go to "Internet Time" - "Change settings..." and press "Update now". After that check if the time is set correctly, and make sure Windows uses the Time Service by entering this into the console:

w32tm /query /source

You should see "VM IC Time Synchronization Provider".