Time synchronization is very important in any production environment, as a correct time configuration is mandatory for many applications. Some examples of this are authentication DB replication, and backup…etc.
I had recently noticed continuous security alerts appearing in my monitoring tool, showing hundreds of repetitive events related to system time change in some VMs running on our virtual infrastructure in vSphere and vCenter 6.7.
After connecting to the concerned VMs and checking closely these events, it was effectively shown in the Security section of the Windows Event Viewer.
Quite interesting! Why such events keeps prompting while no manual change in time has been made on any of these VMs? Moreover, why is the VMware tools daemon (vmtoolsd.exe) playing with this setting?
In this infrastructure, all affected servers were domain joined virtual servers running on vSphere 6.7 environment.
First things first, and the key thing here is to understand how servers are synchronizing time and the steps followed to do so. Below is how a server behaves for time synchronization when it first boots up.
If it is a physical Windows Server, it tries to synchronize first from its hardware clock (CMOS) then from any NTP server (Domain Controller) configured. For most Microsoft infrastructure, where a Microsoft Active Directory Domain is installed, it is the PDC server who is the time source for your domain environment. All clients, Servers, Additional DCs sync their time from PDC by default and PDC sync its time from given time source.
Virtual servers have the exact same behavior when it comes to time synchronization. However, the the hardware clock in this case will be the one of the virtualization host, so the VM will synchronize time from the host it is running as a first step.
In my case, all affected servers were virtual and none of them were restarted recently, so hardware clock synchronization was not intended to happen whatsoever.
After some investigation, I realized that system time on those virtual machines was triggered even though the VMware tools Time Synchronization setting was disabled for those VMs.
What is VMware Tools Time Synchronization?
One of the many settings you can configure on the VM Tools control Panel is the virtual machine and the host operating system setting. This setting allows you to ensure the guest operating system’s time has not fallen behind of the host system. Clearing this option prevents VMware Tools from performing this check while the guest system is up and running.
It’s worth noting though, it does not prevent the synchronization check from being performed when the virtual machine is started, resumed, restored from a snapshot, or when the size of virtual disk is reduced using VMware Tools. This simply means that Time Synchronization will effectively happen in case one of these action happens. Taking into consideration how the underlying hardware behaves with regard to the different interruptions it has to manage, the result of this is that time appears to drift inside those guest VMs, compared to the real hardware clock.
In my case, it was a 2 seconds lag, as shown below.
In fact, and because a planned update that needed to be applied on the hosts on the maintenance mode state, I had to manually evacuate those hosts, and therefore many VMs were moved (vMotion) back and forth across the hosts in the cluster. This resulted in triggering the time synchronization on the VMs and prompting the change time alerts in the monitoring system.
The environment I am managing runs on VMware vSphere, but this actually applies to both VMware and Hyper-V environments. You can find more information on this VMware Knowledge Base article, and this Microsoft Virtualization blog.
How to Disable VM/Host Time Synchronization?
Depending on your environment, you may want to keep time synchronization with the default configuration. However, if you are in situation where you want to go ahead and disable completely this setting and make your VMs avoid the step of adjusting time with their host, and contact directly the NTP server of your environment.
On Microsoft Hyper-V
For Hyper-V virtual machines, simply disable Time Synchronization in its VM settings.
- Open Hyper-V Manager.
- Select the Virtual Guest DC.
- Click Settings.
- Click Integration Services.
- Clear the Time Synchronization option.
- Click OK then restart your VM.
On VMware vCenter
For VMware virtual machines, you have a finer grained control over time synchronization settings, but it is necessary to edit the virtual machine’s .vmx configuration file and set the respective options to either TRUE or FALSE based on the following table:
| Configuration Setting | Description |
|---|---|
| time.synchronize.tools.startup | Specifies whether a time synchronization between guest and host is performed when the virtual machine is started or rebooted. |
| time.syncTime | Specifies whether a check should be performed every minute to ensure the time of a running guest system does not lag that of the host. |
| time.synchronize.restore | Specifies whether a time synchronization between guest and host is performed when the virtual machine is restored from a snapshot. |
| time.synchronize.resume.disk | Specifies whether a time synchronization between guest and host is performed when a suspended virtual machine session is resumed. |
| time.synchronize.continue | Specifies whether a time synchronization between guest and host is performed when a snapshot of the virtual machine is taken. |
| time.synchronize.shrink | Specifies whether a time synchronization between guest and host is performed when VMware Tools are used to shrink a virtual disk assigned to the virtual machine. |
Once these settings are applied, running, rebooted, resumed, or restored VMs should not trigger any time synchronization with the host whatsoever. All time synchronization will happen against the configured NTP server.
Disabling VM/Host Time Synchronization from Windows
If you’re not sure or not want to take the risk of messing up with VM settings, or may also change the time synchronization behavior for the Windows OS itself, by changing below registry key to TRUE or FALSE.
HKLM\SYSTEM\CurrentControlSet\Services\W32Time\TimeProviders\VMICTimeProvider and the Enabled value to 0 (disabled) or 1 (Enabled).
Alternatively, you may also run this command from the command line Window:
Reg add HKLM\SYSTEM\CurrentControlSet\Services\W32Time\TimeProviders\VMICTimeProvider /v Enabled /t reg_dword /d 0
After disabling synchronization by any of the described methods, it is necessary to restart the time service, this will reset it to a new source.
net stop w32time net start w32time w32tm /resync /force
Changing the time synchronization from registry can be used as a backup solution in case the settings changed from the VM configuration don’t work. Preparing a VM template with this registry setting may also apply.
Salam alikum….Bravo!!!