A question that I almost always see come up on a regular basis is around the topic of auditing or understanding what configuration changes were made to a given Virtual Machine. Today, the process of identifying this information is actually quite difficult at least without resorting to a custom built solution which allows you to compare the configuration changes over time. This is definitely an area that VMware is investing heavily in and although I can not go into specific details, believe me when I say they are taking this very seriously both from a data completeness standpoint as well as simplifying the user experience.

Having said that, what options exists for customers today?

  1. Custom Solution - A system that could periodically snapshot your VM configurations into some type of data warehouse or CMDB platform. There are many challenges here but the biggest one is efficiently capturing the changes and ensuring you do not impact the overall performance of your vCenter Server, especially with larger inventories.
  2. vRealize Air Compliance - This is a new VMware SaaS offering which provides both compliance and remediation for your VM Configurations. I had a chance to preview this awhile back and I have to say it is a pretty slick solution. You can easily step back in time to see exactly what has changed for a given set of VMs, very intuitive UI. You can also add additional configurations to monitor and be alerted on when you are out of compliance. Definitely something worth checking out, especially for customers who must go through regular audit reviews.
  3. vCenter Configuration Manager - I have not personally used this tool before, but I have been told it would also be possible to detect configuration changes for your VMs.
  4. Enabling "Trivia" Logging in vCenter Server - Though this is an option, it is not one that I recommend for variety of reasons. The "Trivia" logging mode is very verbose and will generate huge amounts of data which will causes your logs to quickly rotate out if you are not forwarding to a remote syslog server. There's also additional overhead cost for this type of logging and more importantly, it may not capture all of the required data. This is an approach that some customers have tried but is not really a practical solution.

This topic has always been interesting to me and with several recent inquiries from the field, it got me thinking about this area again. While working on a completely different project, I ended up on Luc Dekens awesome blog and came across his Events Part 3: Auditing VM Device Changes article. If you take a look at the article, you will see that Luc shows you how you can easily audit changes to a VM's devices (e.g. Virtual Disk, CD-ROM, etc). What Luc demonstrated in his script is just a specific type of configuration, but the point is that this type of information has always been available, just not easily consumable.

The secret is to key off of the VmReconfiguredEvent which includes a configSpec property that captures the exact set of configuration changes for a given VM. Below is an example of the configSpec dump of one of these events. We can clearly see that this VM had its vCPUs modified to 4 and its vMEM modified to 20GB.

what-changed-when-vm-is-reconfigured-2
With this information, we can now easily query the configuration changes for a given VM by looking through its past events. Leveraging the awesome work that Lud has already done with his script, I slightly enhanced it to cover more than just device changes but overall VM configuration changes. With that, here is a PowerCLI script/function that I created called Get-VMConfigChanges.ps1

Note: The amount of historical events that you will be able to search through will purely depend on your Center Server DB's retention period of Tasks/Events. For VMs which have been deleted, you will not be able to retrieve any events as they must be associated with an object in the database.

Below is an example of how to use this function which accepts a VM object and the number of hours (default to 8) to search through the VM's events:

From the output below, we can clearly see the following configuration changes have been applied:

  1. Change vCPU to 2
  2. Change vMEM to 4GB
  3. Change vMEM to 5GB and Edited Virtual Disk (you can of course get further details by dumping more information)

what-changed-when-vm-is-reconfigured-1
Although this solution is not as clean as the vRealize Air Compliance offering, it does allow anyone to quickly pull out the relevant configuration changes for a given VM along with the user and time the configurations was performed. Hopefully this goes to show how powerful the vSphere Platform APIs really are and it is definitely worth while in learning how they work.

15 thoughts on “How to audit VM reconfigurations and see what exactly changed?

  1. You could properly also use the vmware.log files and to ease the process send them to a syslog, then there should be potential for some datacenter wide analytics on changes. Using log insight there could also be created events and send til vRops and there from reporting should be easy.

  2. In William’s example the VM had the number of CPU’s and memory changed. That data is in the VCDB and William has done a great job showing how to get that information out.

    The issue is that that same information is not expressed in the logs. All you’ll get is “William reconfigured the VM “. That’s it. Logging systems like Log Insight and others are only as good as the data they get. To make the data actionable, you’d be looking for keywords. For example, if the logging was “William changed the VM memory on VM from 2048 to 4096” then we have something we can work with on a logging system. I can parse for “changed the VM memory” and the values and create an alert accordingly.

    My favorite example is I have a VM on the PCI network. An admin changes that to the non-PCI network. The log is “JBlow@foo.com reconfigured . Doesn’t tell me what from/to. If the log was “jBlow@foo.com reconfigured network from “PCI Network” to “External Network” THEN I have some actionable items.

    From an operational standpoint this is far easier to react to than going in after the fact to retrieve the information or setting up alerts and scripts. (I’m not dismissing the great work William has done at all. I think he’d agree with me. Mostly :))

    Here’s hoping it gets better in the future (of which we cannot speak about)

    mike

    • @Mike – I think @michael was asking about vmware.log (Virtual Machine’s log file that’s generated when its powered up) and not general VC/ESXi logging which I agree doesn’t contain the details.

      The vmware.log DOES contain all VM configurations, however you would have to periodically dump the file and/or forward to a remote syslog server and then compare the deltas which can be quite difficult. So though you *could* use this method to retrieve the properties, its probably not very efficient

  3. Hey William, so I have this up and running on an automation VM and it emails me a report every 12 hours. So far it looks ok, but what the information I’m really looking for is auditing of disk expansions. Like, who added 50GB to a drive on server X and when was it done? So far this script is spotty at best at even reporting that a change is made, but it doesn’t include information as to what drive and how much space was added. Hoping this is including in something like vROps soon.

    • Adam,

      I’m not sure what you mean by “spotty”, assuming you’ve specified the right time interval, any “Reconfigure” operation should be caught by the script. Below is an example of expanding a VMDK for a VM called “Base-ESXi”, you can see the details below on the specific user, start/stop time and also that this was an Edit operation on the VMDK. It doesn’t go to the next level of detail which I agree would be useful and something Engr is quite aware of and will be improving in the future.

      ChangeVersion : 2016-06-08T20:20:21.077253Z
      DeviceChange : {VMware.Vim.VirtualDeviceConfigSpec}
      Files : VMware.Vim.VirtualMachineFileInfo
      VMName : Base-ESXi
      Start : 6/17/2016 7:03:39 PM
      End : 6/17/2016 7:03:39 PM
      State : success
      User : VGHETTO.LOCAL\Administrator
      ChainID : 276545
      Device : VirtualDisk
      Operation : edit

      • Spotty meaning the reporting isn’t very consistent. I have this running every 12 hours as a scheduled task and the script pulls 12 hours of data. Last week during testing I would make disk changes (add, delete or expand) and only 6 out of 10 changes showed up in the report. Today I tried changes on 2 VMs and neither showed up in the report. Not really looking for a solution now (although it would be nice), but hoping that something like this will be included in the rumored vROps update later this year.

      • Finding more…the script as is will report on a disk expansion, but not when a new disk is added to a VM.

      • Replying to myself again. I’m using Datastore clusters and SDRS is enabled, so adding a disk is a vCenter event “Apply Storage DRS recommendations” and not a VM level reconfigure machine vent. At least there’s an explanation for it!

  4. Hi. Just love this script. 🙂
    Any chance you can get the same information about changes in vCenter environment? What I’m looking for is host configuration changes or vCenter cluster settings changes and so on?

  5. Hi William, Thanks for the script. Im looking for a way to track vcpu and memory changes. Is there a way to get what the setting was changed from as well? E.g. the VM was 8GB and is now 10GB. Thanks, Geoff.

Thanks for the comment!