VMware recently published a new knowledge base (KB) article 52085 that outlines instructions for enabling the Hypervisor-Assisted Guest Mitigation (CVE-2017-5715), also known as the Spectre vulnerability. This KB also provides steps to verify the updated microcode (included in the ESXi patch) has been applied along with Virtual Machine verification (those applicable) to ensure that they are seeing the new CPU features. While following the KB and patching one of my vSphere environments, I had noticed the verification steps was not only manual but it also to difficult to scale beyond a few VMs as it required customers to look for a specific set of strings from within the vmware.log file which is generated for each powered on VM, which could easily be several hundreds or thousands of VMs.

I figured there had to be a better way to help customers automate this at scale and remove the human factor. The other reason I was not fond of the current method is that it would require customers to either enable ESXi Shell/SSH access or to manually or through automation to download every single vmware.log file to inspect for specific log entries which can take quite a bit of time for any sizable environment. I had an idea on how this could be done without having to look at the vmware.log file while leveraging our vSphere APIs and did some investigation. Before proceeding further, please familiarize yourself with KB 52085. For complete background on both Spectre (CVE-2017-5753 & CVE-2017-5715) and Meltdown (CVE-2017-5754) as it relates to all VMware products, please carefully read through this top level KB 52245 which is being updated as new information is available, so you may want to subscribe to the KB for all the latest updates.

UPDATE 4 (01/23/18) - VMware has just updated KB 52345 with the current list of Intel CPUs affected by Intel Sightings. I have also updated my script to reflect all these changes. Make sure to download the latest version to ensure you have the latest changes.

UPDATE 3 (01/16/18) - I have just enhanced the script further to also collect the current microcode version running on a given ESXi host. Unfortunately, this information can only be collected when SSH is enabled and is something a user must explicitly allow. The benefit here is that Intel Sighting impact reporting is more robust as the code now checks for both impacted CPU signature as well as the microcode affected by Intel Sighting as outline in KB 52345. See below for more details.

UPDATE 2 (01/14/18) - As mentioned in the last update, I have been working on a Intel Sighting remediation script which can help customers automate the temporary workaround as recommended in KB 52345. Please see this blog post for complete details.

UPDATE 1 (01/13/18) - VMware just published a new KB 52345 outlining certain Intel Broadwell and Haswell CPUs being affected by Intel Sightings after applying latest microcode updates. Please refer to the KB for the complete details. I am currently working on a script to help with the remediation as the remediation method outlined in the KB would require customers to enable SSH access and manually update /etc/vmware/config. In the meantime, I wanted to provide a way for customers to easily identify whether their system are affected by Intel Sightings and thanks to community member Adam Robinson who just submitted a patch this morning to update my existing script to include these details. I have also added the CPU model to the output as additional information.

Note: This script only provides information about your existing vSphere environment, it does not make any changes or provides any remediation steps, please follow the KB for those instructions.

The PowerCLI script is called VerifyESXiMicrocodePatch.ps1 and it contains the following two functions:

  • Verify-ESXiMicrocodePatchAndVM
  • Verify-ESXiMicrocodePatch

The first function verifies both ESXi hosts and VMs and provides the output from the VM's point of view. The script will ensure VMs are at least  vHW9+ (as older vHW are not applicable), powered on and that one of the three new CPU features are available as outlined in the KB.  If the HypervisorAssistedGuestAffected column for a given VM row shows false, it means BOTH the microcode + ESXi patch has been applied and that the VM has gone through a complete power cycle to see the new CPU features. The second function only verifies that the ESXi microcode has been applied (this could have come a hardware vendor BIOS/Firmware update) but as mentioned earlier, it is also bundled within the ESXi patch. Similar to the first function, it also verifies that one of the three new CPU features are exposed to the ESXi host and you will also see an HypervisorAssistedGuestAffected column for each ESXi host. For columns which show true, please refer back to the KB above and follow the remediation procedure. In addition to information related to KB 52085, the second function has also been recently enhanced (see Update #1, 2 & 3 above) which include details about Intel Sighting. You can find more details below.

The Verify-ESXiMicrocodePatchAndVM function can run in one of three ways. If you do not pass in any parameters, then it will query all VMs for a given vSphere environment whether that is connecting to a vCenter Server or directly to a specific ESXi host. You have the ability to limit the scope by providing a specific vSphere Cluster by using the -ClusterName parameter or you can check a particular VM by using the -VMName parameter. Below is a screenshot exercising the three different methods:

The Verify-ESXiMicrocodePatch function can also be run in one of three ways. If you do not pass in any parameters, then it will query all ESXi hosts for a given vSphere environment when connected to a vCenter Server or a specific ESXi host if you are directly connected to a host. You have the ability to limit the scope by providing a specific vSphere Cluster by using the -ClusterName parameter or you can check a particular ESXi host by using the -VMhostName parameter. Below is a screenshot exercising the three different methods:

As mentioned earlier, the second function also contains additional information about the recent Intel Sighting issue which is represented by the IntelSighting column as seen in the screenshot above.

The value of IntelSighting can contain four potential values:

  • True = ESXi host contains microcode update is affected by Intel Sighting, you will need to apply the workaround as outlined in KB 52345
  • AffectedOncePatched = CPU is affected by Intel Sighting, but does not need the work around unless it has been patched or has a BIOS update that includes the fault microcode
  • False = CPU not affected by Intel Sighting it is currently recommended to only apply one of the ESXi patches (until Intel provides a microcode update fix), please refer to KB 52345 for full details
  • N/A = CPU is not Intel

The "default" method in how the script identifies whether an ESXi host is affected by IntelSighting is by comparing against the known CPU signatures as outlined in KB 52345 which can be done so using the vSphere API. Although this method is sufficient at this time since no updated microcodes have not been released (as far as I know), the results may be incorrect when new microcodes have been released and applied to remediate against Intel Sighting. A more robust solution would be to verify BOTH the affected CPU signature and the microcode version that are known to be impacted by Intel Sighting which is also available in the KB. The main issue from an Automation standpoint is that to be able to retrieve the microcode version that is currently deployed on an ESXi host, you would need to either enable ESXi Shell and/or SSH to system and use the vsish interface as this information is currently not available through a remote vSphere API.

Here is the vsish command you can manually run to retrieve the current microcode version:

vsish -e cat /hardware/cpu/cpuList/0 | grep "Current Revision:"

Since information can not be retrieved through a remote API, I have decided to enhance the function for those customers who wish to collect this information and have SSH enabled (can just be for the duration of running the script). To do so, I have introduced four optional parameters that must be provided if you wish to retrieve the microcode version:

  • IncludeMicrocodeVerCheck - By default, this is set to $false, you will need to set to $true
  • PlinkPath - You will need to provide the full path to the Plink executable which I use as an SSH client to perform the remote operation
  • ESXiUsername - The username to login to ESXi host and perform the operation, this would normally be root
  • ESXiPassword - The password for the ESXi username provided

Note: If you enable microcode version retrieval and SSH is disabled on the ESXi host, then you will see the microcode column have a value of SSHNeedsToBeEnabled. You can easily mass enable/disable SSH access for your ESXi hosts using PowerCLI. Here is an example blog post which shows you how.

Here is a screenshot of an ESXi host which is impacted by IntelSighting which the script has been able to determine by checking BOTH the CPU signature and the microcode version (which is now available as part of the output) as seen below.

64 thoughts on “Verify Hypervisor-Assisted Guest Mitigation (Spectre) patches using PowerCLI

  1. FYI – There’s a bug in the Verify-ESXiMicrocodePatch function with the incorrect variable $STIBPPass/$STIBPass, so you may get inconsistent results.

  2. Thank you! Can you explain the difference between the Hypervisor only patch (201711101-sg) and the Hypervisor-Assisted Guest Mitigation patch? Should you be installing both or one or the other?

      • Will,

        I need your help. Why the script is not working for me when I try to run it for an esxi or even through vcenter?

        I get the following:

        C:\Users\irving\Desktop> .\Verify-ESXiMicrocodePatchAndVM.ps1 Verify-ESXiMicrocodePatch

        It returns nothing.

          • I’m a Powershell noob as well, but try this:

            Import-Module Verify-ESXiMicrocodePatchAndVM.ps1

          • Marc is correct, but just to correct the filenames used here:

            Import-Module .\VerifyESXiMicrocodePatch.ps1

            Then use either:

          • Another way to run the script, which is nothing more than functions, is to simply “sourcing” the script by running: . ./VerifyESXiMicrocodePatch.ps1

  3. Thanks William!

    One thing I came across is that Powered Off VMs are listed as affected, since Runtime.FeatureRequirement returns null for them.

    I added Runtime.PowerState to the get-view and a few other lines to note them.

    • Yea, this was something that had crossed my mind as well for handling poweredOff VMs … I could filter for only poweredOn VMs to avoid this and even VMs suspended. I may update the script to add that logic

    • YOU ARE A F’KIN SAVIOR!! I MEAN THAT WITH EVERYTHING!! THANK YOU SOO MUCH. I am a noob in powershell i will admit it but you have saved me a ton of work and I appreciate it!

  4. Something seems off. I updated my Cisco UCS B200 M3 and M4 blades with the latest VMware patches. When manually checking the hosts based on the VMware KB, I do not see that the M3 blades received a microcode update, as the values are all 0, but the M4 blades do update

    VM with hardware v10 on M3 after patches
    2018-01-11T22:13:38.039Z| vmx| I120: hostCPUID level 00000007, 0: 0x00000000 0x00000000 0x00000000 0x00000000

    When I run your powershell script to check just the hosts in this cluster, it shows the M4 blades that have been patched as accurately not being affected, and the M3 blade that has not been patched yet it shows as affected, but on the M3 that has received all of the patches, it shows all 3 as present and affected as false which the line in the vmware.log of a VM doesn’t support. The CPU in this host shows up as:
    Intel(R) Xeon(R) CPU E5-2665 0 @ 2.40GH

    • It appears that after the patches are installed, then the host includes the featurecapability for cpuid.IBRS, cpuid.IBPB, and cpuid.STIBP, but there isn’t any verifying that the value is set to 1 in this table.

      I think this line in Verify-ESXiMicrocodePatch:
      $cpuFeatures = $vmhost.Config.FeatureCapability

      Should be replaced with
      $cpufeatures = $vmhost.Config.FeatureCapability | where {$_.value -eq 1}

      A similar change needed for the VM check.

  5. Just submitted a pull request. Verify-ESXiMicrocodePatch is only looking for the cpuid key names and not the values of them.

    This resulted in my hosts that do not have microcode updates showing that they had the new instructions when they in-fact do not.

    • Adam,

      Thanks for the pull request. I thought it was sufficient to just check for the CPU feature but didn’t realize the value needed to be checked. I just approved and hopefully its showing the correct results now

  6. For Verify-ESXiMicrocodePatchAndVM with -ClusterName argument, I get this error message:
    Get-View : Cannot validate argument on parameter ‘VIObject’. The argument is null, empty, or an element of the argument collection contains a null value. Supply a collection that does not contain any null values and then
    try the command again.
    At *****
    + $vms = Get-View ((Get-View $cluster.ResourcePool).VM)
    + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo : InvalidData: (:) [Get-View], ParameterBindingValidationException
    + FullyQualifiedErrorId : ParameterArgumentValidationError,VMware.VimAutomation.ViCore.Cmdlets.Commands.DotNetInterop.GetVIView

    Any ideas how to fix?

    • You probably miss-typed the name of your vSphere Cluster, make sure you spell it correctly and if there’s spaces, use a single tick mark between the string

    • Simply replace:

      $cluster = Get-View -ViewType ClusterComputeResource -Property Name,ResourcePool -Filter @{“name”=$ClusterName}
      $vms = Get-View ((Get-View $cluster.ResourcePool).VM) -Property Name,Config.Version,Runtime.PowerState,Runtime.FeatureRequirement

      with the following code:

      $cluster = get-view -ViewType ClusterComputeResource -Property Name -Filter @{“Name” = $ClusterName } | select -ExpandProperty MoRef
      $vms = Get-View -ViewType VirtualMachine -Property Name,Config.Version,Runtime.PowerState,Runtime.FeatureRequirement -SearchRoot $Cluster

  7. The script will ensure VMs are at least vHW9+ (as older vHW are not applicable) and that one of the three new CPU features are available as outlined by the KB.
    Wanted to confirm this, is it true that vms with hw version 9 and above are the only ones affected by CVE-2017-5715, anything below are not affected by CVE-2017-5715 ?

  8. vm’s with a hw version lower than 9 ARE affected, you have to update the vHW to at least vHW9 in order to be able to patch them.

  9. We ran this script for 12 ESXi which are at 6.0 build 7504637 (patched with ESXi600-201801401-BG & ESXi600-201801402-BG ).
    when I ran this report, I saw that few host reported with IBRPresent, IBPBPresent, STIBPresent & Affected as “True” and few came with “False”. All the Host are same H/w version (gen8, with no h/w patch for spectra) and are single host in Cluster. any reason am seeing this mismatch.

  10. I get a bunch of error when trying to run the script. I am using vm powercli and connect to vcenter but as soon as i run the script i get a bunch of invalid char lines is this script bad?

      • Yes Sir I am using VMware PowerCLI and have connected to my vcenter server through it using the Connect-VIserver Command

      • Security warning
        Run only scripts that you trust. While scripts from the internet can be useful,
        this script can potentially harm your computer. If you trust this script, use
        the Unblock-File cmdlet to allow the script to run without this warning
        message. Do you want to run
        [D] Do not run [R] Run once [S] Suspend [?] Help (default is “D”): R
        At C:\Users\isantos\Desktop\VerifyESXiMicrocodePatch.ps1:129 char:19
        + ~
        The ‘<' operator is reserved for future use.
        At C:\Users\isantos\Desktop\VerifyESXiMicrocodePatch.ps1:133 char:19
        + ~

        • The ‘<' operator is reserved for future use.
          At C:\Users\isantos\Desktop\VerifyESXiMicrocodePatch.ps1:1326 char:42
          + … 2018 GitHubGitHub‘ in expression or statement.
          Not all parse errors were reported. Correct the reported errors and try again.
          + CategoryInfo : ParserError: (:) [], ParseException
          + FullyQualifiedErrorId : RedirectionNotSupported

  11. Ok so i got it to go but when i enter the command Verify-ESXiMicrocodePatchAndVM i dont get a response back. its not spitting any information back into the cli. am i missing something?

  12. When I run verify-esximicrocodepatch -clustername I do not see the column intelsighting.
    This is what I see:
    VMHost CPU IBRPresent IBPBPresent STIBPresent HypervisorAssistedGuestAffec
    —— — ———- ———– ———– —————————- Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz True True True False Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz True True True False Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz True True True False Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz True True True False Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz True True True False Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz True True True False Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz True True True False

    • Somehow it now works showing intelsighting…
      Thanks for the great info and script. Much appreciated!!

  13. Anyone please help me. I do not get any response back when i run the script. I am doing everything the right way. but I dont know why i am not getting a response! Please help!

  14. how are you getting the text to format correctly? when I run the function I just get line after line, it’s not in columns

  15. Maybe you can add the server vendor + model with bios version to quickly check for new BIOS updates. Something like: Dell Inc. PowerEdge R620;2.5.4

  16. It should be noted that Intel came out today, and now recommends users and vendors do not retract their microcode.

    See this:

    And the section under January 17:

    Intel recommends that these partners maintain availability of existing microcode updates already released to end users. Intel does not recommend pulling back any updates already made available to end users
    NEW – Intel recommends that these partners, at their discretion, continue development and release of updates with existing microcode to provide protection against these exploits, understanding that the current versions may introduce issues such as reboot in some configurations
    NEW – We further recommend that OEMs, Cloud service providers, system manufacturers and software vendors begin evaluation of Intel beta microcode update releases in anticipation of definitive root cause and subsequent production releases suitable for end users

    • Jeff -From the same 1/18 statement:

      “For PCs and Data Center infrastructure, Intel recommends that patches be applied as soon as they are available from your system manufacturer, and software vendors
      For data center infrastructure, Intel additionally recommends that IT administrators evaluate potential impacts from the reboot issue and make decisions based on the security profile of the infrastructure”

      So as an IT/VMware administrator, how can Intel reasonably ask me to “evaluate potential impacts from the reboot issue and make decisions” when there is NO public data describing the precise reboot findings, as best I can tell? For example, the phrase “in some configurations” appears three times in that statement without any indication of which specific configurations fit the reboot risk profile. Unless of course it is as I suspect to eventually be revealed – that the answer to that question is “all configurations using the specified processors which run User mode programs”.

      Can anyone point out a good technical resource with specific details of the Intel Sightings reboot issue? Specifically – the “some configurations” under which these reboots have been observed?

  17. William – I a a bit confused by your definition of the value “AffectedOncePatched = CPU is affected by Intel Sighting, but does not , need the work around unless it is patched or has a BIOS update.” If my understanding of this issue is correct, a more complete definition might be “CPU is affected by Intel Sighting, but the hypervisor does not need this work around unless it has been patched to include the now-recalled patch levels specified in VMSA-2018-0004.2, regardless of whether or not the underlying host BIOS has also been updated with a version containing the new faulty microcode.”

    I am thinking specifically of the case where a customer may have updated the host BIOS from their vendor but has NOT yet applied the now-withdrawn ESXi patches. In this case even guest vm’s to which mitigating OS patches have been applied (and have also been subsequently powercycled) will NOT be able to use those new CPU instructions IBRSPresent, IBPBPresent, and STIBPPresent (and thus will not be at risk of reboot per Intel Sightings) because those new instructions are not present in the vCPU presented to those guests. Those new instructions will not be exposed to guest vCPUs for use by guest OS’s until the ESXi build has been patched with builds newer than those specified in VMSA-2018-0002.2 AND that guest has subsequently power-cycled to see the newly-exposed instructions in the vCPU..

    That is a long-winded way of simply asking -is an ESXi host or guest vm at risk of Intel Sightings reboots if It has a patched host BIOS and patched guest vm OS (including power cycle) but HAS NOT patched ESXi with anything newer than the VMSA-2018-0002.2 recommended versions?

    • Not exactly. The ESXi patches isn’t the issue but rather the fault microcode that was provided by Intel which is bundled with the ESXi patch. This faulty microcode could have also been delivered as part of a BIOS update from your HW vendors or through another means. The “AffectedOncePatched” is merely stating that because the CPU signature was identified to be on the affected list AND it currently does not have the faulty microcode, that if you did happen to pull the patch before VMware recalled it, then you would be affected by Intel Sighting, if not, you are fine for now and you can follow the rest of the existing guidance

  18. Hi,

    If hosts were patched using VMSA-2018-0004.2 before the patches were withdrawn and they’re not affected by sighting (E5-2600 v2), shouldn’t ‘Verify-ESXiMicrocodePatch’ show True for the new CPU features (IBRS,STIBP,IBPB)?

    Thanks for the explanation.

  19. Hi After executing the script, i dont see Intel sighting column on the output:
    Verify-ESXiMicrocodePatch -VMHostName Hostname

    VMHost CPU Microcode IBRSPresent IBPBPresent STIBPPresent HypervisorAssistedGuestAffect
    —— — ——— ———– ———– ———— —————————–
    Hostname Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz N/A False False False True

  20. Hi!
    Thank you for this post. But I dont understand the values. Is this result bad?
    CPU is not effected, but still get Guestaffected. I have read the KB, but I don´t understand

    PowerCLI C:\Script> Verify-ESXiMicrocodePatch

    VMHost CPU Microcode IBRSPresent IBPBPresent STIBPPresent HypervisorAssistedGuestAffected IntelSighting
    —— — ——— ———– ———– ———— ——————————- ——–
    esx50 Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz N/A False False False True False

    PowerCLI C:\Script> Verify-ESXiMicrocodePatchAndVM

    VM IBRSPresent IBPBPresent STIBPPresent vHW HypervisorAssistedGuestAffected
    — ———– ———– ———— — ——————————-

    SERVER False False False vmx-09 True

  21. Does the hypervisor-assisted-guest mitigation require that the guest operating system also include the latest security fixes that mitigates Spectre and Meltdown?

  22. i must be missing something. I have applied ESXi550-201709101-SG and ESXi550-201801301-BG to my esxi hosts. I run this script and and everthing still comes back false. HypervisorGuestAffect is True. The build i show i am runing 5.5u3f at this time after the updates are applied.

  23. Gidday William, I think that you may need to update your scripts now. I have updated a server to the new HPE Proliant 460c Gen9 updates, and it has updated the Microcode to 0x0b00002a, ( which is higher than 0x0B000025 for the Intel Xeon E5-2600-v4 ). I think that this will mean that you might need to check if the microcode is higher than the blacklisted, so that it reports that it has been update.

Thanks for the comment!