virtuallyGhetto

vsish

Verify Hypervisor-Assisted Guest Mitigation (Spectre) patches using PowerCLI

by 77 Comments

VMware recently published a new knowledge base (KB) article 52085 that outlines instructions for enabling the Hypervisor-Assisted Guest Mitigation (CVE-2017-5715), also known as the Spectre vulnerability. This KB also provides steps to verify the updated microcode (included in the ESXi patch) has been applied along with Virtual Machine verification (those applicable) to ensure that they are seeing the new CPU features. While following the KB and patching one of my vSphere environments, I had noticed the verification steps was not only manual but it also to difficult to scale beyond a few VMs as it required customers to look for a specific set of strings from within the vmware.log file which is generated for each powered on VM, which could easily be several hundreds or thousands of VMs.

I figured there had to be a better way to help customers automate this at scale and remove the human factor. The other reason I was not fond of the current method is that it would require customers to either enable ESXi Shell/SSH access or to manually or through automation to download every single vmware.log file to inspect for specific log entries which can take quite a bit of time for any sizable environment. I had an idea on how this could be done without having to look at the vmware.log file while leveraging our vSphere APIs and did some investigation. Before proceeding further, please familiarize yourself with KB 52085. For complete background on both Spectre (CVE-2017-5753 & CVE-2017-5715) and Meltdown (CVE-2017-5754) as it relates to all VMware products, please carefully read through this top level KB 52245 which is being updated as new information is available, so you may want to subscribe to the KB for all the latest updates.

UPDATE 4 (01/23/18) - VMware has just updated KB 52345 with the current list of Intel CPUs affected by Intel Sightings. I have also updated my script to reflect all these changes. Make sure to download the latest version to ensure you have the latest changes.

UPDATE 3 (01/16/18) - I have just enhanced the script further to also collect the current microcode version running on a given ESXi host. Unfortunately, this information can only be collected when SSH is enabled and is something a user must explicitly allow. The benefit here is that Intel Sighting impact reporting is more robust as the code now checks for both impacted CPU signature as well as the microcode affected by Intel Sighting as outline in KB 52345. See below for more details.

UPDATE 2 (01/14/18) - As mentioned in the last update, I have been working on a Intel Sighting remediation script which can help customers automate the temporary workaround as recommended in KB 52345. Please see this blog post for complete details.

UPDATE 1 (01/13/18) - VMware just published a new KB 52345 outlining certain Intel Broadwell and Haswell CPUs being affected by Intel Sightings after applying latest microcode updates. Please refer to the KB for the complete details. I am currently working on a script to help with the remediation as the remediation method outlined in the KB would require customers to enable SSH access and manually update /etc/vmware/config. In the meantime, I wanted to provide a way for customers to easily identify whether their system are affected by Intel Sightings and thanks to community member Adam Robinson who just submitted a patch this morning to update my existing script to include these details. I have also added the CPU model to the output as additional information.

Note: This script only provides information about your existing vSphere environment, it does not make any changes or provides any remediation steps, please follow the KB for those instructions.

The PowerCLI script is called VerifyESXiMicrocodePatch.ps1 and it contains the following two functions:

  • Verify-ESXiMicrocodePatchAndVM
  • Verify-ESXiMicrocodePatch

[Read more...] about Verify Hypervisor-Assisted Guest Mitigation (Spectre) patches using PowerCLI

How to clear the ARP cache in ESXi prior to vSphere 5.5

by 3 Comments

Yesterday, I wrote an article about a new ESXCLI command that will be available as part of the soon to be released vSphere 5.5 release which allows a user to clear the ARP cache on an ESXi 5.5 host. It was my understanding that with past releases of ESXi, that it is not possible to clear the ARP cache. While working on that article, I came across an internal thread and learned that it was possible to clear ARP entries prior to ESXi 5.5, however the method is not as ideal as using the new ESXCLI command.

Disclaimer: This is not officially supported by VMware, please use at your own risk

In ESXi 5.1, you can list the current ARP entries for an ESXi host by running the following ESXCLI command:

esxcli network ip neighbor list

To clear a particular ARP entry, you will need to use the unsupported vsish interface. To delete a specific ARP entry, you will need to run the following command:

vsish -e set /net/tcpip/v4/neighbor del [IP-ADDRESS]

Here is a screenshot of listing the current ARP entries using ESXCLI and then deleting one of entries using vsish:

As mentioned in my previous article, I am glad we have made this into an official command in ESXCLI 5.5, but if you are in a crunch you can still clear an ARP entry if you are not running ESXi 5.5 using vsish.

What’s New in VMware Vsish for ESXi 5.1

by Leave a Comment

While going through and cleaning up my ghettoSkunkworks projects folder before going on PTO, I came across this little post which I must have forgotten to publish. With the latest release of ESXi 5.1, there are now a total of 162 new vsish configuration parameters, with 63 of those being hidden parameters.
***As usual with any hidden configurations, please be careful and use at your own risk as the vsish interface is not officially supported by VMware***
Here is the complete list of new vsish configurations in ESXi 5.1 - https://s3.amazonaws.com/virtuallyghetto-download/complete_vsish_config_510.html
 Additional Resources:

Configuring ESXi Power Management Policy Using the CLI

by 10 Comments

An interesting question on the VMTN forum caught my eye today, which was around configuring ESXi's Power Management Policy using the command-line via a kickstart script. I found this question to be interesting as I never had to tweak this configuration and was curious myself to see how you might be able to perform this via the command-line as I never recall seeing a command relating to the power management settings.

After a few minutes of digging, I found that the standard set of CLI's such as ESXCLI, vim-cmd, etc. do not provide a way to configure ESXi's power management settings but did find it was possible using my other favorite "not officially supported" CLI called vsish. Now, you can of course create a remote script using the vSphere API to configure this setting, but if you are looking to modify this within a kickstart script, this is route you will want to take.

UPDATE (01/12/15) - I just found out today from Engineering that it is possible to configure ESXi power management policy using ESXCLI. Though the parameters are currently set to hidden, you can use the following command to set the appropriate policy based on your enviornment.

esxcli system settings advanced set --option=/Power/CpuPolicy --string-value="High Performance"

UPDATE (11/04/14) - It turns out configuration changes made directly through vsish do not persist after a reboot, this might be problematic for most of you 😉 Luckily, Alan Castonguay who works in our GSS organization reached out and created a nice pyvmomi (vSphere SDK or Python) script that can be executed in the ESXi shell and of course it can easily be integrated into a Kickstart script. I have tested his sample script to verify its functionality and have also checked it into my Github repository so that others can benefit from. You can download the script which I have named configure_esxi_power_policy.py

If you run the script without any arguments, it will display the current power policy that has been enabled as seen in the screenshot below:

configure-esx-power-policy-0
To change the policy, you will need to specify the "shortName" power policy, in this example I want to change it from "static" to "low":

configure-esx-power-policy-1
To check whether your ESXi host supports power management, run the following command:

~ # vsish -e get /power/hardwareSupport
Hardware power management support {
CPU power management:Enhanced Intel SpeedStep(R)
Memory power management:Not available
}

To view the current power management setting, run the following command:

~ # vsish -e get /power/currentPolicy
Host power management policy {
ID: 2
Short name:dynamic
Long name:Balanced
Description:Reduce energy consumption with minimal performance compromise
}

Just like the vSphere Client, you have 4 options which maps to the "ID" property as seen above. You can get more details by querying each of the policy (1-4), here is an example:

~ # vsish -e get /power/policy/1
Host power management policy {
ID: 1
Short name:static
Long name:High Performance
Description:Do not use any power management features
}

Here's a quick table that maps the policy ID to power management policy which is the same order as shown in the vSphere Client:

Policy ID Power Management Policy
1 High Performance
2 Balanced
3 Lower Power
4 Custom

To change the power management policy, run the following command:

~ # vsish -e set /power/currentPolicy 1

So now you can integrate power management settings in your ESXi kickstart script for automated deployment and configurations!

What’s New in VMware Vsish for ESXi 5

by 2 Comments

I wrote about What Is VMware Vsish back in 2010 which included a list of 771 configuration parameters some public and some hidden. With the latest release of ESXi 5, I have compiled a list of the net new configuration parameters that comes out to the following:

Total: 232 
Public: 146
Hidden: 86

***As usual with any hidden configurations, please be careful and use at your own risk as the vsish interface is not officially supported by VMware***

Some of the interesting hidden parameters such as /VMFS3/EnableBlockDelete and /VMFS3/BlockDeleteThreshold could be useful in dealing with Dead Space Reclamation also known as the UNMAP VAAI primitive. For more details about the UNAMP feature, take a look at Duncan Epping's detailed post here.

For the complete list, please take a look at https://s3.amazonaws.com/virtuallyghetto-download/complete_vsish_config_500ga.html.

How to query for MACs on internal vSwitch on ESXi

by 9 Comments

There was an interesting question this week on the VMTN community forums about querying a vSwitch on an ESX(i) host. The user was trying to locate a particular virtual machine's MAC Address due to an IP conflict that was identified. The internal VMware vSwitch is pretty much closed off as a blackbox. The vSwitch is not exposed like a traditional physical switch in which you can run commands against such as "show mac-address-table" to display the MAC addresses found on the switch.

However, you can still perform a lookup of all the MAC Addresses found on a particular ESX(i)/vCenter host by using the vSphere APIs. You can search for all virtual machines and dump out their associated MAC Addresses and correlate that back to a particular vSwitch. You can easily do this through a script such as using the vSphere SDK for Perl script: getvSwitchMacTable.pl which supports both stand vSwitch and distributed vSwitch or if you prefer a GUI, you can use the popular RVTools. I am sure there is most likely a PowerCLI solution to solving this problem as well.

The solution described above is the proper and most flexible way of solving this problem, but what if you really wanted to query the internal vSwitch and extract out the MAC Addresses that way? Well the answer is, you can so using vsish on ESXi (vsish is not available on ESX unless you have the VMware debugging RPM package installed).

Here are some of things you can view for a given vSwitch using vsish:

~ # vsish -e ls /net/portsets/vSwitch0
ports/
overlays/
uplinks/
type
mtu
unlink
link
destroy
properties
stats

The "ports" section is what we are interested in:

~ # vsish -e ls /net/portsets/vSwitch0/ports
16777217/
16777218/
16777219/
16777220/
16777358/
16777359/

When looking at a particular port, it provides quite a bit of information on what is connected and various metrics/statistics:

~ # vsish -e ls /net/portsets/vSwitch0/ports/16777220/
respool/
e1000/
vmxnet3/
pktSizes/
clusterSizes/
worlds/
coalesceDetailed/
ip
schedTeamUplink
teamUplink
blocked
injectIGMP
txCompCoalesce
txCoalesce
rxCoalesce
controlChain
notifyStats
inputStats
outputStats
vmxnet2clientStats
clientStats
gateway
setPassthru
status
stats

As you can see it is pretty tedious to go through each of the ports and it does not easily allow you to figure out what is exactly connected to the port until you view the "status" property.

I decided to write a tiny script that would allow a user to dump out all the MAC Addresses from the vSwitch(s) found on an ESX(i) host. Not only does it provide this mapping but also what is specifically using a given port whether it is mapped to internal interface or a particular virtual machine.

You can download the script vswitchInfo.sh which runs directly on ESXi's TSM (Tech Support Mode). The script can be called with the "-l" option to provide a high level dump of all MAC Addresses. Once you have identified the particular vSwitch and port, then you can get further details by specifying "-v" for vSwitch name and "-p" for the port number as displayed from the previous execution.

Here is an example output of just listing all MAC Addresses from all vSwitch(s) in an ESXi host:

Here is an example of getting more details on a particular port on a vSwitch:

Here you can see the clientName which is either a VM or interface using the port. You will also notice there is a mapping to set of pNICS that are attached to the vSwitch and various other details that I will let you explore.

You might have noticed the vSwitch port-ids looks kind of familiar? If you did, they actually are, as they part of the "networking" section in esxtop/resxtop output.

Unfortunately with esxtop/resxtop, it does not display the associated MAC Addresses, but now you have a way to easily query for details on the internal ports of a vSwitch.

Note: The second solution falls under the "not supported" category as you might have guessed.