VMware Tools 10.2.0 enables Virtual Machine vNIC exclusion and priority re-ordering

VMware Tools 10.2.0 just GA'ed (release notes / download and open-vm-tools release notes / open-vm-tools download) and this release includes a number of new features like an offline bundle for VMware Tools VIB for ESXi and support for deploying VMware Tools using Microsoft System Center Configuration Manager (SCCM) to just name a few. There are also two additional new capabilities that I wanted to share as I think customers can benefit from and take advantage of immediately around how Virtual Machine vNICs are displayed. One of the challenges with having the broadest Guest Operating System (GOS) support in vSphere is dealing with some of the different behaviors of each GOS. One such example are the various ways in how both physical and logical networks interfaces are enumerated by an OS.

Take the example below, I have a PhotonOS VM which has eth0 as the primary interface and it is configured with an IP Address of However, as you can see from the screenshot below I am actually getting back a different address and interface. In addition to this, we also see other logical interfaces showing up in the IP Address list such as Docker interfaces as well as virtual and other pseudo interfaces that may or may not be useful to VI Admins.

Historically, there was not a way to control what would show up in the network interface list which is then propagated from VMware Tools up to both the vSphere API as well as vSphere UI. With this new release of VMware Tools, which can be applied asynchronously to a given vSphere release, customers now have the ability to filter on a per-VM basis on what interfaces actually show up as well as a relative priority for interfaces that customers care more about.

Continue reading

Getting started with Hybrid Cloud Extension (HCX) on VMware Cloud on AWS

I had been hearing a lot of cool things about VMware's Hybrid Cloud Extension (HCX) but never tried the solution myself nor had a good understanding of what it actually provided. With the recently announced Hybrid Cloud Extension (HCX) on VMware Cloud on AWS (VMWonAWS) offering being available, I thought this was a great way to get hands on with HCX and take advantage of my VMWonAWS infrastructure. Having only spent a couple of days with the solution, I can see why customers are excited for HCX and the new offering on VMWonAWS makes it super easy to consume.

There are a number of impressive capabilities that HCX offers, but two that really stood out to me which I thought was quite unique and interesting compared to other VM-based "migration" options. The first is that HCX can perform live VM migrations (vMotion) or replicated migrations (vSphere Replication) which includes scheduled switch over across different versions of vSphere (vSphere 5.x to/from vSphere 6.x). This is great for customers who may not be able to upgrade their underlying vSphere environment to 6.0 or later and take advantage of things like Cross vCenter vMotion feature which only supports VM migration between vSphere 6.0u3 to/from 6.x.

The second capability is that HCX can abstract and protect the underlying ESXi hosts by not requiring direct connectivity between the source and destination ESXi hosts. Traditionally, for vMotion and vSphere Replication traffic, you either had to stretch the VLAN or ensure the VMkernel interface was routable so that it can communicate with the destination ESXi hosts for data transfers. This was not always possible and adds additional networking requirements which can be challenging to implement depending on how your network infrastructure is configured. The way HCX solves this problem is by using a special HCX Cloud Gateway which securely proxy vMotion and vSphere Replication traffic from the on-premises environment out to the respective HCX Cloud Gateway Peer which then gets transfered to destination vSphere environment. Below is a diagram to help illustrate this:

Note: HCX also supports WAN optimization (compression and de-duplication) out of the box, which the diagram includes as that is what I had deployed in my env. This is an optional virtual appliance that can be deployed at each location ensuring efficient data transfer between the source and destination vSphere environments.

While going through and getting HCX configured on both my VMWonAWS and onPrem environment, I had ran into a few minor gotchas and to help others avoid some of the issues I had ran into, I figure I would outline the process and include some additional tips that can be help.

Continue reading

Can the VCSA 6.5 forward to multiple syslog targets?

I had a couple folks ping me recently asking whether the latest vCenter Server Appliance (VCSA) 6.5 release supports forwarding to multiple syslog targets? Currently today, only a single syslog target is officially supported which can be configured using the VAMI UI. I know this is something our customers have been asking about and I know this is something the VC Engineering team is considering.

Having said that, it is possible to configure additional syslog targets on the VCSA, but please be aware this is not officially supported. A couple of these customers understood the support impact and were still interested in a solution as some of their environments mandated multiple redundant syslog targets and using a syslog forwarder/relay was not an option for them.

Disclaimer: This is not officially supported by VMware, please use at your own risk.

When configuring syslog forwarding from the VAMI UI, the configurations are all written to /etc/vmware-syslog/syslog.conf on the VCSA.

With this information, if we want to add additional targets (which can be of the same configuration or different), you simply append additional targets to the syslog configuration file. For example, if I have two syslog targets and and I wish to use the default log level, TCP and 514, I would use the following:

Once you have saved your changes, you will need to restart the rsyslog service for the change to go into effect. To do so, run the following two commands on the VCSA:

systemctl stop rsyslog
systemctl start rsyslog

One additional thing to note is that the VAMI UI will only show the very last syslog target within the configuration file but if you monitor syslog servers, you will see that logs are indeed being forward to all servers that have been configured in the syslog configuration file.

Deployment models for vSphere Content Library

When talking to customers about vSphere Content Library deployments, one question I normally get is how best deploy Content Library for optimal workload deployment, especially in scenarios where remote or branch offices are involved? There are two main deployment models for vSphere Content Library as the title has alluded to. The main difference between the two is whether you have a single vCenter Server or if you have multiple vCenter Servers, with each managing its own vSphere infrastructure?

Lets refer to the single vCenter Server case as Scenario 1 and the multi-vCenter Server case as Scenario 2 and below are the two scenarios outlined with additional details.

Scenario 1 (Single vCenter Server):

In this scenario, which is a fairly common deployment for many smaller to mid-size organizations, where you only have a single or very few vCenter Server(s). They are used to manage several remote locations which only consists of ESXi hosts running at each of the locations and storage local to the site is available. In addition, there are several expected behaviors of the content itself which I have formulated into the following table below:

Content Management Content Distribution Content Deployment
Centrally Managed Sync across the WAN Workloads stored and deployed locally

For this type of an environment, you would first setup a published library which stores all the content that you wish to distribute across your remote sites. Next, you would create subscriber library(s) consuming the published library, but instead of storing the replicated content locally, it is actually stored at each of the remote locations and their respective vSphere Datastore(s). This ensures that content is synchronized from our published library out to each of the remote locations, but when content is requested for deployment, the traffic is local to the site rather than going across the WAN.

In the above scenario, since there is only a single vCenter Server, if it ever becomes unavailable then provisioning and management to the remote location will also be unavailable. This is the expected behavior regardless if Content Library is configured.

Continue reading

How to audit vSphere API usage?

I was recently reminded of an excellent VMworld 2017 session that given by Ravi Soundararajan, a Principal Engineer at VMware working in our vCenter Server Performance Team. In his session, vCenter Server Performance Deep Dive, Ravi provides some great insights into things to consider that may have an impact on vCenter Server performance. In addition, he also covered a few additional topics, one of which that comes up every so often which around auditing vSphere API usages for a given user. Below are links to both the recording as well as the deck.

If you were not able to watch Ravi's session live, I highly recommend giving the session a watch and downloading the deck as it contains a ton of useful nuggets!

After re-watching Ravi's session on auditing vSphere API usage, I thought it would be cool to automate the manual process he had outlined. With that, I created a PowerShell script called vSphereAPIUsage.ps1 which contains a single function called Get-vSphereAPIUsage. This script requires access to the vpxd.log which a user will need to download from vCenter Server by either running a VC Support bundle or manually retrieving it from the vCenter Server. In addition, you will need to also provide the user session ID that you wish to query. In Ravi's session, he pointed users to the vpxd-profiler.log but I had found that this can also be found within the vpxd.log which saves users from having to look at another file.

Once you have downloaded the vpxd.log locally on your system, go ahead and open it up with your favorite text editor. I highly recommend Microsoft Visual Studio Code, if you do not have one handy or prefer something beyonds notepad or vi. You will need to search for the particular user you wish to perform the query and the string to search for should look like the following (replace with your SSO or AD domain and username)

[Auth]: User VSPHERE.LOCAL\Administrator

I would also recommend searching from the bottom up as you may want the last login from this particular user. Once you have identified the line, you then need to go up three lines until you see "vim.SessionManager.loginByToken" entry and to the right of that (highlighted in green) is the session ID that you need to make a note of. You can also use the opID value to ensure the session ID is in fact related to this login as you may have other log entries in between.

After making a note of the session ID, you can simply call the Get-vSphereAPIUsage and provide it the full path to the downloaded vpxd.log and the session ID that you had retrieved above. Here is an example using the session ID from the screenshot above:

Get-vSphereAPIUsage -VpxdLogFile "C:\Users\lamw\Dropbox\vpxd.log" -SessionId "52bb9a98-598d-26e9-46d0-ee85d3912646"

The results of the script is a tally of all the different vSphere APIs that have been invoked by this particular session/user and its frequency from lowest to highest. In the example above, I had created a new Datacenter entity, created a couple of Clusters, created several VMs, powered on/off and created/deleted snapshot. These operations were all invoked using the vSphere H5 Client, so there will be other vSphere APIs that are in-directly used by the UI such as inventory lookup that may show up. Hopefully this script will come in handy for those that are interested in this information and beats going through the vpxd.log line by line 🙂

Lastly, Ravi also mentioned that you can use the vSphere Flex/H5 Client to get useful information for a given vCenter Server Session such as the client IP Address as well as the number of API invocations. These details can also be retrieved by using the vSphere API itself, have a look at this article here which provides more details.