vSphere Client Login UI customizations do not persist in VCSA 6.5 Update 1

The much anticipated release of vSphere 6.5 Update 1 just GA'ed late last week and like many in the community, I also went ahead and upgraded my home lab to this latest release. vSphere 6.5 Update 1 contains a ton of fixes as well as several new capabilities which you can read all about in the release notes here and here.

One neat little trick I take advantage of in my lab environments when deploying the vCenter Server Appliance (VCSA) is actually pre-filling out the credentials for the vSphere Client UI (not recommended for production environments of course) which I had blogged about here a few years back. Sine I have many different environments for different scenarios, I find myself being lazy and having to type in the credentials to each one of these environments. Instead, I can pre-fill either the username and/or password (which I will stress again, not recommended for production) within the vSphere Client Login UI page which is simply just using HTML.

After making the necessary changes to my VCSA 6.5u1 system, I needed to reboot my ESXi host and when everything came back up, I was surprised to find my changes to the vSphere Client Login UI had disappeared. It took me awhile to figure out why the changes were not persisting across reboots. There seems to be a change in behavior compared to prior releases of the VCSA (6.0 & 6.5) on when this capability was actually possible.

Continue reading

PowerCLI script to help correlate vCenter, ESXi & vSAN build/versions w/o manual VMware KB lookup

I can still remember when I was a VI Admin and how annoying it was to try to correlate the build numbers for my ESX(i) hosts and vCenter Servers that I have deployed with the versions listed on VMware's website. This especially gets challenging when there are multiple patch releases (a, b, c or 01, 02, 03) in between major releases (5.5, 6.0, 6.0u1, 6.0u2, 6.5, etc.). Historically, most customers including myself would retrieve the respective build numbers and then manually comparing them to either the release notes and/or download website which was very tedious.

Although VMware has exposed the version number within our vSphere products since day 1 which can also be retrieved programmatically using the vSphere API (here), it unfortunately does not provide more details than simply the major/minor version (e.g. 5,5, 6.0, 6.5, etc) of the software. Recently, VMware had released a series of VMware KBs which provides a mapping between the build numbers for vCenter Server, ESXi and vSAN to their respective versions which can be found in the links below:

These are definitely a great set of resources that I know many customers including myself have been using since its release. Having said that, the process today is still pretty manual since you need to manually retrieve the build numbers for either a VC, ESXi or vSAN Host (can be automated using vSphere APIs) and then comparing that to the KBs to get the correct versions. How cool would it be if you could *easily* just point to YOUR environment and retrieve the version information for either a vCenter Server (Windows or VCSA), ESXi host(s) or vSAN host(s) without needing to manually perform this lookup each time? Well, I have just done that! I have taken all three KBs and converted that information into a simple PowerCLI script called VCESXivSANBuildVersion.ps1 leveraging our vSphere API and it provides three functions:

  • Get-VCVersion - Retrieves the vCenter Server version given a VC connection
  • Get-ESXiVersion - Retrieves the ESXi version for all hosts given a vSphere Cluster
  • Get-VSANVersion - Retrieves the vSAN version for all hosts given a vSAN Cluster

Here is an example output using the first two functions:

For the vCenter Server version output, you will notice that I am also including the OS platform of your vCenter Server, so you can distinguish between a Windows vCenter Server and a vCenter Server Appliance (VCSA) which can be useful to see if you have been #migrate2vcsa ;). For the ESXi version output, you will notice the "OriginalInstallDate" value, this is actually new API property that was introduced in vSphere 6.5 and it provides you with the original installation date of your ESXi host (more details can be found here) which is pretty neat.

Here is an example output using last function:

If you wanted to take this a step further, you could even take this output and dynamically update the vSphere UI using either Custom Attributes or vSphere Tags so you know what version the software is at any given moment. Its easy enough to set this up as a scheduled task that could run periodically so you always have the latest information provided in the vSphere UIs.

Although this is a significant improvement over the existing manual methods, I think most of you will agree that it would be ideal if this information was natively available within the product which means BOTH UI and APIs. I think we all appreciate versioning of software is not always easy and it can change from release to release for a variety of reasons, most of which may not be technical. If the vSphere platform could dynamically pull this information in either real time and/or through an offline mechanism and provide this association by default, it would greatly improve the experience when needing to troubleshoot or perform maintenance of the vSphere platform. If this is something you would like to see, please leave a comment below providing your feedback. I know I have already pinged our PMs about this and I am sure they would love to hear form you as well!

Additional Information:

Note1: Update levels can be found using the vSphere API, take a look at this article here for more details.

Note2: As of ESXi 6.5 Update 1, the Update levels are also included by default in the Embedded Host Client as shown in the screenshot below:

Note3: As of vSAN 6.2, the vSAN Management API already includes vSAN version information that can be queried. Take a look at this script here which exercises this new API. For example above, I decided to not use this new API since customers may be running older releases of vSAN which is not covered by the vSAN Mgmt API.

Note4: VMware has also published simliar build to version mapping for other VMware products which can find the complete list here.

AHCI (vmw_ahci) performance issue resolved in ESXi 6.5 Update 1

For customers who had SATA controllers that consumed the VMware Advanced Host Controller Interface (AHCI) driver found that after upgrading to ESXi 6.5, the disk performance for those devices were significantly impacted. Basic operations such as cloning or uploading an OVF/OVA would literally double if not triple in time. In fact, I too had observed this same behavior when I had upgraded my Intel NUC (not an officially supported platform) to ESXi 6.5. One thing I had noticed at the time when others were reporting simliar issues was that their HW platforms were also not on the VMware HCL, so I was not sure if this was limited to only home-lab environments?

In any case, I and others eventually stumbled onto this blog article by Sebastian Foss who I believe may have been the first to identify a workaround which was to simply disable the new AHCI Native Driver which loads by default and forcing it fall back to using the legacy AHCI driver which made the issue go away after a reboot. Although the folks who had reported seeing simliar issue were all using hardware platforms that were not officially on the VMware HCL, I decided to still file an internal bug and hoped someone could take a look to see what was going on.

With the release of ESXi 6.5 Update 1, I am happy to report the observed performance issues with the Native AHCI driver have now been resolved! I have been running on earlier release of ESXi 6.5 Update 1 build for couple of weeks now and have not seen any of the problems I had before. For those interested, the official fix went is in version 1.0.0-37vmw or greater of the vmw_ahci driver.

You can easily verify for this by running the following ESXCLI command to retrieve the version of your vmw_ahci driver:

If you had disabled the Native AHCI driver, you will definitely want to re-enable it. You can check if its been disabled by running the following ESXCLI command and checking the second column to see if it shows "false":

esxcli system module list | grep vmw_ahci

If the Native AHCI driver is disabled as shown in the previous command, then you can re-enable it by running the following ESXCLI command:

esxcli system module set --enabled=true --module=vmw_ahci

Once you have re-enabled the driver, you will need to reboot for the changes to go into effect.

Quick Tip - Locating SRM Placeholder VMs using the vSphere API

I had received a question the other day from a reader where they were trying to distinguish between the running VM and its placeholder VM due to their use of VMware Site Recovery Manager (SRM). Since the VM name is exactly the same in both vCenter Servers, it was not clear how to identify between the two. As mentioned in my reply to the reader, there are a couple of ways. You could use the SRM API in-conjunction with the vSphere API (in his case, he was using PowerCLI) to be able to check whether the VM in question was the placeholder VM or not.

Another option is to simply use the vSphere API and querying for the managedBy property which is populated when SRM and/or other solutions are associated with managing a set of VMs. In the case of SRM, you will see an extensionKey with value of "com.vmware.vcDR" and type with value of "placeholderVm" which tells you that the VM is an SRM Placeholder VM, pretty easy, right!? 🙂

Since I did not have an SRM environment handy, the next best thing was to check out VMware Hands-On-Lab environment which anyone can access for free. Lab HOL-1705-SDC-1 was exactly what I needed and here is a quick screenshot of the vSphere MOB showing you what the managedBy property looks like in the vSphere API.

To demonstrate the use of this vSphere API, I wrote a quick PowerCLI function called PlaceholderVMs.ps1 and below is an example of running the Get-PlaceholderVM command: