vSphere 7.0

Automated Lab Deployment Script for vSphere with Tanzu using NSX Advanced Load Balancer (NSX ALB)

04/05/2021 by William Lam 2 Comments

After spending a few days playing with the NSX Advanced Load Balancer (NSX ALB) APIs, I am happy to share my latest automation lab deployment script for deploying vSphere with Tanzu using the new NSX ALB which was introduced with the latest vSphere 7.0 Update 2 release.

🙌 BOOM!!!

Fully Automated vSphere with @VMwareTanzu using the new @vmwarensx Advanced Load Balancer introduced in vSphere 7.0 Update 2 Lab Deployment in just 32 minutes! 🔥

Still need to clean up some things, but this beats clicking around the UI! My 🤲 thanks me pic.twitter.com/hN32Qk3oDc

— William Lam (@lamw) March 29, 2021

Lab Deployment Automation

You can find the new automation script along with all the details at the following Github Repo: https://github.com/lamw/vsphere-with-tanzu-nsx-advanced-lb-automated-lab-deployment#enable-workload-management

In my environment, it took about ~32 minutes for the deployment to finish, but YMMV based on the performance of your underlying hardware.

Workload Management Automation

In addition to the automated lab deployment script above, I have also updated my community VMware.WorkloadManagement module to add support for enabling Workload Management on a vSphere Cluster using NSX ALB. This is introduced as a new function creatively called New-WorkloadManagement3. You use the Get-Help cmdlet to get a list of supported arguments or you can take a look at this example.

Simplified Nested ESXi installation in ESXi 7.0 Update 2 using HTTP Boot over VirtualEFI

03/22/2021 by William Lam 17 Comments

Deploying an ESXi scripted installation aka Kickstart running within a VM (Nested ESXi) has a number of benefits, especially for testing and development purposes. This was something I did regularly as a customer, especially with new releases of ESXi to ensure our existing automation scripts and processes continued to work before rolling out into production. ESXi kickstart itself is pretty straight forward, but the required supporting infrastructure (PXE Server, DHCP, TFTP, etc) that needs to be configured, especially for a greenfield deployment can often be challenging for new comers.

Even with an existing PXE infrastructure, it can often be difficult to configure or troubleshoot depending on your level of access which does not add any value in actually testing or automating the ESXi scripted installation process. In ESXi 7.0 Update 2, an enhancement was made to the Virtual Machine's UEFI firmware called VirtualEFI that would enable ESXi to perform an HTTP Boot given the ESXi bootloader URL and without requiring any of the traditional PXE infrastructure.

To take advantage of this new capability, you just need to have a physical server running ESXi 7.0 Update 2 and a VM that is configured with the latest vHW19 compatibility. To configure HTTP boot, you will need to add the following two VM Advanced Settings:

networkBootProtocol - httpv4 or httpv6
networkBootUri - HTTP URL to the ESXi bootloader (bootx64.efi)

Disclaimer: Nested ESXi and Nested Virtualization is not officially supported by VMware

Quick Tip – How to disable the vCenter Server Update Notification banner?

03/17/2021 by William Lam 3 Comments

I received this question on Twitter from Andreas asking the following:

Is there a way to disable or postpone the #vCenter update notification logon message in the web client? @lamw @vmwarecares @VMwarevSphere pic.twitter.com/tYsikiesIP

— Andreas Peetz🛡️ (@VFrontDe) March 17, 2021

When a new vCenter Server update is available, a notification banner is automatically displayed in the vSphere UI. This functionality was introduced as part of vSphere 7.0 and part of the new vSphere Lifecycle Manager (vLCM) capability. This is a very useful feature since administrators spend most of their time in the vSphere UI and when new update was available, it would only be displayed in the VAMI UI, which most folks were not logging into on a regular basis.

Today, the update notification banner is always displayed and there is no way to temporarily disable it. This can be annoying if you do not intend to update your vCenter Server immediately and I assume this is why Andreas was asking about either postponing or disabling the notification all together.

Currently, the only way I am aware of for disabling this notification is to actually disable the vCenter Server Life-Cycle Manager Remote Plugin itself. You can do this by navigating to Administration->Solutions->Client Plugins and then selecting "vCenter Server Life-cycle Manager" and click on the Disable button. You can refresh the webpage or logout and you should no longer see the notification banner.

Disclaimer: By disabling the vCLM plugin, you are disabling more than just the banner but all vCenter vLCM functionality including Interop and Update Planner capabilities. If these are things you require, do not disable the plugin.

I can certainly see a nice feature enhancement in the future where vLCM notifications can be postponed or deferred to a later date. I will share this blog post and feedback with the vLCM PM for consideration.

ESXi 7.0 Update 2 enhancement for USB NIC only installations

03/16/2021 by William Lam 3 Comments

The USB Network Native Driver for ESXi Fling has been an extremely popular Fling that has allowed customers to easily add additional networking capabilities by using a supported USB-based network adapter even though ESXi traffic over USB networking is not officially supported.

In most deployments, the USB network adapter is usually a supplement to the existing onboard network adapter of a system. However, there have been scenarios where the onboard network adapter is either not available or functional and customers would still like to be able to install ESXi and have it running over just the USB network adapter.

Although installing ESXi using just a USB network adapter is possible today, one downside is that an additional workflow is needed to fix the network binding after installing ESXi.

During the interactive ESXi installation, you will see the following error at 81% which will cause installer to get stuck

Exception: No vmknic tagged for management was found.

At this point, the installer has completed and you need to switch to the console (Alt+F1) and perform a reboot. Once ESXi reboots, you will need to go into the DCUI and manually bind the vusb0 interface for ESXi management for connectivity. To persist this USB NIC binding, you will need to add small snippet of code as outlined here.

Obviously, this was not an ideal user experience and I personally had to use this workaround on several occasions, especially for newer hardware platforms where the onboard network adapter may not be recognized by ESXi and being able to use the USB Network Fling definitely came in handy.

With the release of ESXi 7.0 Update 2, we have improved the user experience for installing ESXi with just a single USB NIC. This enhancement was added by Songtao after mentioning the undesirable behavior. A new ESXi kernel boot option called usbBusFullScanOnBootEnabled can be added which removes the need for the workaround mentioned above. This new kernel option forces a full bus scan to claim all USB NICs that are attached since USB device claiming is slow compared to PCIe devices.

Aquantia/Marvell AQtion (Atlantic) driver now inbox in ESXi 7.0 Update 2

03/11/2021 by William Lam 3 Comments

Last spring, VMware and Aquantia (now part of Marvell) collaborated and delivered their first ESXi Native Driver for their AQtion (Atlantic) based 10GbE network adapters. This new driver was primarily focused on enabling network connectivity for ESXi when running on either an Apple 2018 Mac Mini (8,1) and Apple 2019 Mac Pro (7,1) that included the 10GbE networking option. Consequently, this driver also benefited the broader VMware Community as it enabled additional 10GbE networking through a number of Thunderbolt 3 to 10GbE network adapters that customers could now take advantage in their VMware environments.

With all these benefits, VMware has decided to inbox the Aquantia/Marvell driver with the latest ESXi 7.0 Update 2 release, so that customers no longer had to create a custom ESXi Image Profile that included the driver, which was always required when installing ESXi on either the Apple Mac Mini or Mac Pro that were configured with the 10GbE networking option. For a complete list of supported Aquantia/Marvell AQtion based network adapters, please see the VMware HCL.

Here is a screenshot of an earlier release of ESXi 7.0 Update 2 running on the 2018 Mac Mini which now automatically recognizes the 10GbE network adapter out of the box.

How to clean up stale vSphere Container Volumes & First Class Disks?

03/10/2021 by William Lam Leave a Comment

If you are running and deploying Kubernetes (K8s) which includes vSphere with Tanzu and Tanzu Kubernetes Grid (TKG), you might notice vSphere Container Volumes showing up in the vSphere UI under the Monitor tab for a given vSphere-based Datastore. This is normal and expected as new Persistent Volumes (PVs) and Persistent Volume Claims (PVCs) are being requested as part of deploying K8s-based application that require storage.

Typically, when PVs and PVCs are no longer needed, they should be cleaned up within the K8s layer via kubectl either automatically or manually depending on your provisioning process. When you delete a K8s Cluster, these PVs/PVCs are not automatically cleaned up and its for good reason, you may want to reuse them and the way vSphere supports this is by implementing them as First Class Disks (FCD), which means they are lifecycle independent of a VM.

What happens when the K8s Cluster has been deleted and you actually want to clean up these stale FCDs, how do you go about doing that? This is a question I have seen come up more frequently and there are a few options.

Option 1:

If you happen to be on vSphere 7.0 Update 2 (which was just released yesterday), the vSphere UI has been enhanced to allow users to now delete vSphere Container Volume (see screenshot above). Previously, you could only view the FCDs and reapply a storage policy.

Option 2:

Since vSphere Container Volumes are just FCDs and we have FCD APIs, we can use the API to retrieve information as well as clean them up. The easiest way is to use PowerCLI's Get-CnsVolume and Remove-CnsVolume cmdlets.

Here is an example of deleting the 2GB volume:

Get-CnsVolume -Datastore (Get-Datastore "sm-vsanDatastore") -Name "pvc-db6829ad-e1a9-46e8-ace3-7e7c18187a0d" | Remove-CnsVolume

In the case of standalone FCDs, which could have been manually provisioned or through a backup solution, you can also clean them up by using PowerCLI's Get-VDisk and Remove-VDisk cmdlets respectively:

Get-VDisk -Name "fill-me-in" | Remove-VDisk