Nested Virtualization

Simplified Nested ESXi installation in ESXi 7.0 Update 2 using HTTP Boot over VirtualEFI

03/22/2021 by William Lam 17 Comments

Deploying an ESXi scripted installation aka Kickstart running within a VM (Nested ESXi) has a number of benefits, especially for testing and development purposes. This was something I did regularly as a customer, especially with new releases of ESXi to ensure our existing automation scripts and processes continued to work before rolling out into production. ESXi kickstart itself is pretty straight forward, but the required supporting infrastructure (PXE Server, DHCP, TFTP, etc) that needs to be configured, especially for a greenfield deployment can often be challenging for new comers.

Even with an existing PXE infrastructure, it can often be difficult to configure or troubleshoot depending on your level of access which does not add any value in actually testing or automating the ESXi scripted installation process. In ESXi 7.0 Update 2, an enhancement was made to the Virtual Machine's UEFI firmware called VirtualEFI that would enable ESXi to perform an HTTP Boot given the ESXi bootloader URL and without requiring any of the traditional PXE infrastructure.

To take advantage of this new capability, you just need to have a physical server running ESXi 7.0 Update 2 and a VM that is configured with the latest vHW19 compatibility. To configure HTTP boot, you will need to add the following two VM Advanced Settings:

networkBootProtocol - httpv4 or httpv6
networkBootUri - HTTP URL to the ESXi bootloader (bootx64.efi)

Disclaimer: Nested ESXi and Nested Virtualization is not officially supported by VMware

Automated Nested Lab Deployment on SDDC Part 4: Oracle Cloud VMware Solution

10/30/2020 by William Lam Leave a Comment

In this final blog series, we are now going to take a look at deploying our Nested Lab in an Oracle Cloud VMware Solution environment. Similiar to GCVE, we also will have the ability to run Nested vSAN on top of the physical vSAN within OCVS and by default, customers are provided with the full administrator credentials. For networking, customers also have full access to the NSX-T Manager instance and this means that MAC Learning can be enabled and will allow inner-guest workloads will be able to communicate properly within and outside of the Nested Lab deployment.

Disclaimer: Nested ESXi is not officially supported on Oracle Cloud VMware Solution or by VMware.

Automated Nested Lab Deployment on SDDC Part 3: Google VMware Cloud Engine

10/29/2020 by William Lam Leave a Comment

In Part 3 of this series, we will be taking a look at deploying our Nested Lab in a Google Cloud VMware Engine environment. One thing you may notice that is different between VMConAWS and AVS is that we are actually running a Nested vSAN on top of the physical vSAN within GCVE. In the current version of GCVE, customers have the ability to "escalate" their privileges from the default and this would allow us to configure the necessary ESXi advanced setting so that we do not have to rely on an NFS setup. For networking, customers also have full access to the NSX-T Manager instance and this means that MAC Learning can be enabled which will allow inner-guest workloads will be able to communicate properly within and outside of the Nested Lab deployment.

Disclaimer: Nested ESXi is not officially supported on Google Cloud VMware Engine or by VMware.

Automated Nested Lab Deployment on SDDC Part 2: Azure VMware Solution

10/28/2020 by William Lam Leave a Comment

In Part 2 of this series, we will be taking a look at deploying our Nested Lab in an Azure VMware Solution environment. Simliar to VMConAWS, we will need to deploy our NFS Photon OS OVA because administrative access to the physical ESXi hosts is not available. This is required to enable an advanced ESXi setting to be able to run Nested vSAN on top of the physical vSAN. From a networking standpoint, customers do have full access to the NSX-T Manager instance and this means that MAC Learning can be enabled which will allow inner-guest workloads will be able to communicate properly within and outside of the Nested Lab deployment.

Disclaimer: Nested ESXi is not officially supported on Azure VMware Solution or by VMware.

Automated Nested Lab Deployment on SDDC Part 1: VMware Cloud on AWS

10/27/2020 by William Lam 2 Comments

While preparing for this years VMworld, I had the unique opportunity to work across a number of VMware Cloud SDDC solutions such as VMware Cloud on AWS (VMConAWS), Azure VMware Solution (AVS), Google Cloud VMware Engine (GCVE) and Oracle Cloud VMware Solution (OCVS). During the the development of several demos, I found it invaluable to be able to deploy a Nested vSphere environment to validate my configurations prior to connecting our real vSphere on-premises infrastructure.

Putting aside VMworld demos, this can certainly be extended to other use cases such as accelerated pilots, proof of concepts and lab/development purposes. Customers have been leveraging Nested Virtualization technology for more than a decade plus now and it definitely makes sense that they would also want to do the same for certain workloads running within a VMware Cloud SDDC. With that said, Nested ESXi is not officially supported by VMware or on any other VMware-based platform.

As part of building my VMworld demos, I also had spent some time on creating some automation that would make it easier for me to re-deploy these Nested Lab environments and also being aware of the specific VMware Cloud SDDC solutions, so that I only have a single script to maintain. In case folks are also interested in being able to do this, you can follow this 4-part blog series which I will be kicking it off with VMware Cloud on AWS (VMConAWS).

Heads Up – Nested ESXi crashes in ESXi 7.0 running on older CPUs

04/17/2020 by William Lam 27 Comments

Thanks to Patrik Kernstock, who works in our Technical Support organization at VMware, for making me aware of an issue related to Nested ESXi running on an ESXi host that has been upgraded to ESXi 7.0. Several folks in the community have noticed after upgrading their Intel NUC 7th Gen and deploying a Nested ESXi VM and powering on an inner-guestOS would causes the Nested ESXi VM to crash.

Upon further investigation, it looks like this is not specific to the Intel NUC platform but rather with a specific generation of CPUs which are Intel Sky Lake-based and as a result, some customers are noticing this affect on their 7th Gen NUC.

UPDATE (06/23/20) - ESXi 7.0b has just been released and contains the fix for the Nested ESXi VM crash. If you are using an Intel NUC 10, do not just apply the patch as the updated ne1000 VIB within the patch will override existing Intel NIC driver causing the network adapter to no longer function. It is recommended that you download the patch and replace the default ne1000 VIB using Image Builder with the Intel NIC version before applying the update. To download the patch, please visit VMware Patch Portal site.

The good news is that this issue has already been reported and we should have a fix in a future update of ESXi. In the meantime, you can still run Nested ESXi and Nested Virtualization on these affected CPUs, you just will not be able to power on inner-guest VMs. Big thanks to Patrik for helping out with the testing and triaging this internally.