Nested Virtualization

Heads Up – Nested ESXi crashes in ESXi 7.0 running on older CPUs

04/17/2020 by William Lam 26 Comments

Thanks to Patrik Kernstock, who works in our Technical Support organization at VMware, for making me aware of an issue related to Nested ESXi running on an ESXi host that has been upgraded to ESXi 7.0. Several folks in the community have noticed after upgrading their Intel NUC 7th Gen and deploying a Nested ESXi VM and powering on an inner-guestOS would causes the Nested ESXi VM to crash.

Upon further investigation, it looks like this is not specific to the Intel NUC platform but rather with a specific generation of CPUs which are Intel Sky Lake-based and as a result, some customers are noticing this affect on their 7th Gen NUC.

UPDATE (06/23/20) - ESXi 7.0b has just been released and contains the fix for the Nested ESXi VM crash. If you are using an Intel NUC 10, do not just apply the patch as the updated ne1000 VIB within the patch will override existing Intel NIC driver causing the network adapter to no longer function. It is recommended that you download the patch and replace the default ne1000 VIB using Image Builder with the Intel NIC version before applying the update. To download the patch, please visit VMware Patch Portal site.

The good news is that this issue has already been reported and we should have a fix in a future update of ESXi. In the meantime, you can still run Nested ESXi and Nested Virtualization on these affected CPUs, you just will not be able to power on inner-guest VMs. Big thanks to Patrik for helping out with the testing and triaging this internally.

Automated vSphere 7 and vSphere with Kubernetes Lab Deployment Script

04/13/2020 by William Lam 69 Comments

I know many of you have been asking me about my vSphere with Kubernetes automation script which I had been sharing snippets of on Twitter. For the past couple of weeks, I have been hard at work making the required changes between the vSphere 7 Beta and GA workflows, some additional testing and of course documentation. Hopefully the wait was worth it (I think it is) and if you enjoy the script or have benefited, please consider adding 🌟to the Github repo to show your support! Thanks and enjoy

Had to make some updates to one of my vGhetto Automated Lab Deployment Scripts

💥44min to automate all required #vSphere7 infrastructure! 🤛🎤🥳

1 x VCSA 7.0
3 x ESXi + vSAN 7.0
1 x NSX-T 3.0 UA
1 x NSX-T Edge

Need to clean up #ProjectPacific wording but its working great! pic.twitter.com/ZInPgVgbGS

— William Lam (@lamw) April 4, 2020

The Github repository:

https://github.com/lamw/vghetto-vsphere-with-kubernetes-external-nsxt-automated-lab-deployment

Before getting started, please carefully read through the requirements section along with the complete sample end-to-end execution if you are new to vSphere with Kubernetes. You will need to have a VMware Cloud Foundation (VCF) 4.0 license before you can get started and specifically an NSX-T Advance license which is one of the required parameters within the script. If you do not have access to a VCF 4 license, I strongly recommend taking part in the recent VMUG Advantage Homelab Group Buy effort which I had started to easily get access to the latest VMware releases along with a nice 15% discount!

The script supports deploying both a standard vSphere 7 environment with just VCSA, ESXi and vSAN as well as the complete solution which includes NSX-T to support vSphere with Kubernetes. For more details, please refer to the FAQ.

Configure NSX-T Enhanced Data path / Network Stack (ENS) for Nested ESXi

12/10/2019 by William Lam Leave a Comment

After publishing my Running Nested ESXi, NSX-V or NSX-T on top of NSX-T article which actually turned out to be quite popular, I received an interesting question on whether ENS for NSX-T could also be configured within a Nested ESXi deployment? I was a little familiar with ENS, which I will explain in a second. However, I was not completely sure about the benefits of running ENS in a Nested environment.

With the help from my friend Frank Escaros-Buechsel, who actually works in our NFV group at VMware. Frank helped validate the instructions but he also provided some additional insights on why this could be useful in a lab setup for verifying configuration and behaviors when additional tuning maybe required. If you are NOT running NFV-based workloads, ENS is not something you need to configure when running NSX-T using Nested ESXi.

So what is ENS? Enhanced Network Stack (ENS) also referred to as Enhanced Data Path is an NSX-T capability which was first introduced with NSX-T 2.3. ENS is specifically designed for Network Function Virtualization (NFV) workloads that require a high performance data path. Such workloads include Telco, 5G and IoT based deployments where improved packet throughput is critical for the responsiveness of these applications.

Running Nested ESXi, NSX-V or NSX-T on top of NSX-T

11/22/2019 by William Lam 6 Comments

Nested Virtualization is an extremely useful tool that helps customers easily test and try out new VMware products and solutions before rolling that into a proper development environment for further validation. This is especially handy for those wanting to setup an NSX-based environment and simulating their actual deployment topology, configuration and upgrade workflows.

In this past year, I have seen a 10x increase in the number of NSX-T based questions that have come up from customers and our field, the adoption of NSX-T is definitely in full swing. As expected, questions about running Nested ESXi on top of a physical NSX-T deployment has come up and there has actually been several variations that have been asked about whether that is Nested ESXi using VSS, VDS, NSX-V or even NSX-T running on top of an N-VDS, which is the virtual switch that NSX-T uses.

Luckily all of these combinations work and just require some basic configuration changes within NSX-T. However, before I continue, let me remind folks again that VMware does NOT officially support Nested Virtualization.

ESX 3.x on VMware Cloud on AWS?

10/10/2019 by William Lam 2 Comments

VMworld Barcelona is just around the corner and this week I started working on building out the different demo environments which will all be running on VMware Cloud on AWS. In one of the demos, I need to have ESX 3.0 running, yes you read that correctly! ESX as in the original version with the Service Console (cos), some of you maybe too young to remember these good ol days? 😉

First, Let me be clear, there really is no good reason for this except for the nostalgia purposes and for what I am trying to demonstrate in our VMworld session. If you are curious about the demo and attend VMworld, be sure to sign up for HBI1967BE Workload Migration Techniques for On-Premises and Cloud Infrastructures which I will be co-presenting with Emad Younis. Secondly, Nested Virtualization whether it is the latest version of ESXi or our very first release, is not officially supported.

Nested ESXi on VMware Cloud on AWS (VMC)

09/28/2018 by William Lam 5 Comments

I have had a few folks ask about Nested ESXi support on VMware Cloud on AWS (VMC), so lets get that out of the way first. Nested ESXi is NOT supported by VMware on any of our platforms, whether that is an on-premises or a cloud environment like VMC or any 3rd party vendors that maybe using VMware software. For those wanting an "official" statement on Nested ESXi support, you can refer to KB 2009916.

UPDATE (02/10/20) - Updated my Automated vSphere Lab Deployment Script to support "basic" Nested vSphere environment running on VMC.

Was asked if it was possible to setup a “basic” Nested vSphere environment for Automation/API testing running in #VMWonAWS

Just updated my Automated vSphere Deployment Lab Script https://t.co/50tJmcDcH2 to enable support for #VMC 😊 pic.twitter.com/n0SeNIaJm8

— William Lam (@lamw) February 10, 2020

Now, we all know Nested ESXi works and it runs extremely well on vSphere. In fact, vSphere is the best platform for running any Hypervisor in a VM. This is also true for VMC, you can run a Nested ESXi VM in an SDDC, however there are some caveats compared to what you would experience in an on-prem environment. Below are some of the caveats to be aware of if you are considering running Nested ESXi on VMC.