virtuallyGhetto

Nested ESXi

Heads Up – Nested ESXi crashes in ESXi 7.0 running on older CPUs

by 27 Comments

Thanks to Patrik Kernstock, who works in our Technical Support organization at VMware, for making me aware of an issue related to Nested ESXi running on an ESXi host that has been upgraded to ESXi 7.0. Several folks in the community have noticed after upgrading their Intel NUC 7th Gen and deploying a Nested ESXi VM and powering on an inner-guestOS would causes the Nested ESXi VM to crash.

Upon further investigation, it looks like this is not specific to the Intel NUC platform but rather with a specific generation of CPUs which are Intel Sky Lake-based and as a result, some customers are noticing this affect on their 7th Gen NUC.

UPDATE (06/23/20) - ESXi 7.0b has just been released and contains the fix for the Nested ESXi VM crash. If you are using an Intel NUC 10, do not just apply the patch as the updated ne1000 VIB within the patch will override existing Intel NIC driver causing the network adapter to no longer function. It is recommended that you download the patch and replace the default ne1000 VIB using Image Builder with the Intel NIC version before applying the update. To download the patch, please visit VMware Patch Portal site.

The good news is that this issue has already been reported and we should have a fix in a future update of ESXi. In the meantime, you can still run Nested ESXi and Nested Virtualization on these affected CPUs, you just will not be able to power on inner-guest VMs. Big thanks to Patrik for helping out with the testing and triaging this internally.

Running Nested ESXi, NSX-V or NSX-T on top of NSX-T

by 6 Comments

Nested Virtualization is an extremely useful tool that helps customers easily test and try out new VMware products and solutions before rolling that into a proper development environment for further validation. This is especially handy for those wanting to setup an NSX-based environment and simulating their actual deployment topology, configuration and upgrade workflows.

In this past year, I have seen a 10x increase in the number of NSX-T based questions that have come up from customers and our field, the adoption of NSX-T is definitely in full swing. As expected, questions about running Nested ESXi on top of a physical NSX-T deployment has come up and there has actually been several variations that have been asked about whether that is Nested ESXi using VSS, VDS, NSX-V or even NSX-T running on top of an N-VDS, which is the virtual switch that NSX-T uses.

Luckily all of these combinations work and just require some basic configuration changes within NSX-T. However, before I continue, let me remind folks again that VMware does NOT officially support Nested Virtualization.

[Read more...] about Running Nested ESXi, NSX-V or NSX-T on top of NSX-T

ESX 3.x on VMware Cloud on AWS? 

by 2 Comments

VMworld Barcelona is just around the corner and this week I started working on building out the different demo environments which will all be running on VMware Cloud on AWS. In one of the demos, I need to have ESX 3.0 running, yes you read that correctly! ESX as in the original version with the Service Console (cos), some of you maybe too young to remember these good ol days? 😉

First, Let me be clear, there really is no good reason for this except for the nostalgia purposes and for what I am trying to demonstrate in our VMworld session. If you are curious about the demo and attend VMworld, be sure to sign up for HBI1967BE Workload Migration Techniques for On-Premises and Cloud Infrastructures which I will be co-presenting with Emad Younis. Secondly, Nested Virtualization whether it is the latest version of ESXi or our very first release, is not officially supported.

[Read more...] about ESX 3.x on VMware Cloud on AWS? 

Nested ESXi on VMware Cloud on AWS (VMC)

by 5 Comments

I have had a few folks ask about Nested ESXi support on VMware Cloud on AWS (VMC), so lets get that out of the way first. Nested ESXi is NOT supported by VMware on any of our platforms, whether that is an on-premises or a cloud environment like VMC or any 3rd party vendors that maybe using VMware software. For those wanting an "official" statement on Nested ESXi support, you can refer to KB 2009916.

UPDATE (02/10/20) - Updated my Automated vSphere Lab Deployment Script to support "basic" Nested vSphere environment running on VMC.

Now, we all know Nested ESXi works and it runs extremely well on vSphere. In fact, vSphere is the best platform for running any Hypervisor in a VM. This is also true for VMC, you can run a Nested ESXi VM in an SDDC, however there are some caveats compared to what you would experience in an on-prem environment. Below are some of the caveats to be aware of if you are considering running Nested ESXi on VMC.


[Read more...] about Nested ESXi on VMware Cloud on AWS (VMC)

How to simulate Persistent Memory (PMem) in vSphere 6.7 for educational purposes? 

by 4 Comments

A really cool new capability that was introduced in vSphere 6.7 is the support for the extremely fast memory technology known as non-volatile memory (NVM), also known as persistent memory (PMem). Customers can now benefit from the high data transfer rate of volatile memory with the persistence and resiliency of traditional storage. As of this blog post, both Dell and HP have Persistent Memory support and you can see the list of supported devices and systems here and here.


PMem can be consumed in one of two methods:

  • Virtual PMem (vPMem) - In this mode, the GuestOS is actually PMem-aware and can consume the physical PMem device on the ESXi host as standard, byte-addressable memory. In addition to using an OS that supports PMem, you will also need to ensure that the VM is running the latest Virtual Hardware 14
  • Virtual PMem Disks (vPMemDisk) - In this mode, the GuestOS is NOT PMem-aware and does not have access to the physical PMem device. Instead, a new virtual PMem hard disk can be created and attached to a VM. To ensure the PMem hard disk is placed on the PMem Datastore as part of this workflow, a new PMem VM Storage Policy will be applied automatically. There are no additional GuestOS or VM Virtual Hardware requirement for this scenario, this is great for legacy OS that are not PMem-aware

Customers who may want to familiarize themselves with these new PMem workflows, especially for Automation or educational purposes, could definitely benefit from the ability to simulate PMem in their vSphere environment prior to obtaining a physical PMem device. Fortunately, this is something you can actually do if you have some additional spare memory from your physical ESXi host.

Disclaimer: This is not officially supported by VMware. Unlike a real physical PMem device where your data will be persisted upon a reboot, the simulated method will NOT persist your data. Please use this at your own risk and do not place important or critical VMs using this method.

[Read more...] about How to simulate Persistent Memory (PMem) in vSphere 6.7 for educational purposes? 

Leveraging Instant Clone in vSphere 6.7 for extremely fast Nested ESXi provisioning

by 18 Comments

The idea of "Instant Cloning" a Nested ESXi VM (running ESXi in a VM) is not a new concept. In fact, I had shared a solution back in 2015 using the private VMFork APIs. However, what has changed is the ease of consumption, primarily due to the re-architecture of Instant Clone in vSphere 6.7 (more details here and here) which resulted in a public and simplified API. Some of you might ask, why not simply clone a Nested ESXi VM or create a Link Clone? What benefit would I get by using Instant Clone?

The answer is not only speed, but the fact that the instantiated VM is fully operational and ready to start executing where as a traditional full clone or linked clone requires a full OS boot up that can take up to several minutes to deploy and configure. This may not sound like much for a small number of Nested ESXi VMs, but as you increase the number of instances, Instant Clone really shines while still maintaining speed and the instant availability of the VM. As you can imagine, this definitely opens up for some interesting use cases whether it be for personal home lab or educational purposes like VMware HOL. In addition, we also have customers who deploy Nested ESXi not only at high scale but also with a high churn rate for development purposes, think CI/CD type of a workload who can also benefit from Instant Clone.

So how fast are we talking about? Lets say you wanted to test out the latest version of VSAN in vSphere 6.7, you would normally deploy 3 Nested ESXi VMs, power them up and wait for them to be ready on the network. With Instant Clone, you can deploy three fully functional Nested ESXi VMs in just 30seconds! As the VMs are instantly available for consumption, you can start the VSAN enablement workflow immediately and even parts of that can be baked into the Instant Clone workflow. With the ease of provisioning Nested ESXi VMs, you can simply maintain a catalog of ESXi templates which are in "frozen" states and then leverage Instant Clone to deploy just-in-time Nested ESXi environments and discard them once you are done. Pretty slick if you ask me! and something I plan on using going forward.

Disclaimer: Nested ESXi is still not officially supported by VMware. Please use at your own risk.

[Read more...] about Leveraging Instant Clone in vSphere 6.7 for extremely fast Nested ESXi provisioning