Nested Virtualization

Configure NSX-T Enhanced Data path / Network Stack (ENS) for Nested ESXi

12/10/2019 by William Lam Leave a Comment

After publishing my Running Nested ESXi, NSX-V or NSX-T on top of NSX-T article which actually turned out to be quite popular, I received an interesting question on whether ENS for NSX-T could also be configured within a Nested ESXi deployment? I was a little familiar with ENS, which I will explain in a second. However, I was not completely sure about the benefits of running ENS in a Nested environment.

With the help from my friend Frank Escaros-Buechsel, who actually works in our NFV group at VMware. Frank helped validate the instructions but he also provided some additional insights on why this could be useful in a lab setup for verifying configuration and behaviors when additional tuning maybe required. If you are NOT running NFV-based workloads, ENS is not something you need to configure when running NSX-T using Nested ESXi.

So what is ENS? Enhanced Network Stack (ENS) also referred to as Enhanced Data Path is an NSX-T capability which was first introduced with NSX-T 2.3. ENS is specifically designed for Network Function Virtualization (NFV) workloads that require a high performance data path. Such workloads include Telco, 5G and IoT based deployments where improved packet throughput is critical for the responsiveness of these applications.

Running Nested ESXi, NSX-V or NSX-T on top of NSX-T

11/22/2019 by William Lam 5 Comments

Nested Virtualization is an extremely useful tool that helps customers easily test and try out new VMware products and solutions before rolling that into a proper development environment for further validation. This is especially handy for those wanting to setup an NSX-based environment and simulating their actual deployment topology, configuration and upgrade workflows.

In this past year, I have seen a 10x increase in the number of NSX-T based questions that have come up from customers and our field, the adoption of NSX-T is definitely in full swing. As expected, questions about running Nested ESXi on top of a physical NSX-T deployment has come up and there has actually been several variations that have been asked about whether that is Nested ESXi using VSS, VDS, NSX-V or even NSX-T running on top of an N-VDS, which is the virtual switch that NSX-T uses.

Luckily all of these combinations work and just require some basic configuration changes within NSX-T. However, before I continue, let me remind folks again that VMware does NOT officially support Nested Virtualization.

ESX 3.x on VMware Cloud on AWS?

10/10/2019 by William Lam 2 Comments

VMworld Barcelona is just around the corner and this week I started working on building out the different demo environments which will all be running on VMware Cloud on AWS. In one of the demos, I need to have ESX 3.0 running, yes you read that correctly! ESX as in the original version with the Service Console (cos), some of you maybe too young to remember these good ol days? 😉

First, Let me be clear, there really is no good reason for this except for the nostalgia purposes and for what I am trying to demonstrate in our VMworld session. If you are curious about the demo and attend VMworld, be sure to sign up for HBI1967BE Workload Migration Techniques for On-Premises and Cloud Infrastructures which I will be co-presenting with Emad Younis. Secondly, Nested Virtualization whether it is the latest version of ESXi or our very first release, is not officially supported.

Nested ESXi on VMware Cloud on AWS (VMC)

09/28/2018 by William Lam 5 Comments

I have had a few folks ask about Nested ESXi support on VMware Cloud on AWS (VMC), so lets get that out of the way first. Nested ESXi is NOT supported by VMware on any of our platforms, whether that is an on-premises or a cloud environment like VMC or any 3rd party vendors that maybe using VMware software. For those wanting an "official" statement on Nested ESXi support, you can refer to KB 2009916.

Now, we all know Nested ESXi works and it runs extremely well on vSphere. In fact, vSphere is the best platform for running any Hypervisor in a VM. This is also true for VMC, you can run a Nested ESXi VM in an SDDC, however there are some caveats compared to what you would experience in an on-prem environment. Below are some of the caveats to be aware of if you are considering running Nested ESXi on VMC.

How to simulate Persistent Memory (PMem) in vSphere 6.7 for educational purposes?

05/24/2018 by William Lam 4 Comments

A really cool new capability that was introduced in vSphere 6.7 is the support for the extremely fast memory technology known as non-volatile memory (NVM), also known as persistent memory (PMem). Customers can now benefit from the high data transfer rate of volatile memory with the persistence and resiliency of traditional storage. As of this blog post, both Dell and HP have Persistent Memory support and you can see the list of supported devices and systems here and here.

PMem can be consumed in one of two methods:

Virtual PMem (vPMem) - In this mode, the GuestOS is actually PMem-aware and can consume the physical PMem device on the ESXi host as standard, byte-addressable memory. In addition to using an OS that supports PMem, you will also need to ensure that the VM is running the latest Virtual Hardware 14
Virtual PMem Disks (vPMemDisk) - In this mode, the GuestOS is NOT PMem-aware and does not have access to the physical PMem device. Instead, a new virtual PMem hard disk can be created and attached to a VM. To ensure the PMem hard disk is placed on the PMem Datastore as part of this workflow, a new PMem VM Storage Policy will be applied automatically. There are no additional GuestOS or VM Virtual Hardware requirement for this scenario, this is great for legacy OS that are not PMem-aware

Customers who may want to familiarize themselves with these new PMem workflows, especially for Automation or educational purposes, could definitely benefit from the ability to simulate PMem in their vSphere environment prior to obtaining a physical PMem device. Fortunately, this is something you can actually do if you have some additional spare memory from your physical ESXi host.

Disclaimer: This is not officially supported by VMware. Unlike a real physical PMem device where your data will be persisted upon a reboot, the simulated method will NOT persist your data. Please use this at your own risk and do not place important or critical VMs using this method.

Native MAC Learning in vSphere 6.7 removes the need for Promiscuous mode for Nested ESXi

04/25/2018 by William Lam 28 Comments

Over the years, several solutions have been developed here and here to help reduce the impact of promiscuous mode, which is a requirement for running Nested ESXi as a workload. Although these solutions worked extremely well, it however did require users to install additional software to enable this functionality. The most recent solution was a new Learnswitch VMkernel module (released as a VMware Fling) that enables MAC learning capabilities on ESXi.

Today, I am pleased to announce that with the release of vSphere 6.7, the MAC Learning functionality is now available as a native feature of the VMware Distributed Virtual Switch (VDS) and as some of you may have guessed from the title, promiscuous mode is also no longer a requirement for running Nested ESXi! I wanted to take a moment and thank Subin, Jobin, Sriram, Rajeev & Samuel from our Network and Security Business Unit (NSBU) at VMware who worked tirelessly to get this integrated and productized into ESXi. Not only will this benefit Nested ESXi workloads but also other solutions and use cases that have historically required the use of promiscuous mode. For customers who are still running ESXi 6.0 or 6.5, you should continue to use the Learnswitch Fling until you fully upgrade to vSphere 6.7.

To use the new MAC Learning functionality, you will of course need to upgrade to vSphere 6.7 (both vCenter and ESXi) but also upgrade to the latest VDS version which is 6.6. MAC Learning can be enabled on a per Distributed Virtual Portgroup bases and today, it is only available when using the vSphere API. For those that have used the VDS API to manage their VDS, you will simply use the existing ReconfigureDVPortgroup_Task() method and in 6.7, there now a new macManagementPolicy property which allows you to enable and define your MAC Learning settings. This new MAC Management Policy will also be the new preferred method for managing security policies going forward for a DV Portgroup and the previous security policy settings should no longer be used.

Disclaimer: Nested ESXi is still not officially supported by VMware. Please use at your own risk. [Read more...] about Native MAC Learning in vSphere 6.7 removes the need for Promiscuous mode for Nested ESXi