Many of us who run Nested ESXi in our home labs for development/testing purposes are pretty familiar with the requirements to properly setup a Nested ESXi environment such as CPUs supporting both Intel-VT+EPT or AMD-V+RVI and enabling both Promiscuous Mode and Forged Transmits on the portgroup that your Nested ESXi VM is connected to. Though these requirements have become second nature to most of us, it may not always be obvious on why they are required, especially for new users of Nested ESXi.
UPDATE 09/01/2014 – Take a look at this article for an updated solution to the problem mentioned below.
I specifically wanted to focus on the networking requirements where both Promiscuous Mode and Forged Transmits are required to be enabled. At a high level, most of us have understood this as a prerequisite for proper network connectivity for the Nested Virtual Machines running inside of your Nested ESXi host, but why is that?
Both VMware VSS (Virtual Standard Switch) and VDS (vSphere Distributed Switch) do not implement MAC Learning like a traditional network switch, since the vSphere platform already knows which MAC addresses are assigned to a particular Virtual Machine. This means that the virtual switch will only forward network packets to a Virtual Machine if the destination MAC Address matches the ESXi vmnic’s (pNIC) MAC Address.
In a Nested ESXi environment where you can have Nested Virtual Machines, the destination MAC Address for network packets destined to those Virtual Machines will differ from the Nested ESXi vmnic’s MAC Address. Due to this, the physical ESXi host’s virtual switch will drop the packet if Promiscuous Mode is not enabled. Promiscuous Mode allows the underlying Nested ESXi VM vmnic to monitor all traffic of the virtual switch it is connected to and thus providing connectivity to the underlying Nested Virtual Machines.
An interesting observation was recently made by Anthony Spiteri with his article about Reduced Network Throughput with Promiscuous Mode PortGroups. Since Promiscuous Mode allows all traffic from the virtual switch to be visible on the configured portgroup, there is definitely going to be some amount of overhead when enabling this setting. If you drive a large amount of network traffic for your regular Virtual Machines, you may want to consider separating out your Nested ESXi environment.
Chris Wahl has already written an excellent article on Forged Transmits and its implication with Nested ESXi. I highly recommend you check out this blog post for the details.