virtuallyGhetto

vesxi

Nested Virtualization APIs For vSphere & vCloud Director 5.1

by 2 Comments

Did you know with the release of vSphere 5.1 and vCloud Director 5.1, there are now APIs that allow you to enable/disable Nested Virtualization aka (Virtual Hardware Virtualization) on a Virtual Machine?

Disclaimer: Nested Virtualization is not officially supported by VMware.

In the vSphere 5.1 API, there are two new properties related to Nested Virtualization:

  • nestedHVSupported is a new capability property which indicates whether your physical ESXi 5.1 host supports Nested Virtualization. This property is only true IF your physical CPU supports BOTH Intel-VT/EPT OR AMD-V/RVI. For more details, please refer to this article.
  • nestedHVEnabled is new property on a Virtual Machine which allows you to enable or disable Nested Virtualization. You will need to ensure you are running a Virtual Machine with ESXi 5.1 Compatibility (e.g. virtual hardware 9) and your physical ESXi 5.1 host supports Nested Virtualization.

Here is a screenshot of performing the same operation manually using the new vSphere Web Client:

In the vCloud Director 5.1 REST API, there two new operations related to Nested Virtualization:

  • /action/enableNestedHypervisor is a new operation on a Virtual Machine that enables Nested Virtualization. Here is an example using the curl utility enable VHV:

    curl -i -k -H "Accept:application/*+xml;version=5.1" -H "x-vcloud-authorization: xX4IrWWi+Ofq77zOqPJaMEYHHJt4jxrwP+ntkO2tecQ=" -X POST https://vcd.primp-industries.com/api/vApp/vm-d9870545-a29a-4175-bff4-ae075f1c1bc0/action/enableNestedHypervisor

  •  /action/disableNestedHypervisor is a new operation on a Virtual Machine that disables Nested Virtualization. Here is an example using the curl utility to disable VHV:

    curl -i -k -H "Accept:application/*+xml;version=5.1" -H "x-vcloud-authorization: xX4IrWWi+Ofq77zOqPJaMEYHHJt4jxrwP+ntkO2tecQ=" -X POST https://vcd.primp-industries.com/api/vApp/vm-d9870545-a29a-4175-bff4-ae075f1c1bc0/action/disableNestedHypervisor

Here is a screenshot of performing the same operation manually using the vCloud Director UI:

If you plan to leverage Nested Virtualization in your environment, you now have simple way of automating this feature for Virtual Machines that you wish to support Nested Virtualization.

Having Difficulties Enabling Nested ESXi in vSphere 5.1?

by 21 Comments

I noticed there were a few folks having some difficulties enabling Nested ESXi (VHV Virtual Hardware Virtualization) in the latest release of ESXi 5.1 and I thought I share some additional info and tips on troubleshooting your setup in case you are running into similar problems.

*** DISCLAIMER **** This is not officially supported by VMware, do not bother asking if it is supported or calling into VMware support for details or help.

If you wish to run nested ESXi or other hypervisors on ESXi 5.1 and run 32-bit nested virtual machines, you must meet the following hardware requirement:

  • CPU supporting Intel VT-x or AMD-V

If you wish to run nested 64-bit virtual machines in your nested ESXi or other hypervisors, in addition to the requirement above, you must also meet the following hardware requirement:

  • CPU supporting Intel EPT or AMD RVI

If you only meet the first criteria, you CAN still install nested ESXi or other hypervisors on ESXi 5.1, BUT you will only be able to run 32-bit nested virtual machines. When you create your virtual machine shell using the new vSphere Web Client, in the expanded CPU view, the "Hardware Virtualization" box will be grayed out. This is expected as you do not have full support for VHV, but you can still continue with your installation of ESXi or other hypervisors.

In ESXi 5.0, you may have been able to run 64-bit nested virtual machines without EPT/RVI support but performance was extremely poor. With ESXi 5.1, VHV now requires EPT/RVI.

Note: During the installation of ESXi, you may see the following message "No Hardware Virtualization Support", you can just ignore it.

If you are using sites such as Intel's ark.intel.com to check your CPU requirements, be aware that it is COMMON even for the hardware vendors to publish incorrect information about their websites. However, there is a quick way you can validate on your ESXi host whether you have full VHV support.

In vSphere 5.1, there is a new capability property called nestedHVSupported which specifies whether your physical ESXi 5.1 host has full VHV support. This property will only be true IF your CPU has both Intel-VT+EPT or AMD-V+RVI. A quick and easy way to validate this is using the vSphere MOB to retrieve the value.

To check nestedHVSupported property, please enter the following into a web browser (substitute the IP Address/hostname of your ESXi host):

https://himalaya.primp-industries.com/mob/?moid=ha-host&doPath=capability

After you login, search for the nestedHVSupported property on the page and you should see a value of either true or false. As mentioned earlier, if it is false, you might still be able to install nested ESXi or other hypervisors but you will not be able to run nested 64-bit virtual machines. I would also recommend taking a look at your system BIOS to ensure things like Intel-VT/EPT and AMD-V/RVI are enabled and sometimes it might just be as simple as a BIOS upgrade (you can always confirm by contacting the hardware vendor if you have further questions).

For proper networking connectivity, also ensure that either your standard vSwitch or Distributed Virtual Switch has both promiscuous mode and forged transmit enabled either globally on the portgroup or distributed portgroup your nested ESXi hosts are connected to.

Additional Resources: 

Nested ESXi 5.1 Supports VMXNET3 Network Adapter Type

by 14 Comments

I noticed something interesting while extracting the contents of ESXi 5.1 ISO for some kickstart configurations ....

Do you see it? It's a VMXNET3 driver for the VMkernel! I also confirmed by running the following ESXCLI command querying for the VMkernel module "vmxnet3":

# esxcli system module get -m vmxnet3
Module: vmxnet3
Module File: /usr/lib/vmware/vmkmod/vmxnet3
License: GPL
Version: Version 1.1.32.0, Build: 799733, Interface: 9.2 Built on: Aug  1 2012
Signed Status:
Signature Issuer:
Signature Digest:
Signature FingerPrint:
Provided Namespaces:
Required Namespaces: [email protected], [email protected]_1_0_0

***Disclaimer***: This is for educational purposes only, this is not officially supported by VMware. Use at your own risk. There is also a mention of this in the vSphere 5.1 release notes that VMs running on nested ESXi hosts using VMXNET3 driver could potentially crash. Again, not supported user at your own risk.

Next I decided to create a Nested ESXi 5.1 VM, but instead of selecting the e1000 driver which was the only network adapter type that would function for running a nested ESXi host, I choose the VMXNET3 adapter and to my surprise ESXi's networking stack was fully functional.

You can see from the above screenshot, I have a two VMXNET3 network adapters for my nested ESXi 5.1 VM. Here are two additional screenshot of the physical adapters as seen by nested ESXi 5.1 host and you can see that it shows up as VMware Inc. VMXNET3

I have not tried any performance tests, so not sure if there are going to be any significant benefits but pretty cool nonetheless!

How to Enable Nested ESXi & Other Hypervisors in vCloud Director 5.1

by 5 Comments

The process to enable  "Nested Virtualization" in the latest release of vCloud Director 5.1 and create your own virtual lab similar to VMware's vSEL (Virtual Sales Enablement Cloud) is very similar to the previous steps outlined for vCloud Director 1.5 release. The only change is how VHV (Virtual Hardware-Assisted Virtualization) aka "Nested Virtualization" is enabled in vCloud Director 5.1 and ESXi 5.1.

In the vCloud Director 1.5, to enable VHV, you needed to add a special SQL statement that would enable VHV for the underlying ESXi 5.0 hosts. With the latest release of vCloud Director 5.1, that is no longer necessary and you now enable it on a Per VM basis within the vCloud Director 5.1 UI.

Here are the steps for enabling VHV for vCloud Director 5.1

  • Insert SQL statements into VCD Database that perform the following:
    • Enable new "VMware" guestOS Family
    • Enable new guestOS Type ESXi 4.x and 5.x
    • Enable host preparation to enable VHV (vSphere 5.0 & vCloud 1.5 only)
  • Enable promiscuous mode
    • Insert SQL statement into VCD Database for Network Pool that is being used for your ESXi VMs
    • Enable both Promiscuous Mode and Forged Transmit for vSphere Backed Portgroup within vCenter Server or ESXi host

The SQL statements can be found in this article and have not changed for vCloud Director 5.1

Here is a screenshot of what you should see in the vCloud Director 5.1 UI for creating a new VM and you should now have the ability to select a new guestOS Type called "VMware" and select either an ESXi 4.x or ESXi 5.x guestOS Version.

To enable VHV for the VM, you will need to also check the box "Exposed hardware-assisted CPU virtualization to guestOS" and this will allow you to run a nested ESXi VM as well as 64-bit nested VMs, assuming your physical CPUs support it. To learn more about running VHV on ESXi 5.1, take a look at this article here for more details.

How to Enable Nested ESXi & Other Hypervisors in vSphere 5.1

by 88 Comments

There are a ton of new features with the latest release of vSphere 5.1, but the one "unsupported" feature I always test first is "Nested Virtualization" (aka Nested ESXi) and with the latest release, it seems to have gotten even better. You will still need to have the same physical CPU prerequisites as you did in the past to run "Nested Virtualization" as well as nesting 64-bit VMs.

  • Intel VT-x or AMD-V is required for running "Nested Virtualization" which supports nested 32-bit VMs
  • Intel EPT or AMD RVI is required for running nested 64-bit VMs.

A quick way to verify whether your CPU truly supports both Intel-VT+EPT or AMD-V+RVI, you can paste the following into a browser:  

https://[your-esxi-host-ip-address]/mob/?moid=ha-host&doPath=capability

You will need to login with your root credentials and then look for the "nestedHVSupported" property and if it states false, it means you maybe able to install nested ESXi or other hypervisor, but you will not be able to run nested 64-bit VMs, only 32-bit VMs, assuming you have either Intel-VT or AMD-V support on your CPUs.

For more details, take a look at this article for troubleshooting: Having Difficulties Enabling Nested ESXi in vSphere 5.1?

Disclaimer: This is not officially supported by VMware, please use at your own risk.

There are some changes with Nested Virtualization in vSphere 5.1 also officially known as VHV (Virtual Hardware-Assisted Virtualization). If you are using vSphere 5.0 to run Nested ESXi or other nested Hypervisors, then please take a look at the instructions in this article. With vSphere 5.1, there have been a few minor changes to enable VHV.

  1. The new Virtual Hardware 9 compatibility will be required when creating your nested ESXi VM, Virtual Hardware 8 will not work if you are running ESXi 5.1 on your physical host. You will still need to enable promiscuous mode on the portgroup that will be used for your nested ESXi VM for network connectivity.
  2. vhv.allow = "true" is no longer valid for ESXi 5.1 to enable VHV. A new parameter has been introduced called vhv.enable = "true" that is now defined on a per VM basis to provide finer granularity of VHV support. This also allows for better portability between VMware's hosted products such as VMware Fusion and Workstation as they also support the vhv.enable parameter.
  3. You can now enable VHV on a per VM basis and using the new vSphere Web Client which basically adds the vhv.enable = "true" parameter to the VM's .VMX configuration file.

Note: You can run a nested ESXi 5.1 VM on top of a physical ESXi 5.0 host, just follow the instructions here.

Enabling VHV (Virtual Hardware-Assisted Virtualization)

Step 1 - Create a new Virtual Hardware 9 Virtual Machine using the new vSphere Web Client that's available with vCenter Server 5.1.

Step 2 - Select "Linux" as the guestOS Family and "Other Linux (64-bit)" as the guestOS Version.

Step 3 - During the customize hardware wizard, expand the "CPU" section and select "Hardware Virtualization" box to enable VHV.

Note: If this box is grayed out, it means that your physical CPU does not supported Intel VT-x + EPT or AMD-V + RVI which is required to run VHV OR that you are not using Virtual Hardware 9. If your CPU only supports Intel-VT or AMD-V, then you can still install nested ESXi, but you will only be able to run nested 32-bit VMs and not nested 64-bit VMs.

Step 4 - It is still recommended that you change the guestOS Version to VMware ESXi 5.x after you have created the VM shell, as there are some special settings that are applied automatically. Unfortunately with the new vSphere Web Client, you will not be able to modify the guestOS after creation, so you will need to use the C# Client OR manually go into the .VMX and update guestOS = "vmkernel5"

Now you are ready to install nested ESXi VMs as well as run nested 64-bit VMs within.

If you have followed my previous article about How to Enable Support for Nested 64bit & Hyper-V VMs in vSphere 5 you may recall a diagram about the levels of "Inception" that can be performed with nested ESXi. That is, the number of times you could nest ESXi and still have it be in a "functional" state. With vSphere 5.0, the limit that I was able to push was 2 levels of nested ESXi. With latest release of vSphere 5.1, I have been able to push that limit now to an extraordinary 3 levels of inception!

You might ask why would someone want to do this ... well I don't have a good answer other than ... because I can? 😉 VHV is one of the coolest "unsupported" feature in my books and I'm glad it is working beyond what it was designed for.

For proper networking connectivity, also ensure that either your standard vSwitch or Distributed Virtual Switch has both promiscuous mode and forged transmit enabled either globally on the portgroup or distributed portgroup your nested ESXi hosts are connected to.

Nesting "Other" Hypervisors

For those of you who feel inclined to run other hypervisors such as Hyper-V, you can do so with latest release of ESXi 5.1. The process if very straight forward just like running nested ESXi host.

Step 1 - Create a Virtual Hardware 9 VM and select the appropriate guestOS. In this example, I selected Windows Server 2012 (64-bit) as the guestOS version.

Step 2 - Enable VHV under the CPU section if you wish to create and run nested 64-bit VMs under Hyper-V

Step 3 - You will need to add one additional .vmx parameter which tells the underlying guestOS (Hyper-V) that it is not running as a virtual guest which in fact it really is. The parameter is hypervisor.cpuid.v0 = FALSE

How to Configure Nested ESXi 5 to Support EVC Clusters

by 10 Comments

Dave Hill recently wrote an article about running nested ESXi and a gotcha with EVC (Enhanced vMotion Compatibility). In vSphere 4.x, you could not join a nested ESXi host into a cluster with EVC enabled. With vSphere 5, there's actually a way to connect a nested ESXi 5 host to an EVC enabled cluster AND still power on 64bit nested guestOSes.

I have to thank my friend and partner in crime Tuan Duong for showing me this trick awhile back. Tuan was performing some tests using both nested and physical ESXi 5 hosts and discovered this method after a bit of tinkering. At the time, I was not sure if others would find this useful and I did not document the process.

Disclaimer: As usual, this is not officially supported by VMware, use at your own risk. 

Here are the steps:

1. You must be running vSphere 5, create a nested ESXi 5 host using this article How to Enable Support for Nested 64bit & Hyper-V VMs in vSphere 5

2. Create an EVC enabled cluster or use an existing cluster with whatever baseline you would like and click on the "Current CPUID Details" in cluster settings.

3. Copy down the CPU mask flags for that particular EVC baseline, you will need this in the next step

4. Shutdown your nested ESXi 5 host and edit the VM's settings and under "Options" tab click on "CPUID Mask->Advanced". You will take the CPU mask from the above step and update the nested ESXi 5 VM to make it match

5. Go ahead and power on your nested ESXi 5 host and join it to the EVC enabled cluster you created earlier. You should not see any errors when connecting to the cluster and after that you can create a nested 64bit VM within that virtualized ESXi 5 host.

There you have it, running a nested ESXi 5 host and joined to an EVC enabled cluster! Isn't VMware technology awesome! 🙂