There are a ton of new features with the latest release of vSphere 5.1, but the one "unsupported" feature I always test first is "Nested Virtualization" (aka Nested ESXi) and with the latest release, it seems to have gotten even better. You will still need to have the same physical CPU prerequisites as you did in the past to run "Nested Virtualization" as well as nesting 64-bit VMs.

  • Intel VT-x or AMD-V is required for running "Nested Virtualization" which supports nested 32-bit VMs
  • Intel EPT or AMD RVI is required for running nested 64-bit VMs.

A quick way to verify whether your CPU truly supports both Intel-VT+EPT or AMD-V+RVI, you can paste the following into a browser:  

https://[your-esxi-host-ip-address]/mob/?moid=ha-host&doPath=capability

You will need to login with your root credentials and then look for the "nestedHVSupported" property and if it states false, it means you maybe able to install nested ESXi or other hypervisor, but you will not be able to run nested 64-bit VMs, only 32-bit VMs, assuming you have either Intel-VT or AMD-V support on your CPUs.

For more details, take a look at this article for troubleshooting: Having Difficulties Enabling Nested ESXi in vSphere 5.1?

Disclaimer: This is not officially supported by VMware, please use at your own risk.

There are some changes with Nested Virtualization in vSphere 5.1 also officially known as VHV (Virtual Hardware-Assisted Virtualization). If you are using vSphere 5.0 to run Nested ESXi or other nested Hypervisors, then please take a look at the instructions in this article. With vSphere 5.1, there have been a few minor changes to enable VHV.

  1. The new Virtual Hardware 9 compatibility will be required when creating your nested ESXi VM, Virtual Hardware 8 will not work if you are running ESXi 5.1 on your physical host. You will still need to enable promiscuous mode on the portgroup that will be used for your nested ESXi VM for network connectivity.
  2. vhv.allow = "true" is no longer valid for ESXi 5.1 to enable VHV. A new parameter has been introduced called vhv.enable = "true" that is now defined on a per VM basis to provide finer granularity of VHV support. This also allows for better portability between VMware's hosted products such as VMware Fusion and Workstation as they also support the vhv.enable parameter.
  3. You can now enable VHV on a per VM basis and using the new vSphere Web Client which basically adds the vhv.enable = "true" parameter to the VM's .VMX configuration file.

Note: You can run a nested ESXi 5.1 VM on top of a physical ESXi 5.0 host, just follow the instructions here.

Enabling VHV (Virtual Hardware-Assisted Virtualization)

Step 1 - Create a new Virtual Hardware 9 Virtual Machine using the new vSphere Web Client that's available with vCenter Server 5.1.

Step 2 - Select "Linux" as the guestOS Family and "Other Linux (64-bit)" as the guestOS Version.

Step 3 - During the customize hardware wizard, expand the "CPU" section and select "Hardware Virtualization" box to enable VHV.

Note: If this box is grayed out, it means that your physical CPU does not supported Intel VT-x + EPT or AMD-V + RVI which is required to run VHV OR that you are not using Virtual Hardware 9. If your CPU only supports Intel-VT or AMD-V, then you can still install nested ESXi, but you will only be able to run nested 32-bit VMs and not nested 64-bit VMs.

Step 4 - It is still recommended that you change the guestOS Version to VMware ESXi 5.x after you have created the VM shell, as there are some special settings that are applied automatically. Unfortunately with the new vSphere Web Client, you will not be able to modify the guestOS after creation, so you will need to use the C# Client OR manually go into the .VMX and update guestOS = "vmkernel5"

Now you are ready to install nested ESXi VMs as well as run nested 64-bit VMs within.

If you have followed my previous article about How to Enable Support for Nested 64bit & Hyper-V VMs in vSphere 5 you may recall a diagram about the levels of "Inception" that can be performed with nested ESXi. That is, the number of times you could nest ESXi and still have it be in a "functional" state. With vSphere 5.0, the limit that I was able to push was 2 levels of nested ESXi. With latest release of vSphere 5.1, I have been able to push that limit now to an extraordinary 3 levels of inception!

You might ask why would someone want to do this ... well I don't have a good answer other than ... because I can? ;) VHV is one of the coolest "unsupported" feature in my books and I'm glad it is working beyond what it was designed for.

For proper networking connectivity, also ensure that either your standard vSwitch or Distributed Virtual Switch has both promiscuous mode and forged transmit enabled either globally on the portgroup or distributed portgroup your nested ESXi hosts are connected to.

Nesting "Other" Hypervisors

For those of you who feel inclined to run other hypervisors such as Hyper-V, you can do so with latest release of ESXi 5.1. The process if very straight forward just like running nested ESXi host.

Step 1 - Create a Virtual Hardware 9 VM and select the appropriate guestOS. In this example, I selected Windows Server 2012 (64-bit) as the guestOS version.

Step 2 - Enable VHV under the CPU section if you wish to create and run nested 64-bit VMs under Hyper-V

Step 3 - You will need to add one additional .vmx parameter which tells the underlying guestOS (Hyper-V) that it is not running as a virtual guest which in fact it really is. The parameter is hypervisor.cpuid.v0 = FALSE

70 thoughts on “How to Enable Nested ESXi & Other Hypervisors in vSphere 5.1

  1. One thing to note, don’t do the vhv.enable on the physical host if you plan on running VSE on it. It will complain that it doesn’t support Virtual Hardware-Assisted Virtualization. Just enable it per VM.

    • It’s actually best to enable VHV on a per VM basis, that way you won’t hit this problem due to the virtual hardware version differences in the environment.

  2. Note: If this box is grayed out, it means that your physical CPU does not supported Intel VT-x or AMD-V which is required to run VHV OR that you are not using Virtual Hardware 9.

    I cannot understand above part.

    I have enabled Intel-VT in BIOS on PowerEdge 1950 server. I added vhv.enable = “TRUE” as well – yet, I cannot select “Hardware Virtualization” box to enable VHV when creating virtual machine using web client. Where am, I wrong. Appreciate your input.

  3. Hi Ram. Same situation here. We have two VT-supported processors, one 5130 and one E5335. The latter works, the first doesn’t. What processor do you have?

  4. Hey Ram,

    Same thing here Dell Precision T5400 with dual X5460, “hardware Virtualization” is grayed out, and when I start a VM I get a VM Message stating “Virtualized Intel VT-x/EPT is not supported on this platform”.

    Output of esxcfg-info shows supported and enabled

    ~ # esxcfg-info | grep “HV Support”
    |—-HV Support……………………………………..3
    |—-World Command Line……………………………grep HV Support

    Worked fine with 4.1 and 5 –

  5. Hi,

    Same thing here. Hp dl380g5 e5345 and dl380g6 l5520 both show

    ~ # esxcfg-info | grep “HV Support”
    |—-HV Support……………………………………..3
    |—-World Command Line……………………………grep HV Support

    But no luck getting it enabled.

  6. Same thing for me too!

    HP DL380 G5 with 1 of Dual-Core 5160 Intel CPU
    In Bios, “Intel Virtualisation” is enabled.
    But “Hardware Virtualisation” on vSphere Web Client 5.1 is greyed out for any newly created HW V.9 Virtual Machine.

  7. Having the same problem on Dell 1950 –
    In Bios, “Intel Virtualisation” is enabled.
    But “Hardware Virtualisation” on vSphere Web Client 5.1 is greyed out for any newly created HW V.9 Virtual Machine.
    Used to work ok on vSphere 5.0.

  8. Yes thank you
    You just need to create the VM in the webclient and then
    The option is not grayed out.
    If you create the vm in the viclient and then goto the webclient you can not select it.
    Still strange that there is a difference between the web and
    Windows client. But on the other hand probably going away from Ms windows dependency

    Regards Toine

  9. Are you sure about last entry, just tried creating VM via Web Client, still greyed out. on Dell 1950 Host.
    There some inconsistency here. Works for some and not others.

  10. I am also having same problem here. I was able to create virtual esxi5.1 – Created vm server 2012 through web client – yet Hardware Virtualization box is grayed out.

    I even checked intel site for xeon e5410 processor with regard to EPT – It is available but, I don’t see an option to enable it in BIOS.

    Even checked esxcfg-info command to check the status of HV Support
    The output is HV Support…3
    World Command Line Grep HV Support

    I even ran coreinfo -v to check the status of EPT. It shows a DASH which means the processor does not support EPT. If you have * then the processor support EPT.

    Coreinfo -v output

    Intel(R) Xeon(R) CPU E5410 @ 2.33GHz
    Intel64 Family 6 Model 23 Stepping 6, GenuineIntel
    HYPERVISOR * Hypervisor is present
    VMX – Supports Intel hardware-assisted virtualization
    EPT – Supports Intel extended page tables (SLAT)

    • Same issues here:

      Dell PowerEdge 2950III with dual Xeon E5430’s
      Dell Precision T7400 with dual Xeon E5420’s

      both have latest BIOS revisions, all virtualisation switches enabled in BIOS, and Intel ARK clearly states that these CPU’s support VT-X with EPT.

      Not a happy bunny at the moment :(

    • Ram/Seymour,

      Can you guys provide a bit more details on how you’re configuring your nested ESXi VMs? Which method are you using? Please provide the step by step that you’ve walked through

  11. Completed steps on physical pe 1950 esxi 5.1

    1. Enabled vhv on physical esxi 5.1
    2. Enabled promiscuous mode to accept on physical esxi 5.1
    3. Created Virtual machine using web client 5.1 – Guest OS – Other Linux x64.
    4. At Customize hardware – Expanded CPU to select Hardware Virtualization – Greyed out.

    What is going on, I can’t understand.

  12. Same problem here. No option for Hardware Virtulisation. I’m running an E3-1240 which should have the extensions required.

  13. nestedHVSupported boolean false

    Now, I understand why, I cannot run nested vm’s.

    By the way PowerEdge 1950 III is listed in HCL – VMware esxi 5.1

    • @Ram,

      It looks like the issue is that you don’t have EPT support. I know you pointed to ark.intel.com but I was told that even their site is not always 100% accurate and these actual bits are detected and that is why you see nestedHVSupported = false.

      What this means is you won’t be able to check the “Hardware Virtualization” box and you WILL be able to install nested ESXi or other hypervisors, just that you won’t be able to run 64-bit guestOSes, only 32-bit just like previous releases.

  14. @William – Thanks for all your effort.

    Yes coreinfo did point out that my pe1950 does not support EPT. Wrong info on intel site.

    My main purpose to run vesxi is for learning purpose only. I will play with my lab using 2 other physical pe 1950.

  15. @ William – I was able to install vESXi successfully. As pointed out b4 cannot run 64bit OS. That’s OK!

    By the way, I am getting this error on DP – “the most common cause for this error is that the time on the vdp appliance and your sso server are not in sync” Any thoughts……

  16. @Ram – I am curious on what your resolution was for the VDP “most common cause for this error is that the times on the VDP appliance…” error message?

  17. Failed to install nested Red Hat Enterprise Virtualization Hypervisor v6.3 on ESXi v5.1:
    I enabled promiscuous mode on the portgroup and used Web Client to create a new VM HW v9 as Linux/Other Linux (64-bit) and successfully enabled Hardware Virtualization.
    When I boot VM from ISO and select “Install or Upgrade” option – it just stays frozen there.
    Any thoughts?
    Thank you!

    • Found the workaround: hit to get further to boot: prompt, enter “install” and the installation will continue.
      Ended up with fully installed and configured RHEV Hypervisor and RHEV Manager VMs on ESXi 5.1, but inner-guest VMs (Win7 & WinXP) will not boot at all (staling with “Wait For Launch” message)…
      Any hint would be appreciated.
      prezha

    • Solved (by restart and some manual work).

      Bottom line: latest RHEV v6.3 *can* run virtualized on latest ESXi v5.1.

      Enjoy!

    • Hello all,

      I have to many problems put RHEV v6.3 run virtualized on latest ESXi v5.1.
      Can some one off you guys help me please?

      When I boot VM from ISO and select “Install or Upgrade” option (or any other option) it just stays frozen.

      @prezha can you tel me better wht you do on your workaround?

      tks in advance.

      Regards.

    • prezha was missing a VERY important word. You must press the “Escape” key at the install screen. That will present you with a “boot:” prompt at which point you must enter “install”. Wizard continues from then on as normal.

  18. How do I enable hardware assisted virtualization without the web client, I only have the c# client (free version)? I have a Xeon E3-1230 Sandy bridge CPU.

  19. Hopefully this helps someone at some point:

    Was able to turn on “Expose hardware assisted virtualisation”, but when I powered on a VM, vCenter reported that “The target host does not support the virtual machine’s current hardware requirements”

    This was odd, as my Intel x5650 is capable of EPT, and nestedHVSupported was listed as “true”…

    Turns out the EVC mode on my cluster was too low, hiding the functionality from the VMs. Increasing the EVC level solved the problem.

  20. hi, i have installed server 2k8 r2 Hyper v on ESXi 5.1, i am able to create vm in server manager but when i tried to power on it i am getting the following error message “the virtual machine could not be started because the hypervisor is not running”, checked the HyperV services are running. thanks in advance.

    Cheers
    Venkat Kurela

  21. Have you been able to create a filer at level 1 and present shared storage to subsequent levels? I cannot get this to work. I’d like to run iSCSI over vmxnet3 from level 2 to level 3 so that I don’t lose as much disk I/O, ideally.

  22. Hi William,

    Great article, as always. What base hardware specs would you suggest for a nested 5.1 host to be used for training purposes?

    Thanks!

    • As a note, when I created teaching pods for 5.0, this is what I used:
      CPU: 1 socket, 1 core
      RAM: 4GB
      Net Card: E1000
      SCSI: LSI Logic SAS
      Disk: 40GB Thin, Independent, Persistant

  23. Has this changed again? The advanced option setting “vhv.enable=’TRUE'” for the VM does not persist from the vSphere Client. However, after tweaking the option in the Web Client, the exposed configuration parameter was “featMask.vm.hv.capable=’Min:1′” and the nested feature works… (vSphere 5.1)

    • Looks to hide the “vhv.enable” from vSphere Client while adding and exposing the “featMask.vm.hv.capable” setting. Hmmm…

      (prior to power-on)
      # diff vm03\ \(ESXi\ 4.0\).vmx /tmp/vm03\ \(ESXi\ 4.0\).vmx
      — vm03 (ESXi 4.0).vmx
      +++ /tmp/vm03 (ESXi 4.0).vmx
      @@ -120,5 +120,4 @@
      config.readOnly = “FALSE”
      -softPowerOff = “FALSE”
      -featMask.vm.hv.capable = “Min:1″
      -vhv.enable = “TRUE”
      +softPowerOff = “TRUE”
      floppy0.present = “FALSE”

    • Appears that when adding the setting “vhv.enable” from the vSphere Client, it will not show-up in subsequent listings and will not be persistent.

      However, if you remove it from inventory, add it via shell and re-inventory it, the setting sticks (but is still not listed in advanced settings) – this is verifiable in the Web Client.

      Have you worked-out the process for migrating a running vESXi VM from a 5.0 to a 5.1 host?

  24. Hi William,

    In my nested ESXi, everything is OK. I have a problem after build 50 ESXi VMs. 50 ESXi VMs can ping outside and anywhere but can’t ping each other. I don’t know the reason why, I use E1000 & E1000E Nic for testing but the result is the same. Can you help me?

    Thanks.
    Randy

  25. Great informative post friend.
    I had trouble getting Nested HVH to work when nesting OracleVM 3.x cluster within ESXi 5.1 until I read this great post.

    Thanks

    –Tibbz

  26. William, thank you for sharing this information. I’m a newbie on VMware and recently started playing around with ESXi 5.5 on two self built, i7 980X 6-core & i7 950 4-core based servers. I need virtualization for my network labs (exam prep). On the fly I try to pick up some knowledge of Windows Server, Linux and VMware.

    Looking for a way to share my hardware with others I stumbled upon your article about nested 64 bit on vSphere v5. After following all the leads and reading some more documentation I was able to install ESXi 5.5 as guest on my ESXi host. But I got the “HARDWARE_VIRTUALIZATION WARNING: Hardware Virtualization is not a feature of the CPU …” message.

    When I looked further I ran into this post, for nested 64 bit on vSphere v5.1. The comments on this post gave me the last push. Here’s what I do:

    – I create my guest based on VMware Hardware Version 9 in accordance with your articles
    – I do some tweaking in ESXi shell
    – I finalize my configuration with vSphere Client 5.5 (that comes with ESXi 5.5); this INDEED removes vhv.enable=”TRUE” from my guest’s VMX file
    – when I’m done with configuring I add vhv.enable=”TRUE” through ESXi shell; as long as I refrain from further reconfiguration through vSphere Client it will stay there (I guess)
    – After bootup of my ESXi 5.5 guest, “https:///mob/?moid=ha-host&doPath=capability” shows me “nestedHVSupported boolean true” so I think I’m OK with this procedure

    But still: Would you know if there is a more elegant (permanent) solution to the disappearing vhv.enable=”TRUE” line with ESXi 5.5 and vSphere 5.5?

    Thanx once again,

    Steijn van Essen
    Network Professional
    The Netherlands

    From i8088 to i7-980X in 25 years and still waiting …

  27. Sorry, need to correct two omissions:
    – 2nd paragraph: … I was able to install ESXi 5.5 as guest on my ESXi 5.5 host …
    – 5th paragraph: … disappearing vhv.enable=”TRUE” line with ESXi 5.5 and vSphere Client 5.5?

    Final note: I run ESXi 5.5 host and vSphere Client 5.5 with an evaluation license (restricted in functionality, but unlimited in time)

    Steijn van Essen

    From i8088 to i7-980X in 25 years and still waiting …

Thanks for the comment!