• Skip to primary navigation
  • Skip to main content
  • Skip to primary sidebar

virtuallyGhetto

  • About
  • Privacy
  • VMware Cloud
  • Home Lab
  • Nested Virtualization
  • Automation
    • VMware Kickstart
    • VMware API/SDK/CLI
    • VMware vMA/VIMA
    • VMware OVF / OVFTOOL
  • Apple Mac
  • VCSA
  • VSAN
You are here: Home / ESXi / How to properly clone a Nested ESXi VM?

How to properly clone a Nested ESXi VM?

12/06/2013 by William Lam 46 Comments

I often hear from users that they would like to be able to just clone from an existing Nested ESXi VM that has already been configured and just create additional Nested ESXi VM instances from that. For me personally, I do not have a use case for this since I just deploy additional ESXi instances using an automated Kickstart deployment. However, I can see why this would be useful for anyone that does not have an automated deployment or just want to quickly deploy additional Nested ESXi instances by just cloning from an existing image and then manually change the networking configuration afterwards.

First off, cloning of a Nested ESXi VM is possible and you can already do this today. You will get a brand new Virtual Machine that will have a unique MoRef ID, InstanceUUID, BIOS UUID and MAC Addresses for each of the virtual network adapters which you can see an example of this from the screenshot below.

Everything from outside of the guest OS looks great as we would expect but there is actually two issues from within ESXi that you may not be aware of.

  • The first issue is that you will get a duplicated MAC Address of the VMkernel interface(s) because the Nested ESXi configuration is exactly the same.
  • The second issue is having a duplicated ESXi System UUID, also known as a VMkernel UUID which should normally be unique and can sometimes be used for tracking purposes. You can see this System UUID by running the following ESXCLI command: esxcli system uuid get or by looking in esx.conf configuration file.

To properly clone an existing Nested ESXi VM, you will need to perform the following two operations within the Nested ESXi VM prior to cloning.

First Configuration - There is an advanced ESXi setting called FollowHardwareMac that will automatically update the VMkernel's MAC Address whenever the Virtual Machine's virtual network adapter MAC Addresses changes. To do so, you will need to run the following ESXCLI command:

esxcli system settings advanced set -o /Net/FollowHardwareMac -i 1

Second Configuration - The other modification that is required is to delete the existing System UUID entry in /etc/vmware/esx.conf configuration file. This will ensure a new System UUID will automatically be generated when the system boots up. To do so, open esx.conf and delete the entire /system/uuid line entry as seen in the screenshot below. Here is a quick snippet you can run without needing to open up VI:

sed -i 's#/system/uuid.*##' /etc/vmware/esx.conf

To ensure the file is persisted, run /sbin/auto-backup.sh

Once both configurations have been performed you are now ready to start cloning additional Nested ESXi instances. You will still need to login to each Nested ESXi VM and manually change the IP Address and hostname which you of course can leverage the Guest Operations API if you have VMware Tools for Nested ESXi installed.

If you plan on joining your "cloned" Nested ESXi instances to a vCenter Server and the ESXi hosts contains a local datastore, you will not be able to add the hosts to the same Datacenter/Cluster. The reason for this is that the cloned ESXi hosts will have a duplicated VMFS UUID. To fix this, you just need to re-signature the VMFS volume by using the following ESXCLI command:

esxcli storage vmfs snapshot resignature -l [VMFS-VOLUME]

More from my site

  • VM serial logging to the rescue for capturing Nested ESXi PSOD
  • vSphere 6.0 Update 2 hints at Nested ESXi support for Paravirtual SCSI (PVSCSI) in the future
  • Deploying Nested ESXi is even easier now with the ESXi Virtual Appliance
  • VMware Tools for Nested ESXi updated to v1.2
  • How to run Qemu & KVM on ESXi?
Share this...
  • Twitter
  • Facebook
  • Linkedin
  • Reddit
  • Pinterest

Filed Under: ESXi, Nested Virtualization Tagged With: clone, esxi, nested, nested virtualization, uuid

Reader Interactions

Comments

  1. Jim Millard says

    12/07/2013 at 4:45 pm

    These tips can also apply to cloning boot LUNs/volumes in physical hosts that use SAN instead of local disks for boot.

    Reply
  2. rnelson0wordpress1 says

    12/11/2013 at 4:12 am

    Good stuff, playing around with this and dynamic disks to create a golden OVF for nested ESXi in my home lab. Just in time for Christmas!

    Since it’s really tiny and can’t C&P from an image, here’s the command to check the FollowHardwareMac setting:

    esxcli system settings advanced list -o /Net/FollowHardwareMac

    Reply
  3. Manfred says

    05/06/2014 at 12:33 am

    Hello William,
    great Post.

    I tried the “esxcli storage vmfs snapshot resignature -l [VMFS-VOLUME]” command but do get the message “No unresolved VMFS snapshots with volume label ‘esxi2-local’ found”.

    Did I miss something?

    Thanks
    Manfred

    Reply
    • Akira Z says

      07/16/2014 at 11:14 am

      The part of resignaturing the vmfs volume is apparently incorrect since the esxi does not see the vmfs as copied or snapshoted thus resignaturing cannot be applied. The vmfs is indeed snapshoted at vmware workstation level instead of the esxi level.

      The only way I know that is working is as below, this method requires recreating the local vmfs datastore, hence all data on it WILL LOST.

      1. # vmkfstools –queryfs -h /vmfs/volumes/
      take note of the “Partitions spanned”, it’s kind like mpx.vmhbal:C0:T0:L0:3 (refered to as hereafter).

      2. # esxcli storage filesystem unmount -l
      make sure the local datastore is unmounted.

      3. # vmkfstools –createfs vmfs5 –blocksize 1m –setfsname /dev/disks/
      this will recreate the vmfs on the partition and give it a new UUID.

      is the local datastore label you want to have the UUID changed, mine was “datastore1”. is the new datastore label, you can specify the same as .

      Reply
      • Giuliano says

        11/29/2014 at 10:28 pm

        Did you try these commands you’re referring to? They don’t seem so right…

        Reply
        • Jack says

          02/22/2015 at 11:13 am

          This should work:

          # enter maintenanceMode mode
          esxcli system maintenanceMode set –enable true –timeout=1

          # determine partition device (look under “Partitions spanned (on “lvm”)” for mpx.)
          vmkfstools -P /vmfs/volumes/

          # dismount datastore by label
          esxcli storage filesystem unmount -l

          # recreate FS
          /usr/sbin/vmkfstools -C vmfs5 -b 1m -S /vmfs/devices/disks/mpx.

          # reboot system
          esxcli system shutdown reboot –reason=”Updated System UUID”

          # exit maintenance mode
          esxcli system maintenanceMode set –enable false –timeout=1

          Reply
        • Jack says

          02/22/2015 at 11:17 am

          Now I see the problem.. this brain-damaged commenting system is purging everything in chevrons.. let’s try one last time with parenthesis..

          # enter maintenanceMode mode
          esxcli system maintenanceMode set –enable true –timeout=1

          # determine partition device (look under “Partitions spanned (on “lvm”)” for mpx.(something))
          vmkfstools -P /vmfs/volumes/(datastore-label)

          # dismount datastore by label
          esxcli storage filesystem unmount -l (datastore-label)

          # recreate FS
          /usr/sbin/vmkfstools -C vmfs5 -b 1m -S (newlabel) /vmfs/devices/disks/mpx.(something)

          # reboot system
          esxcli system shutdown reboot –reason=”Updated System UUID”

          # exit maintenance mode
          esxcli system maintenanceMode set –enable false –timeout=1

          Reply
          • Arun says

            04/11/2015 at 6:59 am

            Above steps worked. Thanks for the solution

          • Mack says

            02/05/2016 at 7:37 am

            Dos this method erase data from datastore1. I cloned ESXi server2 from ESXi server1 by taking one boot disk out from server1 and then inserted on server2 and booted and works fine. But when I added ESX server1 and server2 in vcenter it shows as datastore1(22) mounted for both server1 & server2. I do not want to loose the data. Does this method erase data if I follow this procedure on ESXi server2?

  4. Mike P says

    08/15/2014 at 2:25 pm

    I created a virtual ESXi template using the listed FollowHardwareMac setting and deleted the UUID as described. This works great for vmk0. It seems to regenerate properly every time I deploy a new copy of the template. My template also has several other vmk interfaces as well. What I am finding though is that other any other vmks (except for vmk0) that are part of the cloned VM still retain the same MAC as the original VM after being cloned from the template. Do you have any ideas on this?

    Reply
  5. Gaurav Pruthi says

    09/04/2014 at 8:58 am

    Worked for me even after cloning. Thanks a ton !!

    Reply
  6. Joshua O'Brien says

    10/29/2014 at 4:36 pm

    I was able to successfully clone nested ESX by doing a standard clone then performing a factory reset inside ESXi.

    Reply
    • Ben Spencer says

      05/23/2019 at 5:43 pm

      The factory reset worked for me too, thanks!

      I created a cluster with VCSA 6.0 and ESXi 6.0, and could not get vSAN to run. After the factory reset, it worked.

      Reply
  7. deva says

    10/13/2015 at 10:37 am

    Hi,

    Your article is very useful to me and solved 80% of my problem, But I am not able to add clone vESXi to same VC.

    I ran your command “esxcli storage vmfs snapshot resignature -l [VMFS-VOLUME]”. I am getting error. Can you please help me to fix this issue. How to change UUID of datastore for base image. So that Cloned vESXi will not get same UUID for datastore

    Reply
  8. JenBell says

    07/16/2016 at 9:54 am

    1. Entered follow hardware command via SSH to ESXi nested host.
    2. Used vi to remove UUID.
    3. Shutdown host.
    4. Using VC – get VMDK error.
    I am either missing a step or the solution does not work for 5.5. Can someone help?

    Reply
    • William Lam says

      07/16/2016 at 2:52 pm

      This works on both 5.x & 6.x. What exact error message are you seeing, as VMDK error is not specific enough

      Reply
      • JenBell says

        07/17/2016 at 7:31 am

        Hello William. Thank you for responding. I’ve blanked the config but it appeared to be an issue related to copying the VMDK. What I tried was manually copying the files to another folder, booted the ESXi 5.5 nested. ticked the option I copied it when it showed up. From the other posts under your main post, the it seems the local datastore (1) is linked via UUID or something. I tried doing the resig but SSH CLI was telling me nothing needed to be resig’ed. It would be really nice if you literally show us mere mortals a spoon-fed step by step of everything you did. Being able to clone an ESXi nested image would make my life much easier with VC.

        Reply
        • William Lam says

          07/17/2016 at 7:34 am

          I’m not sure why you’re manually copying the VMDK? If you have vCenter Server, you just need to run through the preparation as shown in the blog post and then right click and select “Clone”

          Reply
          • Madhu says

            06/05/2017 at 7:31 am

            Hi William,

            Can you help in in the below situation please.

            When i follow the above steps am getting the below error while trying to generate new SSID.

            I am running this command: /usr/sbin/vmkfstools -C vmfs5 -b 1m -S name1 /vmfs/devices/disks/mpx.vmhba1:
            C0:T0:L0

            ends up saying
            Checking if remote hosts are using this device as a valid file system. This may take a few seconds…
            Creating vmfs5 file system on “mpx.vmhba1:C0:T0:L0” with blockSize 1048576 and volume label “name1”.
            /vmfs/devices/disks/mpx.vmhba1:C0:T0:L0: Permission denied. (Have you set the partition type to 0xfb?)
            Error: Permission denied

            I have logged in as root.. but it syas permission denied.

            Regards,
            Madhu

  9. Niamh says

    01/18/2017 at 4:11 am

    We are cloning nested ESXi in vCloud Director. What doesn’t seem to change on each linked clone is the bios.uuid which is causing is issues. We’ve done all the above and while system uuid is changed, the bios.uuid is not changing. Any suggestions? thanks!

    Reply
    • William Lam says

      01/18/2017 at 6:01 am

      Niamh,

      Have a look at https://kb.vmware.com/kb/2002506 for your answer

      Reply
  10. Niamh says

    01/30/2017 at 3:24 am

    thanks William, yep working on it now !!

    Reply
  11. Ajay Nadakuditi says

    02/08/2017 at 1:11 pm

    Hi will, Is it possible to strip the VMFS UUID before cloning?

    Reply
  12. Imran Khan says

    11/14/2017 at 11:24 pm

    Very Informative article. Thanks alot. It worked well. Other than following command (entering mantenance mode in CLI dint worked…) But that i easily managed by connecting directly and doing in GUI.

    # enter maintenanceMode mode
    esxcli system maintenanceMode set –enable true –timeout=1

    Reply
    • leemarzke says

      08/26/2018 at 4:18 pm

      I’m following your excellant article, and cloning an ESX 6.5 host.

      I’ve run “esxcli system settings advanced set -o /Net/FollowHardwareMac -i 1” in the template, then cloned with customization wizard. The Management vmk0 kernel got a new Mac and assigned IP, however the vMotion vmk1, and NFS vmk2 did not get a new Mac, or the new assigned IP from the Wizard so I set them manually.

      I’ve confirmed in the vCenter dvs Port listing that my three cloned hosts have different Mgmt Mac’s but duplicate Mac’s on the vMotion and NFS vmkernel ports. This explains why I’m having so many intermittent network issues.

      Is the Customization wizard and FollowHardwareMac known to work for non Mgmt kernel ports on ESX 6.5?

      Lee

      Reply
  13. Mack says

    09/05/2018 at 6:48 am

    Hi,

    I have used clone method when I initially built ESXi 5.1. Then I was able to upgrade it to 5.5 with no issue. Now that I am upgrading to ESXi 6.5, its failing adn I am getting a purple screen and I see following error when I look at the debug output. “The System has found a problem on your machine and cannot continue. System with same UUID have been detected. Make sure you do not have two ESXi installation”. Also I see message “cpu35:65895)FSS: 5817: Detected Duplicate image on device ‘naa.600605b00aa3a4901f08161c4c258e19:5”. I have two data stores. One for OS and one for APPL. this 19:5 is for APPL DS but I could not find such device 19:5 on my server.
    Any idea? I came across many blogs but so far none has helped including the one from vmware to identify why its finding this as duplicate image.

    Reply
  14. Ranjith says

    02/26/2020 at 3:02 am

    Thanks for details. I have created a nested esxi for the DHCP enabled VLAN,but for the new cloned nexted esxi I am not getting proper IP and getting some private IP. any idea?

    Reply
  15. AS says

    10/29/2020 at 10:10 am

    Hello there!
    Great text!
    I wonder, is it necessary to install vmware tools to nested ESXi 7.0 and later?
    I ask because you mention it at the end of the artile and also because there is a light verson of tools already installed from scratch (tools-light – 11.1.1.16303738-16850804).
    Thanks for the article and reply!

    Cheers!
    AS

    Reply

Trackbacks

  1. VMware VSAN APIs | virtuallyGhetto says:
    03/03/2014 at 2:53 pm

    […] ESXi host provides a vsanSystem manager at configManager->vsanSystem which provides the following methods for disk […]

    Reply
  2. Provisioning Nested vSphere Hosts with PowerCLI | Sean Crookston says:
    04/07/2014 at 2:47 pm

    […] you have a nested vSphere template available which has been prepared to be used for cloning. See here for further details on doing […]

    Reply
  3. duplicated datastore UUIDs when cloning nested esxi on the same local datastore | log here before I forget says:
    07/03/2014 at 3:47 am

    […] those points in wlam’s blog, to add those cloned vESXi into the same vCenter, you will likely get errors complaining the […]

    Reply
  4. HowTo: Install Nested ESXi 5.5 | Cloud Solutions Architect says:
    07/26/2014 at 10:01 am

    […] http://www.virtuallyghetto.com/2013/12/how-to-properly-clone-nested-esxi-vm.html […]

    Reply
  5. Cloning Nested ESXi in Workstation - theITHollow says:
    10/06/2014 at 2:41 pm

    […] NOTE:  Steps one and two are completely ripped off from William’s post.  If you want more information please see it directly here:  http://www.virtuallyghetto.com/2013/12/how-to-properly-clone-nested-esxi-vm.html […]

    Reply
  6. VMware Workstaion Home Lab Setup Part 5 – ESXi Template - Virtxpert says:
    11/06/2014 at 1:19 pm

    […] following steps are courtesy of William Lam and virutallyghetto.com, check out his site and the blog post for full details on resetting the virtual machine.  Since we do not have any VMFS datastores, there are only two steps we need to do before cloning, […]

    Reply
  7. Building a Nested ESXi Home Lab | SIPALOOZA says:
    01/21/2015 at 4:35 pm

    […] properly prepare the nested host before you clone it! It’s William Lam again who wrote an excellent article about how to do this.  Here is a short […]

    Reply
  8. vShiza says:
    02/01/2015 at 11:35 pm

    […] properly prepare the nested host before you clone it! It’s William Lam again who wrote an excellent article about how to do this.  Here is a short […]

    Reply
  9. Running Nested ESXi / VSAN Home Lab on Ravello | virtuallyGhetto says:
    03/06/2016 at 8:50 am

    […] not want to do. The steps outlined here will be following the process which I have documented in my How to properly clone a Nested ESXi VM? […]

    Reply
  10. How To Install And Clone Nested ESXi VM? | Settlersoman - A Settler In The SDDC World. says:
    03/13/2016 at 4:59 pm

    […] Before cloning our ESXi VM, we need to do two steps: […]

    Reply
  11. Creación de una imagen de VMware ESXi para laboratorio - VMware Blog says:
    06/20/2016 at 4:33 pm

    […] Antes de poder clonar la máquina tenemos que realizar otras configuraciones para siguiendo los pasos, una vez más, de William Lam en virtuallyGhetto […]

    Reply
  12. Nested ESXi Enhancements in vSphere 6.5 | virtuallyGhetto says:
    10/26/2016 at 5:46 am

    […] Perform two configuration changes within the Nested ESXi VM which will prepare them for cloning. You can find the configuration changes described in my blog post here […]

    Reply
  13. How to create a Nested VMware vSAN 6.6 environment. - Provirtualzone - Virtual Infraestructures says:
    06/05/2017 at 3:27 am

    […] Thanks to William Lam for is help and article HERE regarding cloning Nested […]

    Reply
  14. How to set up a nested vSphere 6 environment - Part 2 says:
    07/24/2017 at 4:32 am

    […] this part as it is beyond the scope of this post but if you wish to learn more, here’s a good article on the […]

    Reply
  15. How to set up a nested vSphere 6 environment - Part 3 says:
    07/24/2017 at 5:51 am

    […] part as it is beyond the scope of this post but if you wish to learn more, here’s a good article on the […]

    Reply
  16. Create an ESXi 6.7 VM Template | iThinkVirtual™ says:
    01/21/2019 at 1:21 pm

    […] How to properly clone nested ESXi VM […]

    Reply
  17. Create a VMware vSAN 6.7 U1 Nested ESXi Lab - Virtualization Howto says:
    04/27/2019 at 7:24 pm

    […] Lam has a great write up here on how to properly clone an ESXi host for nested lab purposes. Quoting from William’s blog […]

    Reply
  18. Create a nested ESXi template in vSphere – thecjtlab says:
    11/17/2020 at 5:22 am

    […] How to properly clone a Nested ESXi VM? Enable SSH […]

    Reply

Thanks for the comment! Cancel reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Primary Sidebar

Author

William Lam is a Senior Staff Solution Architect working in the VMware Cloud team within the Cloud Services Business Unit (CSBU) at VMware. He focuses on Automation, Integration and Operation for the VMware Cloud Software Defined Datacenters (SDDC)

  • Email
  • GitHub
  • LinkedIn
  • RSS
  • Twitter
  • Vimeo

Sponsors

Privacy & Cookies: This site uses cookies. By continuing to use this website, you agree to their use.
To find out more, including how to control cookies, see here: Cookie Policy