I often hear from users that they would like to be able to just clone from an existing Nested ESXi VM that has already been configured and just create additional Nested ESXi VM instances from that. For me personally, I do not have a use case for this since I just deploy additional ESXi instances using an automated Kickstart deployment. However, I can see why this would be useful for anyone that does not have an automated deployment or just want to quickly deploy additional Nested ESXi instances by just cloning from an existing image and then manually change the networking configuration afterwards.
First off, cloning of a Nested ESXi VM is possible and you can already do this today. You will get a brand new Virtual Machine that will have a unique MoRef ID, InstanceUUID, BIOS UUID and MAC Addresses for each of the virtual network adapters which you can see an example of this from the screenshot below.
Everything from outside of the guest OS looks great as we would expect but there is actually two issues from within ESXi that you may not be aware of.
- The first issue is that you will get a duplicated MAC Address of the VMkernel interface(s) because the Nested ESXi configuration is exactly the same.
- The second issue is having a duplicated ESXi System UUID, also known as a VMkernel UUID which should normally be unique and can sometimes be used for tracking purposes. You can see this System UUID by running the following ESXCLI command: esxcli system uuid get or by looking in esx.conf configuration file.
To properly clone an existing Nested ESXi VM, you will need to perform the following two operations within the Nested ESXi VM prior to cloning.
First Configuration - There is an advanced ESXi setting called FollowHardwareMac that will automatically update the VMkernel's MAC Address whenever the Virtual Machine's virtual network adapter MAC Addresses changes. To do so, you will need to run the following ESXCLI command:
esxcli system settings advanced set -o /Net/FollowHardwareMac -i 1
Second Configuration - The other modification that is required is to delete the existing System UUID entry in /etc/vmware/esx.conf configuration file. This will ensure a new System UUID will automatically be generated when the system boots up. To do so, open esx.conf and delete the entire /system/uuid line entry as seen in the screenshot below. Here is a quick snippet you can run without needing to open up VI:
sed -i 's#/system/uuid.*##' /etc/vmware/esx.conf
To ensure the file is persisted, run /sbin/auto-backup.sh
Once both configurations have been performed you are now ready to start cloning additional Nested ESXi instances. You will still need to login to each Nested ESXi VM and manually change the IP Address and hostname which you of course can leverage the Guest Operations API if you have VMware Tools for Nested ESXi installed.
If you plan on joining your "cloned" Nested ESXi instances to a vCenter Server and the ESXi hosts contains a local datastore, you will not be able to add the hosts to the same Datacenter/Cluster. The reason for this is that the cloned ESXi hosts will have a duplicated VMFS UUID. To fix this, you just need to re-signature the VMFS volume by using the following ESXCLI command:
esxcli storage vmfs snapshot resignature -l [VMFS-VOLUME]
These tips can also apply to cloning boot LUNs/volumes in physical hosts that use SAN instead of local disks for boot.
Good stuff, playing around with this and dynamic disks to create a golden OVF for nested ESXi in my home lab. Just in time for Christmas!
Since it’s really tiny and can’t C&P from an image, here’s the command to check the FollowHardwareMac setting:
esxcli system settings advanced list -o /Net/FollowHardwareMac
Hello William,
great Post.
I tried the “esxcli storage vmfs snapshot resignature -l [VMFS-VOLUME]” command but do get the message “No unresolved VMFS snapshots with volume label ‘esxi2-local’ found”.
Did I miss something?
Thanks
Manfred
The part of resignaturing the vmfs volume is apparently incorrect since the esxi does not see the vmfs as copied or snapshoted thus resignaturing cannot be applied. The vmfs is indeed snapshoted at vmware workstation level instead of the esxi level.
The only way I know that is working is as below, this method requires recreating the local vmfs datastore, hence all data on it WILL LOST.
1. # vmkfstools –queryfs -h /vmfs/volumes/
take note of the “Partitions spanned”, it’s kind like mpx.vmhbal:C0:T0:L0:3 (refered to as hereafter).
2. # esxcli storage filesystem unmount -l
make sure the local datastore is unmounted.
3. # vmkfstools –createfs vmfs5 –blocksize 1m –setfsname /dev/disks/
this will recreate the vmfs on the partition and give it a new UUID.
is the local datastore label you want to have the UUID changed, mine was “datastore1”. is the new datastore label, you can specify the same as .
Did you try these commands you’re referring to? They don’t seem so right…
This should work:
# enter maintenanceMode mode
esxcli system maintenanceMode set –enable true –timeout=1
# determine partition device (look under “Partitions spanned (on “lvm”)” for mpx.)
vmkfstools -P /vmfs/volumes/
# dismount datastore by label
esxcli storage filesystem unmount -l
# recreate FS
/usr/sbin/vmkfstools -C vmfs5 -b 1m -S /vmfs/devices/disks/mpx.
# reboot system
esxcli system shutdown reboot –reason=”Updated System UUID”
# exit maintenance mode
esxcli system maintenanceMode set –enable false –timeout=1
Now I see the problem.. this brain-damaged commenting system is purging everything in chevrons.. let’s try one last time with parenthesis..
# enter maintenanceMode mode
esxcli system maintenanceMode set –enable true –timeout=1
# determine partition device (look under “Partitions spanned (on “lvm”)” for mpx.(something))
vmkfstools -P /vmfs/volumes/(datastore-label)
# dismount datastore by label
esxcli storage filesystem unmount -l (datastore-label)
# recreate FS
/usr/sbin/vmkfstools -C vmfs5 -b 1m -S (newlabel) /vmfs/devices/disks/mpx.(something)
# reboot system
esxcli system shutdown reboot –reason=”Updated System UUID”
# exit maintenance mode
esxcli system maintenanceMode set –enable false –timeout=1
Above steps worked. Thanks for the solution
Dos this method erase data from datastore1. I cloned ESXi server2 from ESXi server1 by taking one boot disk out from server1 and then inserted on server2 and booted and works fine. But when I added ESX server1 and server2 in vcenter it shows as datastore1(22) mounted for both server1 & server2. I do not want to loose the data. Does this method erase data if I follow this procedure on ESXi server2?
I created a virtual ESXi template using the listed FollowHardwareMac setting and deleted the UUID as described. This works great for vmk0. It seems to regenerate properly every time I deploy a new copy of the template. My template also has several other vmk interfaces as well. What I am finding though is that other any other vmks (except for vmk0) that are part of the cloned VM still retain the same MAC as the original VM after being cloned from the template. Do you have any ideas on this?
Worked for me even after cloning. Thanks a ton !!
I was able to successfully clone nested ESX by doing a standard clone then performing a factory reset inside ESXi.
Hi,
Your article is very useful to me and solved 80% of my problem, But I am not able to add clone vESXi to same VC.
I ran your command “esxcli storage vmfs snapshot resignature -l [VMFS-VOLUME]”. I am getting error. Can you please help me to fix this issue. How to change UUID of datastore for base image. So that Cloned vESXi will not get same UUID for datastore
1. Entered follow hardware command via SSH to ESXi nested host.
2. Used vi to remove UUID.
3. Shutdown host.
4. Using VC – get VMDK error.
I am either missing a step or the solution does not work for 5.5. Can someone help?
This works on both 5.x & 6.x. What exact error message are you seeing, as VMDK error is not specific enough
Hello William. Thank you for responding. I’ve blanked the config but it appeared to be an issue related to copying the VMDK. What I tried was manually copying the files to another folder, booted the ESXi 5.5 nested. ticked the option I copied it when it showed up. From the other posts under your main post, the it seems the local datastore (1) is linked via UUID or something. I tried doing the resig but SSH CLI was telling me nothing needed to be resig’ed. It would be really nice if you literally show us mere mortals a spoon-fed step by step of everything you did. Being able to clone an ESXi nested image would make my life much easier with VC.
I’m not sure why you’re manually copying the VMDK? If you have vCenter Server, you just need to run through the preparation as shown in the blog post and then right click and select “Clone”
Hi William,
Can you help in in the below situation please.
When i follow the above steps am getting the below error while trying to generate new SSID.
I am running this command: /usr/sbin/vmkfstools -C vmfs5 -b 1m -S name1 /vmfs/devices/disks/mpx.vmhba1:
C0:T0:L0
ends up saying
Checking if remote hosts are using this device as a valid file system. This may take a few seconds…
Creating vmfs5 file system on “mpx.vmhba1:C0:T0:L0” with blockSize 1048576 and volume label “name1”.
/vmfs/devices/disks/mpx.vmhba1:C0:T0:L0: Permission denied. (Have you set the partition type to 0xfb?)
Error: Permission denied
I have logged in as root.. but it syas permission denied.
Regards,
Madhu
We are cloning nested ESXi in vCloud Director. What doesn’t seem to change on each linked clone is the bios.uuid which is causing is issues. We’ve done all the above and while system uuid is changed, the bios.uuid is not changing. Any suggestions? thanks!
Niamh,
Have a look at https://kb.vmware.com/kb/2002506 for your answer
thanks William, yep working on it now !!
Hi will, Is it possible to strip the VMFS UUID before cloning?
Very Informative article. Thanks alot. It worked well. Other than following command (entering mantenance mode in CLI dint worked…) But that i easily managed by connecting directly and doing in GUI.
# enter maintenanceMode mode
esxcli system maintenanceMode set –enable true –timeout=1
I’m following your excellant article, and cloning an ESX 6.5 host.
I’ve run “esxcli system settings advanced set -o /Net/FollowHardwareMac -i 1” in the template, then cloned with customization wizard. The Management vmk0 kernel got a new Mac and assigned IP, however the vMotion vmk1, and NFS vmk2 did not get a new Mac, or the new assigned IP from the Wizard so I set them manually.
I’ve confirmed in the vCenter dvs Port listing that my three cloned hosts have different Mgmt Mac’s but duplicate Mac’s on the vMotion and NFS vmkernel ports. This explains why I’m having so many intermittent network issues.
Is the Customization wizard and FollowHardwareMac known to work for non Mgmt kernel ports on ESX 6.5?
Lee
Hi,
I have used clone method when I initially built ESXi 5.1. Then I was able to upgrade it to 5.5 with no issue. Now that I am upgrading to ESXi 6.5, its failing adn I am getting a purple screen and I see following error when I look at the debug output. “The System has found a problem on your machine and cannot continue. System with same UUID have been detected. Make sure you do not have two ESXi installation”. Also I see message “cpu35:65895)FSS: 5817: Detected Duplicate image on device ‘naa.600605b00aa3a4901f08161c4c258e19:5”. I have two data stores. One for OS and one for APPL. this 19:5 is for APPL DS but I could not find such device 19:5 on my server.
Any idea? I came across many blogs but so far none has helped including the one from vmware to identify why its finding this as duplicate image.