vmfork-aka-instant-clone
The VMware Fling's team recently released an update to the existing PowerCLI Extensions which now exposes the new VMFork aka Instant Clone capability that was introduced in vSphere 6.0. The Fling contains a set of PowerCLI Extension Modules which in turn provides new PowerCLI cmdlets for accessing the Instant Clone feature. The idea behind the Fling is to help VMware understand how customers would like to consume the Instant Clone feature not only from a CLI point of view but also from an API and UI standpoint. Prior to this, Instant Clone was only available through the use of either Horizon View or the Big Data Extensions product. I think this is a great opportunity for customers and partners to help shape how Instant Clone should be consumed more generally.

One of the use cases I had in my mind when I had first heard about the Instant Clone feature was to be able to quickly instantiate new Nested ESXi VMs. When I got the opportunity to help test out early prototypes of the Instant Clone cmdlet to help provide feedback and usability improvements, I knew I had to give Nested ESXi a try!

Requirements:

High level process:

  1. A "preparation" script will be manually uploaded & executed within the Nested ESXi VM (Parent VM) to prep the system for Instant Cloning
  2. As the Parent VM is quiesce, both the pre/post customization script will be uploaded to the Parent VM automatically. The "pre-customization" is also then executed within the Parent VM which properly setups the library path to the VMware Tools binary (applicable to ESXi 6.0 only) and is then placed in a ready state for creating Instant Clones
  3. As new Instant Clone (Child VMs) are spun up, the "post-customization" script is automatically executed to add additional configurations and most importantly ensure newly created Instant Cloned Nested ESXi VMs have unique network identities

Note: For Instant Cloning regular OSes, only step 2 and 3 are really needed. Due to a known issue with VMware Tools for Nested ESXi, I have found that it is easier to prepare the Nested ESXi VM prior to quiescing and creating Instant Clones from the Parent VM.

Instructions:

Step 1 - Download and install both PowerCLI 6.0 Release 1 & Instant Clone PowerCLI Extensions Fling.

Step 2 - Perform a fresh Nested ESXi 6.0 installation in a VM, do not configure additional settings outside of enabling ESXi Shell and SSH.

Step 3 - Download the Nested ESXi 6.0 Instant Clone Scripts which contains the following four files:

  • prep-esxi60.sh - Prepares the Nested ESXi VM and ensures that new Child VMs will not retain the Parent VM's MAC Address which is baked in several places
  • pre-esxi60.sh - Pre-customization script which is used to properly setup the library paths to use the VMware Tools daemon to retrieve guest properties from PowerCLI script
  • post-esxi60.sh - Post-customization script which is used to apply networking configuration and hostnames for example
  • vmfork-esxi60.ps1 - An example PowerCLI script which issues the Instant Clone cmdlets

Note: For out of the box use, the only script that needs to be modified is the PowerCLI "vmfork-esxi60.ps1" script, the rest of the scripts should work or require very little to no modifications assuming you have followed the instruction thus far.

Step 4 - Upload the prep-esxi60.sh to Nested ESXi 6.0 VM (Parent VM) and then execute it using either the ESXi Shell over SSH or through a VMRC session. If you use SSH, you will notice that the script hangs, that is because the VMkernel interface is deleted as part of the script.

Step 5 - Next, we need to make a few edits to the vmfork-esxi60.ps1 script to update the name of your ESXi VM, along with its credentials and the full path to both the pre and post customization scripts. Below is an example of the variables that you will need to edit:

The section shown below will also need to be edited which contains the customization properties which are then passed down to the guestOS for configuration as part of the Instant Clone process.

Step 6 - Lastly, it is time to run the script by issuing the following command:

.\vmfork-esxi60.ps1

instant-clone-nested-esxi-0
If everything was successful, you should see a couple of new powered on Instant Cloned Nested ESXi VMs that have been fully customized and ready for use!

instant-clone-nested-esxi-1
Note: There have been a couple of times where newly Instant Clone VMs have not been properly customized and when looking in the Instant Clone logs under /var/tmp/quiesce.log you may find "Unable to fork" error message. I usually have to re-quiesce the Parent VM which I do so by reverting back to a snapshot that captures the state after Step 4. Once I re-run the PowerCLI script, I am able to successfully deploy N-Number of Instant Clone Nested ESXi VMs. For additional best practices and tips/tricks, be sure to check out this blog post here.

Big thanks to Jim Mattson for some of his earlier research and work on this topic which made implementing these scripts much easier.

13 thoughts on “How to VMFork aka Instant Clone Nested ESXi?

  1. Hi William,

    Thanks you for sharing this awesome news. I was wondering if this method can be applied to instantly clone a vCenter, NSX manager or vShield manager with several esxi hosts (essentially I want to be able to clone the whole setup to multiple dev and qa group so that they don’t have to spent time setup the environment).

    Example: on vlan 100 I have a vCenter, vShield manager register with vCenter, and several esxi hosts added to vCenter, and I want to clones all these vm to vlan 200, 300, etc. (in the child clone setup I don’t need to modify the IP address of any vm because all the vm will be on a separate vlan 200, 300, etc.).

    Thanh

    • This should work as long as they’re isolated on the network since you’ll have duplicate MAC Addresses, etc. In fact, the PowerCLI Development team is doing exactly this in their environment for QE testing and using the Instant Clone feature to quickly spin up test environments

      • Hi William,

        Do you know if there any limitation in what guest vm can be enable as instant clone vm, because I have no problem setup an instant clone of a vCenter server and virtual esxi host, but I am having trouble enable instant clone on the nsx manager guest vm.

        PowerCLI C:\> $parentVM_nsx = Get-VM lab-nsx7

        PowerCLI C:\> $parentVM_nsx

        Name PowerState Num CPUs MemoryGB
        —- ———- ——– ——–
        lab-nsx7 PoweredOn 4 12.000

        PowerCLI C:\> $parentForkVM_nsx = Enable-InstantCloneVM -VM $parentVM_nsx -guestUser “admin” -GuestPassword “Passw0rd123!” -Confirm:$false
        Enable-InstantCloneVM : 8/18/2015 1:43:12 AM Enable-InstantCloneVM Guest Type and Guest OS Family
        have no values for VM ‘lab-nsx7’
        At line:1 char:21
        + $parentForkVM_nsx = Enable-InstantCloneVM -VM $parentVM_nsx -guestUser “admin” – …
        + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        + CategoryInfo : InvalidResult: (:) [Enable-InstantCloneVM], VimException
        + FullyQualifiedErrorId : Fling_InstantClone_GuestOperationsHelper_GetGuestType_UnknwnGuest,VMware.VimAu
        tomation.Extensions.InstantClone.Commands.EnableInstantCloneVM

        • The issue that I suspect with the NSX Manager is that when you login, you’re not placed in a regular shell which is required as vmtoolsd is called to actually enable the “Quiesce” operation. As you probably know when you login to NSX Manager, you’re dropped into a limited shell which doesn’t have access and I suspect this is the issue.

    • I think that would be an awesome idea to hear from the PowerCLI team on how they leveraged the Instant Clone feature to help with development/testing

  2. Hi William,

    Just out of curiosity, how does any Horizon product leverage the instant clone feature?

    I didn’t see anything new that would indicate new instant cloning capabilities in Horizon suite of products.

  3. Always great information William!

    I ran into an issue while trying to go crazy with vmfork. I’m getting errors about maximum vcpu limits and not sure how to bypass.

    I’m running an Ubuntu Parent and vmfork children on top of nested ESXI 6.0 on top of ESXi 5.5 physical. When I was powering on VM’s I was trying for 20 children and got 18 to work. 19 wouldn’t power on and gives this message:

    Maximum virtual machines limit reached: 19 (32768 worlds).
    An error was received from the ESX host while powering on VM UbuntuClone190.
    Failed to start the virtual machine.
    Failed to power on VM.
    Could not power on virtual machine: Number of running VCPUs limit exceeded.

    So I did some more reading and found another blog that mentioned changing the maxvCPUsPerCore & maxvms values (https://deangrant.wordpress.com/2015/01/11/configuring-esxi-host-maximum-number-of-vcpus-on-a-single-core-and-virtual-machines-per-host/) After changing those and rebooting the nested ESXi 6.0 host it’s stuck on booting at the sfcbd-watchdog service.

    So I might have to rebuild my nested ESXi VM’s, but aside from that though. Do you have any idea how to get more VM’s per host and bypass any of the safeguards that may be limiting it?

Thanks for the comment!