instant clone

Leveraging Instant Clone in vSphere 6.7 for extremely fast Nested ESXi provisioning

05/17/2018 by William Lam 4 Comments

The idea of "Instant Cloning" a Nested ESXi VM (running ESXi in a VM) is not a new concept. In fact, I had shared a solution back in 2015 using the private VMFork APIs. However, what has changed is the ease of consumption, primarily due to the re-architecture of Instant Clone in vSphere 6.7 (more details here and here) which resulted in a public and simplified API. Some of you might ask, why not simply clone a Nested ESXi VM or create a Link Clone? What benefit would I get by using Instant Clone?

The answer is not only speed, but the fact that the instantiated VM is fully operational and ready to start executing where as a traditional full clone or linked clone requires a full OS boot up that can take up to several minutes to deploy and configure. This may not sound like much for a small number of Nested ESXi VMs, but as you increase the number of instances, Instant Clone really shines while still maintaining speed and the instant availability of the VM. As you can imagine, this definitely opens up for some interesting use cases whether it be for personal home lab or educational purposes like VMware HOL. In addition, we also have customers who deploy Nested ESXi not only at high scale but also with a high churn rate for development purposes, think CI/CD type of a workload who can also benefit from Instant Clone.

So how fast are we talking about? Lets say you wanted to test out the latest version of VSAN in vSphere 6.7, you would normally deploy 3 Nested ESXi VMs, power them up and wait for them to be ready on the network. With Instant Clone, you can deploy three fully functional Nested ESXi VMs in just 30seconds! As the VMs are instantly available for consumption, you can start the VSAN enablement workflow immediately and even parts of that can be baked into the Instant Clone workflow. With the ease of provisioning Nested ESXi VMs, you can simply maintain a catalog of ESXi templates which are in "frozen" states and then leverage Instant Clone to deploy just-in-time Nested ESXi environments and discard them once you are done. Pretty slick if you ask me! and something I plan on using going forward.

Disclaimer: Nested ESXi is still not officially supported by VMware. Please use at your own risk.

New Instant Clone Architecture in vSphere 6.7 – Part 2

04/30/2018 by William Lam 3 Comments

In the previous article, I provided an overview of the new "Parentless" Instant Clone feature which was introduced in vSphere 6.7 and some of the architectural differences between prior versions of Instant Clone. In this post, I will show you how to use the new Instant Clone feature, which is currently only available with the vSphere API.

There are two important parts when using Instant Clone:

A user defined script which runs within the GuestOS and is responsible for customizing the network identity of the Instant Clone. This script is not just limited to network configurations but can also be used to customize other OS and/or application settings.
The deployment script which runs outside of the GuestOS and instantiates new Instant Clones using the vSphere API. This script is responsible for passing in data (network configuration, OS and/or application settings) to the GuestOS which can then be accessed directly by the user defined script running within each Instant Clone for actual customization.

To demonstrate the use of the Instant Clone API, I will be using PowerCLI and have also created an Instant Clone PowerCLI module called InstantClone.psm1. However, the API is not limited to just PowerCLI and can be consumed using any of the new vSphere 6.7 SDKs. For our example, we will be exercising the creating Instant Clones from a "Frozen" Source VM workflow since that is the most efficient method, especially as you scale to larger number of Instant Clones. For those that wish to experiment with the other workflow of creating Instant Clones from a "Running" Source VM, feel free to take a look at the previous post for the required steps.

For my setup, I will be using an Ubuntu Server 16.04 system as my Source VM and I will create 30 Instant Clones from this system. Each Instant Clone will immediately be available for use and will be fully customized and reachable from a network perspective (e.g. unique IP and MAC Address). Here is a screenshot of one of my deployments to give you an idea of how blazing fast Instant Clone is 🙂

Here is a short video that I just recorded that demos the Instant Clone APIs using PowerCLI (this demo is using an Ubuntu VM configured with 2GB memory where as the original screenshot was 1GB)

Demo of Instant Clone in vSphere 6.7 from lamw on Vimeo. [Read more...] about New Instant Clone Architecture in vSphere 6.7 – Part 2

New Instant Clone Architecture in vSphere 6.7 – Part 1

04/24/2018 by William Lam 6 Comments

Instant Clone or VMFork (as it is referred internally) has been around for a number of years now. It was initially available as part of vSphere 6.0 with the primary consumer being Horizon View and their just-in-time desktop solution. Although Instant Clone was part of the core vSphere platform, public APIs were not available for external consumption. Many customers were interested in the technology to enable other non-VDI use cases such as Dev/Test, Continuous Integration/Continuous Development (CI/CD) and even Container workloads. Part of the reason for not exposing the API was partially due to the original Instant Clone architecture which has certain limitations and constraints.

In addition, VMware was also interested in getting feedback from customers on how they would like to consume Instant Clone from an Automation standpoint, this was important because the current workflows were also some what complex. This started out with the release of a PowerCLI Instant Clone Extension Fling that provided an abstraction on top of the private APIs. Based on that and other feedback, VMware followed that up by releasing Instant Clone for pyvmomi (vSphere SDK for Python) Fling which gave customers more programmatic access to the private APIs. Both Flings were a huge success and we even had customers using the pyvmomi Instant Clone modules in Production to deploy several hundred Instant Clone VMs per day for their CI/CD workloads.

Taking the learnings from both Horizon View and the feedback from customers using the Flings, the Instant Clone Product/Engineering team has been hard at work behind the scenes on simplifying the Instant Clone architecture and removing limitations and constraints that had existed in earlier versions. As you can imagine, this was a non-trivial amount of work that would need to be released in phases, especially as VM lifecycle management touches almost every part of the vSphere stack. The team really focused on ease of consumption, especially from an Automation standpoint which is how most customers prefer to consume Instant Clone.

How Wrecking Crew Inc. leveraged vSphere’s Instant Clone to instantly provision hundreds of VMs

09/18/2015 by William Lam 9 Comments

Okay, Wrecking Crew Inc. is not a real company but a fictitious one that was used during Alan Renouf and Li Zheng's VMworld session, #INF5803 - Deploy Hundreds of VMs Instantly via Forking (aka vSphere Instant Clone). During the session, Wrecking Crew Inc. was described as a modern online gaming company who was about to launch a new product. They found out that their game was going to be more popular than they had originally anticipated and needed to be able to quickly spin up a large number of instances of their game to handle the load.

Though this was a fictional company to help set the stage for the demo, the underlying use case is an actual challenge for many of our customers. One of the examples that Alan used was the traditional Black Friday sale which occurs on the Friday after Thanksgiving in US. As you can imagine, being able to quickly scale up your online application based on customer demand can really give retailers an real edge over their competition. In todays world, waiting a few additional seconds for a site to load can mean the difference between a sale and customer going else where for their business. Online retail is just one of the many industries and verticals that can easily benefit from the Instant Clone capability.

To help further demonstrate the use case, a recorded demo was shown utilizing the new PowerCLI Extensions which provides access to the Instant Clone feature from an Automation standpoint. I actually built the demo for Alan and Li's session and I thought I would share some more details along with the video in case you were not able to attend the session in person, which I was not able to do.

Below are the set of technologies that were used in the demo:

I wanted to also give a quick shoutout to my buddy Rawlinson Rivera for his help with Intel and getting us access to their 64 Node NVMe VSAN Cluster. Our friends over at Intel were kind enough to allow us to quickly record a demo before they had to pack and ship the hardware to San Francisco for VMworld.

To provide some additional context on what you will see in the video, a diagram is provided below. The environment initially consists of two Photon VMs which runs the Consul Cluster which provides service discovery and configuration capabilities and the Nginx Load Balancer which is highlighted in the purple and blue box. Each of those VMs run their respective application which runs inside of a Docker Container and is highlighted in the smaller boxes. On each ESXi host, we have Photon VM which contains our Node.js application that we will be Instant Cloning from. This Photon VM is also referred to as the Parent VM and where all the new forked children VMs will be spawn off of.

This Photon VM also runs an instance of Consul, Registrator and the Node.js application inside of a Docker Container. As new instances are brought online which is highlighted in the black box, they will automatically register themselves with the Consul Cluster. As the Docker Containers are started up, Registrator will be notified to automatically add the new instances to the Nginx Load Balancer, making our application servers available for use immediately. When our application server powers down, Registrator will automatically detect that the Docker Container for our Node.js application is no longer running and automatically unregister it from the Nginx Load Balancer. To simply scale up or down the application, it is simply Instant Cloning our Photon (Parent) VM and powering it on or off, there are no additional steps required.

Without further ado, here is the video of the demo. If you want additional commentary and you attended VMworld US but could not make it to Alan and Li's session, you can now watch it online here. This session will also be repeated in VMworld EMEA for those attending in a couple of weeks. I would highly recommend you check out the session as there is a lot of awesomeness in the session along with technical deep dive of the Instant Clone technology.

VMworld 2015 Demo of how Wrecking Crew Inc. leverages vSphere's Instant Clone feature from lamw on Vimeo.

For more details about Instant Clone, be sure to check out these resources below which includes a Instant Clone script repository for several GuestOSes, including Photon which was used in the demo:

Instant Clone PowerCLI cmdlets Best Practices & Troubleshooting

08/06/2015 by William Lam Leave a Comment

I was fortunate to have been given early access to the VMFork (Instant Clone) PowerCLI cmdlets to help provide early feedback and usability improvements before it was released to customers. Having spent some time with the Fling, I have learned a thing or two about how Instant Cloning works and some of the caveats or gotchas while creating the customization scripts that are used as part of the Instant Clone workflows. I wanted to put together a quick reference on some of my findings as well as well as other recommendations from Engineering who have worked closely with the Instant Clone feature.

The idea is to have this as a living document which I will update as new tips and tricks are identified.

Best Practices

Ensure VMware Tools is installed inside the guestOS and also good time to ensure you are running the latest
Both Pre/Post Customization scripts are uploaded to /var/tmp by the Enable-InstantCloneVM cmdlet
Do not delete Child VMs directly on ESXi, manage it through vCenter Server. There is currently a known issue in which deleting Child VMs will also delete the Parent VM's disk
Additional custom variables can be passed to the post-customization script by adding to the -ConfigParams array of variables.
- An example could be passing in two custom properties called "foo" and "bar" which would look like:
@{foo = "val1";bar ="val2"}
- To retrieve the variable "foo" and "bar" from within the post-customization script, you would issue the following commands:
vmtoolsd --cmd "info-get guestinfo.fork.foo"
vmtoolsd --cmd "info-get guestinfo.fork.bar"
A Forked Child VM will also have a duplicate MAC Address which needs to be updated as it is not automatically picked up.
modprobe -r vmxnet3;modprobe vmxnet3

A Forked Child VM may also have identical kernel entropy pools which means semi-predictable RNG, possibly including TCP sequence numbers (Submited by George Hicken)
A Forked Child VM's system clock may also be out of date (until you call hwclock --hctosys or similar) which can cause problems with ordering of file timestamps (Submited by George Hicken)
Shared host keys if you are using a PKI system or identical asset identifiers in the case of Windows and any sort of AD infrastructure would also need to be either removed prior or updated after a Child VM is created (Submited by George Hicken)
Instant Cloning Nested ESXi has been a bit tricky due to a known issue with the VMware Tools for Nested ESXi. I have found that manually preparing the guest prior to Instant Cloning has yield better results. For more information on how to Instant Clone Nested ESXi, check out the blog post here
Powering off the Parent VM means that the VM is no longer quiesced and this also means that new Child VMs can not be instantiated until all existing Forked Child VMs have been powered off and the Parent VM has been re-quiesced
If you plan on downloading or installing additional software packages on the Parent VM, it is recommended that you perform that operation directly in the VM and not within the pre-customization script. I have noticed that if pre-customization takes too long, the quiesce operation eventually fails even though the operations within the pre-customization script executed successfully.
To ensure Forked Child VMs do not contain duplicate disk ID's from Parent VM such as setting up a VSAN environment using Instant Clone Nested ESXi, add the disks after Forked Child VMs have been created.
For additional OS Customization Scripts, be sure to check out the Instant Clone community customization script repository and consider contributing back scripts that you have developed.
When you hard reset or power off on a child VM it will respawn from the parent, soft resets will not respawn (Submitted by Alan Renouf)

Troubleshooting

Instant Clone guestOS logs are stored in /var/tmp/quiesce.log
Consider enabling tracing within your customization scripts. An example of this for a shell script is using
set -x

Add additional echo or print statements like Start/Stop of certain sections like Pre/Post which can aide in reviewing the Instant Clone logs as seen in the screenshot above
For Instant Cloning Nested ESXi guestOSes, I recommend taking a snapshot after you have prepared the guest and removed any system specific information. This allows you to quickly revert back to a known state for ease of debugging. I found this to be very useful to be able to start back a known clean state while developing the customization scripts for Instant Cloning Nested ESXi
A known issue that is mentioned in the documentation of the Instant Clone cmdlets is after enabling a ParentVM for Instant Cloning, is that it is no longer available for migration to another ESXi host. The reason for this is that after powering off the VM, the "parentEnabled" boolean flag is still set to "true" which prevents the migration. Currently, there is not a work around but hopefully this will be resolved in a future update of the cmdlets. You can see this by running the following PowerCLI snippet:
(Get-VM "MyParentVM").ExtensionData.Config.ForkConfigInfo

Instant Clone community customization script repository

08/04/2015 by William Lam Leave a Comment

I am sure many of you are probably anxious to get your hands on the new VMFork aka Instant Clone PowerCLI Extensions Fling that was recently released! While using the Instant Clone cmdlets to help provide feedback and improvements, I have found that I spent the majority of my time on developing and fine-tuning the pre and post-customization scripts. Instead of having others hit similar issues that I ran into, I wanted to see how I could easily share some of the leanings but also incorporate collaboration?

After thinking about this for a bit, I realized this was a great opportunity to create a community Github repository of Instant Clone customization scripts that anyone can either use and/or contribute back to. I have already added a few OS customization scripts to the repo to start with like ESXi 6.0, Ubuntu 14.x and VMware Photon. To access the repository, simply visit https://github.com/lamw/vmfork-community-customization-scripts

The Instant Clone community customization script repository is broken up by series of OS directories which contain the relevent set of pre/post-customization scripts for that OS and any additional scripts that might be required. It may also contain further instructions on how to use the script as well as an example "driver" script which calls into the Instant Clone cmdlets demonstrating how to use the scripts.

Here is an example for ESXi 6.0 OS:

esxi60
├── post-esxi60.sh
├── pre-esxi60.sh
├── prep-esxi60.sh
└── vmfork-esxi60.ps1

For those that wish to contribute back, just fork the repository and send me a pull request. I am really looking forward to seeing what the community comes up with!