xVC-vMotion

vMotion across different VDS version between onPrem and VMC

09/19/2018 by William Lam 13 Comments

For those of you who have attempted a vMotion (whether that is within a vCenter Server or between different vCenter Servers (including across SSO Domains), if the VM is running on a Distributed Virtual Switch (VDS) and the version of the VDS is not the same between the source and destination (VDS 6.5 to VDS 6.7), the operation will fail with the following error message (UI and API):

Currently connected network interface 'Network adapter 1' cannot use network 'DVPG-VM-Network (VDS-67)', because the destination distributed switch has a different version or vendor than the source distributed switch.

This behavior is no different on VMware Cloud on AWS (VMC) or at least, I thought it was, until I recently learned about a really neat feature that was introduced in the VMC 1.4p2 release, here is a snippet from the release notes:

Cross VDS version vMotion Compatibility
With this advanced configuration option enabled, bi-directional vMotion between on-premises and VMware Cloud on AWS can be achieved across different virtual distributed switch (VDS) versions (greater than or equal to version 6.0). This must be enabled on the on-premises vCenter.

It turns out there is actually a way to allow vMotions across different VDS versions, this is important for VMC because the software stack will always be using a newer version than what we ship to our onPrem customers. However, due to this limitation, we could not benefit from the latest VDS version but had to default it to VDS 6.0 to ensure that customers could migrate their workloads. The advanced setting mentioned in the release notes allows us to disable the strict compatibility check which is performed on the destination vCenter Server when a vMotion is initiated, this setting is now enabled by default on the VMC vCenter Server which is why you can perform migration across different VDS without having to do anything special on your onPrem vCenter Server.

UPDATE (01/02/21) - If you are running vSphere 7.x, an additional advanced setting must be configured called config.vmprov.enableHybridMode and set the value to true. For more details, you can refer to this VMware KB 79446. Thanks to reader Marc Alumbaugh for sharing this finding!

UPDATE (10/16/18) - With the release of vSphere 6.7 Update 1, customers can now also vMotion VMs from on-prem running on a VDS to VMC with NSX-T N-VDS.

Bulk VM Migration using new Cross vCenter vMotion Utility Fling

12/20/2017 by William Lam 53 Comments

Over the last few years, I have spoken to a number of customers who have greatly benefited from the ability to live migrate Virtual Machines across different vCenter Servers that are NOT part of the same vCenter Single Sign-On (SSO) Domain, which I had first shared back in 2015 here and here. This extended capability of the Cross vCenter vMotion feature enabled customers to solve new use cases that were challenging, especially for scenarios such as Datacenter migration, consolidation or even migrating existing workloads from their current environment into new SDDC deployments such as VMware Cloud Foundation (VCF) as an example.

Although customers could initiate Cross vCenter vMotions using the vSphere API which included PowerCLI (Move-VM cmdlet was enhanced in 6.5, more details here), the overall experience was still not as friendly. This was especially true for customers who may only have a small number of VMs to migrate and prefer a UI-based interface rather than an API/CLI only option. In addition, for large number of VM migrations, there was not an easy way to perform "batch" VM migrations that was easily consumable for folks who may not have a strong background in Automation or the vSphere APIs.

Today, I am pleased to share a new VMware Fling called the Cross vCenter Migration Utility that will help simplify the consumption of initiating VM migration(s) across different vCenter Servers, especially between dispart SSO Domains where a graphical interface was not available. This solution was developed out of our VMware Cloud Foundation (VCF) Engineering group which is part of the Integrated Systems Business Unit at VMware. I had spoken to a number of folks within the group about the extended Cross vCenter vMotion capability and I was super excited when I heard they were planning to release this tool as a Fling and make it available to all customers. I was fortunately to have been involved in the project alongside the Engineering lead Vishal Gupta and we are excited that we can finally talk about this project and see how customers will be using this new tool.

UPDATE (05/07/18) - The Fling has just been updated to 2.0 with the following new features:

Added support to select individual host as the placement target
Added support for migrating VMs with shared datastore
Added clone functionality in addition to relocate
Added resource summary details for placement targets
Added a prompt to verify site thumbprint during SSL verification
Added a link to refresh vm list in the inventory view
Updated REST APIs to add operation type parameter

Cross vCenter Migration Utility Fling

Cross vCenter vMotion Requirements: KB 2106952

Download Fling here

Features

Completely UI-driven workflow for VM migration
Provides REST API for managing migration operations
Works with vCenter not part of the same SSO domain
Supports both live/cold migration of VMs
Batch migration of multiple VMs in parallel
Flexible network mappings b/w source and destination sites

When to use Move-VM cmdlet vs xMove.ps1 script for performing Cross vCenter vMotions?

10/26/2017 by William Lam 10 Comments

Since publishing my Automating Cross vCenter vMotion between the same and different SSO Domain article back in early 2016, I have had a large number of customers reach out to me and share their success stories of allowing them to perform datacenter migrations to consolidating vCenter Servers all due to this awesome capability that was introduced in vSphere 6.0. In fact, many of the VM migration numbers were in the 4,000 to 8,000+ range which completely blew me away. It was great to hear from customers on how the xMoveVM.ps1 script had enabled them to do things that was simply not possible before, especially without impacting their workloads.

I still get pinged on a regular basis from customers about using my script and one thing that surprises many customers when I mention to them that this functionality has already been ported over to the native Move-VM cmdlet that was introduced with the PowerCLI 6.5 release. This had always been my original intention to provide an example using our vSphere API and enabling our customers in the short term and working with Alan Renouf and the PowerCLI team to get this folded back into the official PowerCLI cmdlets. This means, you no longer have to use my script for basic Cross vCenter vMotions whether that is between the same or different SSO Domain, which is quite nice as the number of user inputs is significantly reduced by using Move-VM cmdlet.

UPDATE (01/01/2018) - One additional option is the recently released Cross vCenter vMotion Utility Fling. For more details, please have a look at the blog post here.

Lets take a look at an example below where I have a VM called TestVM-1 which is residing in vcenter65-1 and I want to vMotion it to vcenter65-3:

With just 5 simple and easy to read lines of PowerCLI, you can perform this operation:

Uniquely identifying VMs in vSphere Part 3: Enhanced Linked Mode & Cross VC-vMotion

07/11/2017 by William Lam 6 Comments

Back in 2012, I had published two articles which provides details and guidance on how to uniquely identify a Virtual Machine for both a vSphere and/or vCloud Director environment. The primary use case for this information was for customers or partners who have developed their own provisioning solution which requires them to track their VM assets throughout their lifecycle, usually in some sort of configuration management database (CMDB).

Although these articles are almost 5 years old, the content is still very relevant today and I still continue to reference them both with customers, partners and even some of our internal R&D folks. Most recently, I had a question about whether the guidance in these article were still applicable or whether they would be impacted by some of the new VMware technologies and capabilities that had been introduced since writing those articles such as Enhanced Linked Mode (ELM) and Cross vCenter vMotion (xVC-vMotion).

Cross vCenter Server operations (clone / migrate) between versions of vSphere 6.x

02/27/2017 by William Lam 7 Comments

When cross vCenter Server operations such as clone and migrate was first introduced in vSphere 6.0, it required that both the source and destination vCenter Server (includes ESXi hosts) to be running the same vSphere version. With the release of vSphere 6.5, this base requirement still holds true (e.g. vSphere 6.5 for both source and destination), especially when performing these operations using the vSphere Web Client where mixed-vSphere versions is not supported outside of a rolling upgrade.

Having said that, it is possible and supported to clone or migrate a VM across different versions of vSphere 6.x, for example a vSphere 6.5 and a vSphere 6.0 Update 3 environment. This can be accomplished by performing a xVC-vMotion or xVC-Clone operation using the vSphere API. For the the xVC-vMotion use case, I have extensively written about it here and here and with PowerCLI 6.5r1, the Move-VM cmdlet has even been updated based on my feedback to support this capability natively. Furthermore, you can even perform these operations across completely different vCenter Single Sign-On Domains, which enables a new level of mobility for your VMs and access to resources of independently deployed vCenter Server instances.

UPDATE (11/01/17) - The following VMware KB 2106952 has just been updated to reflect what is officially supported in terms of Cross vCenter Operations ( Clone / Migrate) across different versions of vSphere. The matrix in the KB reflects what has been tested by Engineering and one thing you may notice is that Cross vCenter vMotion/Clone from vSphere 6.x to vSphere 6.5 is only supported when running at least vSphere 6.0 Update 3. After speaking with the PM, the reason for this change is that pre-vSphere 6.0 Update 3, there were no pre-checks in the code to prevent Cross vCenter Operations for un-supported target hosts such as ESXi 5.5, which could lead to poor user experience as well as undefined failure scenarios. In addition, vSphere 6.0 Update 3 also includes additional enhancements to properly clean up failed provisioning operations which will make Cross vCenter Operations much more robust. Due to these reasons, though it is possible to perform Cross vCenter vMotion from earlier versions, it will not be officially supported. I have also updated my summarized table below to reflect what is in the VMware KB, but please use the KB as your official source of truth for what VMware supports.

To help make sense of the different combinations of vMotions and cloning operations, below are a few tables to help outline what is possible and supported today.

vMotion

Source vCenter Server	Destination vCenter Server	Supported	UI or API
vSphere 6.0	vSphere 6.0	Yes	UI and API
vSphere 6.x (pre 6.0 Update 3)	vSphere 6.5	Possible but Not Supported	N/A
vSphere 6.0 Update 3	vSphere 6.5	Yes	API
vSphere 6.5	vSphere 6.5+	Yes	UI and API
vSphere 6.5	vSphere 6.x	No	No
vSphere 6.5+	VMware Cloud on AWS	Yes	UI and API
VMware Cloud on AWS	vSphere 6.5+	Yes	UI and API

Cold Migrate

Source vCenter Server	Destination vCenter Server	Supported	UI or API
vSphere 6.0	vSphere 6.0	Yes	UI and API
vSphere 6.x (pre 6.0 Update 3)	vSphere 6.5	Possible but Not Supported	API
vSphere 6.0 Update 3	vSphere 6.5	Yes	API
vSphere 6.5	vSphere 6.5	Yes	UI and API
vSphere 6.5	vSphere 6.x	No	No
vSphere 6.5+	VMware Cloud on AWS	Yes	UI and API
VMware Cloud on AWS	vSphere 6.5+	Yes	UI and API

Clone

Source vCenter Server	Destination vCenter Server	Supported	UI or API
vSphere 6.0	vSphere 6.0	Yes	UI and API
vSphere 6.x (pre 6.0 Update 3)	vSphere 6.5	No	N/A
vSphere 6.0 Update 3	vSphere 6.5	No	N/A
vSphere 6.5	vSphere 6.5+	Yes	UI and API
vSphere 6.5	vSphere 6.x	No	N/A
vSphere 6.5+	VMware Cloud on AWS	Yes	UI and API
VMware Cloud on AWS	vSphere 6.5+	Yes	UI and API

Virtual Networking Migration

Source Type	Destination Type	Supported
VDS	VDS	Yes
VDS	VSS	No
VSS	VSS	Yes
VSS	VDS	Yes

Note1: vMotioning and/or cloning of VMs which uses the new vSphere Encryption feature introduced in vSphere 6.5 is not supported.

Note2: "Compute" only xVC-vMotion insufficient space issue has now been resolved with vSphere 6.0 Update 3, see this post here for more details.

Note3: xVC-vMotion is not supported on 3rd party switches as we can not checkpoint the switching state.

Here are some additional xVC-vMotion and vMotion articles that may also useful to be aware of:

Quick Tip – vSphere 6.0 Update 3 resolves “Compute” only Cross-vCenter vMotion operation

02/27/2017 by William Lam Leave a Comment

Previously when a "Compute" only Cross-vCenter vMotion (xVC-vMotion) was initiated, which only migrates the VMs compute from one vCenter Server to another while maintaining its current storage configuration, an insufficient space error may be thrown under certain conditions. This behavior was due to the way vCenter Server used to calculate the available space on the destination vCenter Server.

Prior to vSphere 6.0 Update 3, vCenter Server used the Managed Object Reference (MoRef) ID of the vSphere Datastore determine whether the source and destination was the same. Even if you have the exact same vSphere Datastore mounted in both the source and destination vCenter Server, there was a good chance the MoRef ID will be different which then causes this calculation to occur. Now, the "insufficient space" error only occurs IF, the free space on the current vSphere Datastore is less than the size of the VM to be migrated which is why this behavior was only observed in some environments. Some customers workaround the problem by simply freeing up enough capacity which then allowed them to perform the operation.

The good news is this has now been resolved in the latest vSphere 6.0 Update 3 release which came out last Friday and has been outlined as one of the resolved issues in the release notes:

Attempts to perform an exclusive compute resource cross vCenter vMotion might fail.
When a VM is migrated using vMotion or cold migrate from a vCenter to another vCenter and space available on datastore is less than size of the Virtual Machine Disk (VMDK), an error similar to the following is displayed:
insufficient space

Rather than using the MoRef ID to determine if the vSphere Datastore is the same in both the source and destination vCenter Server, it is now using the datastore URL path. This means, if you want the correct behavior for a "Compute" only xVC-vMotion, you should ensure that the vSphere Datastore is mounted using the same name in both the source and destination vCenter Server.