Cross vCenter Server operations (clone / migrate) between versions of vSphere 6.x

When cross vCenter Server operations such as clone and migrate was first introduced in vSphere 6.0, it required that both the source and destination vCenter Server (includes ESXi hosts) to be running the same vSphere version. With the release of vSphere 6.5, this base requirement still holds true (e.g. vSphere 6.5 for both source and destination), especially when performing these operations using the vSphere Web Client where mixed-vSphere versions is not supported outside of a rolling upgrade.

Having said that, it is possible and supported to clone or migrate a VM across different versions of vSphere 6.x, for example a vSphere 6.5 and a vSphere 6.0 environment. This can be accomplished by performing a xVC-vMotion or xVC-Clone operation using the vSphere API. For the the xVC-vMotion use case, I have extensively written about it here and here and with PowerCLI 6.5r1, the Move-VM cmdlet has even been updated based on my feedback to support this capability natively. Furthermore, you can even perform these operations across completely different vCenter Single Sign-On Domains, which enables a new level of mobility for your VMs and access to resources of independently deployed vCenter Server instances.

To help make sense of the different combinations of vMotions and cloning operations, below are a few tables to help outline what is possible and supported today.

vMotion (Hot / Cold)

Source vCenter Server Destination vCenter Server Supported UI or API
vSphere 6.0 vSphere 6.0 Yes UI and API
vSphere 6.0 vSphere 6.5 Yes API
vSphere 6.5 vSphere 6.5 Yes UI and API
vSphere 6.5 vSphere 6.0 No N/A

Clone (Hot / Cold)

Source vCenter Server Destination vCenter Server Supported  UI or API
vSphere 6.0 vSphere 6.0 Yes UI and  API
vSphere 6.0 vSphere 6.5 No N/A
vSphere 6.5 vSphere 6.5 Yes UI and API
vSphere 6.5 vSphere 6.0 No N/A

Virtual Networking Migration

Source Type Destination Type Supported
VDS VDS Yes
VDS VSS No
VSS VSS Yes
VSS VDS Yes

Note1: vMotioning and/or cloning of VMs which uses the new vSphere Encryption feature introduced in vSphere 6.5 is not supported.

Note2: "Compute" only xVC-vMotion insufficient space issue has now been resolved with vSphere 6.0 Update 3, see this post here for more details.

Here are some additional xVC-vMotion and vMotion articles that may also useful to be aware of:

Quick Tip - vSphere 6.0 Update 3 resolves "Compute" only Cross-vCenter vMotion operation

Previously when a "Compute" only Cross-vCenter vMotion (xVC-vMotion) was initiated, which only migrates the VMs compute from one vCenter Server to another while maintaining its current storage configuration, an insufficient space error may be thrown under certain conditions. This behavior was due to the way vCenter Server used to calculate the available space on the destination vCenter Server.

Prior to vSphere 6.0 Update 3, vCenter Server used the Managed Object Reference (MoRef) ID of the vSphere Datastore determine whether the source and destination was the same. Even if you have the exact same vSphere Datastore mounted in both the source and destination vCenter Server, there was a good chance the MoRef ID will be different which then causes this calculation to occur. Now, the "insufficient space" error only occurs IF, the free space on the current vSphere Datastore is less than the size of the VM to be migrated which is why this behavior was only observed in some environments. Some customers workaround the problem by simply freeing up enough capacity which then allowed them to perform the operation.

The good news is this has now been resolved in the latest vSphere 6.0 Update 3 release which came out last Friday and has been outlined as one of the resolved issues in the release notes:

  • Attempts to perform an exclusive compute resource cross vCenter vMotion might fail.
    When a VM is migrated using vMotion or cold migrate from a vCenter to another vCenter and space available on datastore is less than size of the Virtual Machine Disk (VMDK), an error similar to the following is displayed:
    insufficient space

Rather than using the MoRef ID to determine if the vSphere Datastore is the same in both the source and destination vCenter Server, it is now using the datastore URL path. This means, if you want the correct behavior for a "Compute" only xVC-vMotion, you should ensure that the vSphere Datastore is mounted using the same name in both the source and destination vCenter Server.

Quick Tip - Connect-OMServer throws The request was aborted: Could not create SSL/TLS secure channel.

While doing some work with PowerCLI and vRealize Operations Manager (vROps), I ran into the following error message when trying to connect to my vROps instance using PowerCLI:

Connect-OMServer : 2/17/2017 5:27:50 AM Connect-OMServer The request was aborted: Could not create SSL/TLS secure channel.
At line:1 char:1
+ Connect-OMServer -Server vrops.primp-industries.com -User admin -Password VMware ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (VMware.VimAutom...tionServiceImpl:OMConnectionServiceImpl) [Connect-OMServer], OMException
+ FullyQualifiedErrorId : OM_ConnectivityServiceImpl_ConnectOMServer_ByUserNameAndPassword_ConnectError,VMware.VimAutomation.vROps.Commands.Cmdlets.ConnectOMServer

Although there were some hits on Google, none of the suggestions has worked. I had also found that this issue was only happening in one of my lab environments which was running Windows 2008 R2, for my other system which had Windows 8.1, the issue was not observed.

I had reached out to the PowerCLI Engineering team and it looks like the issue is due to a change in the hashing algorithm (SHA512) that vROps uses for its SSL Certificates. When using TLS 1.2, SHA512 is not supported by default. The fix is to simply install the following patch here which will resolve the problem.

Extracting VIN (vSphere Infrastructure Navigator) information using PowerCLI & vROps REST API

A request that I continue to receive from customers on a fairly regular basis is a way to extract the virtual machine application services and dependencies that is provided by vSphere Infrastructure Navigator (VIN) solution. Below is an example of what a VIN discovery might look like and in this case, it is actually mapping out the application and dependencies of itself.


Today, there is not a public API for VIN and although I have published several methods here, here and here on how to extract the information from VIN, the experience is still not very user friendly or easy to do.

Last week, while talking to a fellow colleague who works in our VMware Validated Design team, I found out that VIN actually has a vRealize Operations Manager (vROps) Management Pack and could potentially be useful in helping us retrieve the information generated by VIN.


Not having spent much time with vROps Management Packs, I understood at a high level they provided custom dashboards for vROps, but I was not sure if the data provided by the management packs could also be retrieved programmatically? It has also been some time since I have looked at the vROps REST API and specifically the "public" REST API which allows customers to retrieve the metrics collected from within vROps.

Continue reading

Potential ESXi Host Preparation issues with NSX 6.3

While working on updating my vGhetto Automated vSphere Lab Deployment script to add support for NSX 6.3 with vSphere 6.5, I ran into an issue with the Host Preparation step. Although the resolution turned out to be quite simple, it was very difficult to diagnose the problem. I suspect this scenario could easily be encountered by others, so I wanted to make folks aware of what I ran into. There is also another potential gotcha for host preparation that I did not encounter myself, but it was brought to my attention that I thought was also worth sharing as well.

Scenario 1 - Attempted Host Preparation and all "Install agent" tasks fails with "Cannot complete the operation. See the event log for details" and below is a screenshot of the error. There was nothing useful when looking at the event logs for either NSX or ESXi using the vSphere Web Client.


There was also nothing useful in the ESXi log /var/log/esxupdate.log that gave insights to why the NSX VIBs failed to install:

2017-02-16T12:38:53Z esxupdate: 73899: Transaction: DEBUG: Populating VIB list from all VIBs in metadata https://vcenter65-1.primp-industries.com:443/eam/vib?id=d4917629-51d1-4da9-82d6-8da54815447d; depots:
2017-02-16T12:38:54Z esxupdate: 73899: downloader: DEBUG: Downloading https://vcenter65-1.primp-industries.com:443/eam/vib?id=d4917629-51d1-4da9-82d6-8da54815447d to /tmp/tmpdfcbr23q...
2017-02-16T12:38:54Z esxupdate: 73899: Metadata.pyc: INFO: Unrecognized file vendor-index.xml in Metadata file
2017-02-16T12:38:54Z esxupdate: 73899: imageprofile: INFO: Adding VIB VMware_locker_tools-light_6.5.0-0.0.4564106 to ImageProfile (Updated) ESXi-6.5.0-4564106-standard
2017-02-16T12:38:54Z esxupdate: 73899: imageprofile: INFO: Adding VIB VMware_bootbank_esx-vsip_6.5.0-0.0.4987428 to ImageProfile (Updated) ESXi-6.5.0-4564106-standard
2017-02-16T12:38:54Z esxupdate: 73899: imageprofile: INFO: Adding VIB VMware_bootbank_esx-vxlan_6.5.0-0.0.4987428 to ImageProfile (Updated) ESXi-6.5.0-4564106-standard
2017-02-16T12:38:54Z esxupdate: 73899: vmware.runcommand: INFO: runcommand called with: args = '['/bin/localcli', 'system', 'maintenanceMode', 'get']', outfile = 'None', returnoutput = 'True', timeout = '0.0'.
2017-02-16T12:38:54Z esxupdate: 73899: HostInfo: INFO: localcli system returned status (0) Output: Disabled Error:

Continue reading