Something that I really enjoy when I get a chance to, is to speak with our field folks and learn a bit more about our customer environments and some of the challenges they are facing. Last week I had quick call with one of our TAMs (Technical Account Managers) regarding the topic of Virtual Machine migration between vCenter Servers. The process of migrating Virtual Machines between two vCenter Servers is not particularly difficult, you simply disconnect the ESXi hosts from one vCenter Server and re-connect to the new vCenter Server. This is something I have performed on several occasions when I was a customer and with some planning it works effortlessly.

However, there are certain scenarios and configurations when migrating VMs between vCenter Servers that could potentially cause Virtual Machine MAC Address collisions. Before we jump into these scenarios, here is some background. By default, a Virtual Machine MAC Address is automatically generated by vCenter Server and the algorithm is based on vCenter Server's unique ID (0-63) among few other parameters which is documented here. If you have more than one vCenter Server, a best practice is to ensure that these VC IDs are different, especially if they are in the same broadcast domain.

As you can imagine, if you have two vCenter Servers that are configured with the same VC ID, there is a possibility that a duplicate MAC Address could be generated. You might think this is probably a rare event given the 65,000 possible MAC Address combinations. However, it actually happens more frequently than you think, especially in very large scale environments and/or Dev/Test for continuous build/integration environments which I have worked in the past and I have personally seen these issues before.

Going back to our vCenter Server migration topic, there are currently two main scenarios that I see occurring in customr environments and we can explore each in a bit more detail and their implications:

  • Migrate ALL Virtual Machines from old vCenter Server to new vCenter Server
  • Migrate portion of Virtual Machines from old vCenter Server to new vCenter Server

Migrate all Virtual Machines:

vm-migration-between-vcenter-0
In the diagram above, we have vCenter Server 1 and vCenter Server 2 providing a before/after view. To make things easy, lets say they have VC ID 1 and 2. If we migrate ALL Virtual Machines across, we can see their original MAC Addresses will be preserved as we expect. For any new Virtual Machines being created, the 4th octet of the MAC Address will differ as expected and the vCenter Server will guarantee it is unique. If you want to ensure that new Virtual Machines keep a similar algorithm, you could change the vCenter Server ID to 1. No issues here and the migration is very straight forward.

Migrate A portion of Virtual Machines:

vm-migration-between-vcenter-1
In the second diagram, we still have vCenter Server 1 and vCenter Server with unique VC IDs. However, in this scenario we are only migrating a portion of the Virtual Machines from vCenter Server 1 to vCenter Server 2. By migrating VM2 off of vCenter Server 1, the MAC Address of VM2 is no longer registered with that vCenter Server. What this means is that vCenter Server 1 can potentially re-use that MAC Address when it generates a new request. As you can see from the above diagram, this is a concern because VM2 is still using that MAC Address in vCenter Server 2, but vCenter Server 1 is no longer aware of its existence.

The scenario above is what the TAM was seeing at his customer's site and after understanding the challenge, there are a couple of potential solutions:

  1. Range-Based MAC Address allocation - Allows you to specify a range of MAC Addresses to allocate from which may or may not helpful if the migrated MAC Addresses are truly random
  2. Prefix-Based MAC Address allocation -  Allows you to modify the default VMware OUI (00:50:56) which would then ensure no conflicts would be created with previously assigned MAC Addresses. Though this could solve the problem, you potentially could run into collisions with other OUI's within your environment
  3. Leave VMs in a disconnected state - This was actually a solution provided by another TAM on an internal thread which ended up working for his customer. The idea was that instead of disconnecting and removing the ESXi host when migrating a set of Virtual Machines, you would just leave it disconnected in vCenter Server 1. You would still be able to connect the ESXi host and Virtual Machines to vCenter Server 2 but from vCenter Server 1 point of view, the MAC Addresses for those Virtual Machines are seen as in use and it would not be reallocated.

I thought option #3 was a pretty interesting and out of the box solution that the customer came up with. The use case that caused them to see this problem in the first place is due to the way they provision remote environments. The customer has a centralized build environment in which everything is built and then shipped off to the remote sites which is a fairly common practice. Since the centralized vCenter Server is not moving, you can see how previously used MAC Addresses could be re-allocated.

Although option #3 would be the easiest to implement, I am not a fan of seeing so many disconnected systems from an operational perspective as it is hard to tell if there is an issue with the ESXi host and Virtual Machines or because it has been migrated. I guess one way to help with that is to create a Folder called "Migrated" and move all disconnected ESXi hosts into that folder which would help mask that away and disable any alarms for those hosts.

Some additional per-requisite checks that you can perform prior to the partial Virtual Machine migration:

Ensure that the destination vCenter Server is not configured with the same VC ID else you can potentially run into duplicate MAC Address conflicts. You can do this either manually through the vSphere Web/C# Client or leveraging our CLI/API to do so.

Here is an example using PowerCLI to retrieve the vCenter Server ID:

Get-AdvancedSetting -Entity $server -Name instance.id

Ensure no duplicate MAC Addresses exists by comparing the MAC Addresses of the Virtual Machines to be migrated to the Virtual Machines in the new environment. Again, you can either do this by hand (which I would not recommend) or leveraging our CLI/API to extract this information.

Here is an example using PowerCLI to retrieve the MAC Addresses for a Virtual Machine:

Get-VM |  Select-Object -Property Name, PowerState, @{"Name"="MAC";"Expression"={($_ | Get-NetworkAdapter).MacAddress}}

If there are other scenarios or solutions that you have seen with Virtual Machine migrations between vCenter Serves, feel free to leave a comment. I am sure others can benefit from past experiences or any other lesson learns.

18 thoughts on “Considerations when migrating VMs between vCenter Servers

  1. Another consideration revolves around the use of virtual distributed switches (vDS’s). If the VMs in the source vCenter utilize vDS’s, then you need to make sure the same vDS exists in the destination vCenter server. Otherwise, you will import the ESXi hosts and not be able to move your VMs around at all due to identifier mis-matches.

    With vSphere 5.1+ you can utilize PowerCLI or the Web Client to export / import your vDS configuration (making sure check the “Preserve original distributed switch and port group identifiers” option).

      • Hi William,

        Let’s say I have ESXi1 (running 5.0) using vDS’s and its being managed by vCenter1 (running 5.0) with 50 active VMs running. Is it possible to disconnect ESXi1 from vCenter1 and add it to a vCenter2 (running 5.5) without having issues with the running VMs? The vDS configuration has been exported out of vCenter1 and imported onto vCenter2. Could I just simply migrate the active VMs off ESXi1 onto ESXi2 (running 5.5)?

        Basically, what’s the best way to migrate a lot VMs off the 5.0 environment onto the new 5.5 environment with virtual distributed switches? One way is to power off a VM, remove it from inventory in 5.0 and register it in 5.5. But I have hundreds of VMs. What other options do I have?

        Your help is much appreciated and thanks!

  2. Another item to add to this is EVC mode. Be sure to decide on a common EVC mode for your environments so hosts can join clusters and migrate guests between clusters without restarts.

  3. Hello William, do you know, how to migrate VMs between 2 vcenters using vDS?

    I already see some steps on other blogs, and they are using VDS to VSS and then back VDS migration. It represents a lot of work and I hope to accomplish this other easier way. I didn’t see any workflow in vCO or cmdlet in powershell.

    I will explain my situation: due to several upgrades, my vCenter db performs bad, so I decided to start almost from scratch.
    I have deployed a new vCenter, clone VI structures (vm folders, roles, etc..) and now I would like to migrate VMs with minimal downtime (they are using vDS)

    • There are ways to accomplish this without the use of vSS’s. Specifically if you are on vSphere 5.1 or 5.5. You can export your existing vDS configuration from your old vCenter Server and then import it into your new vCenter. Make sure to preserve the identifiers though.

      Remember how vDS’s are designed to work. They continue to function even if their vCenter Server is offline. This means that after you disconnect a host from your old vCenter Server you can import it in your new environment and all VMs will continue to function properly during this transition. I will say, however, that even with the export/import you sometimes run into some anomalies in the new environment. Specifically VMs state that they are connected to their respective portgroups, however, the vDS side of things does not show the connection. This is easily remedied by re-assigning your uplinks one-by-one from your hosts to the new vDS. This is also a non-disruptive action.

      I am currently drafting a blog post on this very topic.

    • One method that I’ve seen many customers use is the VDS import/export feature that was introduced in 5.1/5.5 (forget the exact release). This will then allow you to re-build the exact VDS object (which you will need) for a clean migration.

  4. Thank you guys, at first sorry I am late..
    yes I have a powershell scirpt for Export/Iport vDS, including portgroups.

    However when I tested the migration, ESX couldn’t recognize VDS – of course it is from another vCenter. Yes VM on that vDS still can comunicate even I see in the vSphere client on the VM network property “Invalid backing”.
    Is that behavior related to preserve the identifiers as wrote tjpatter? What kind of identifiers or what is the purpose? I serched powercli doc and there is a decription to param KeepIdentifiers: “You cannot specify this parameter, when the Name parameter is specified.” So I think it should work in my script.
    And Should I preserve old vCenter ID to use this param KeepIdentifiers in the script? – it will be long time migration so I would like to avoid MAC addr. conflict and setup differnt ID on the new vCenter.

    and yes, I can re-assign my uplinks to the new vDS, however what about VMs? – then I have to re-assign all VMs network property “Invalid backing” to the correct portgroup..

    Can you send me the list of tasks step by step? I will appreciate it.

    • For N1000V you can use same procedure which is used for DVS, migrate first to VMware Standard switch, move ESXi hosts from old to new vCenter and again migrate from Standard to new N1000v.

  5. Hi William, I was just wondering if you had considered the implications of cross-vCenter migrations when using vCloud Director and vRealize Automation, particularly with vSphere 6?

Thanks for the comment!