An interesting question that came up on the VMTN forum the other day (thanks to Andreas Peetz for sharing via Twitter) was how to split two vCenter Servers configured in an Enhanced Linked Mode (ELM)? Due to an organization changes in the customers environment, they needed to separate out their two vCenter Servers and run them independently of each other. Although this may sound like an rare event, I have actually seen this use case come up several times now which maybe from a business unit restructuring, spinning out or selling off company assets which then requires the customer to split their existing vCenter Servers that is configured with ELM.
Below is a diagram depicting an example where the original source environment (left) which is composed of two vCenter Servers and two external Platform Services Controller (PSC) configured in an ELM and the desired destination environment (right) which are two separate vCenter Server instances no longer configured in ELM.
The solution to this problem is actually pretty straight forward and leverages the existing vCenter Server and/or Platform Services Controller (PSC) "decommission" workflow. Rather than decommissioning the nodes, we are just simply keeping them around. Below are the instructions on how to achieve this outcome.
Disclaimer: Although this solution uses an existing supported workflow, this particular use case has not been tested by VMware. As such, this would not be officially supported by VMware until the appropriate testing has been done by our Engineering teams. One potential option in the short term if you are looking for support from VMware is to file an RPQ request through your VMware account team.
- Environment running vSphere 6.0 or greater
- Enhanced Linked Mode configured w/External PSC (e.g. No ELM using Embedded vCenter Server)
- SSH/RDP access and VM Console access to PSCs
Here is a screenshot of my vSphere 6.5 environment configured like the diagram above. At the end of this article, we would have walked through the process to split up our ELM configuration and have two independent vCenter Server instances running while preserving their existing configurations.
Step 1 - Verify the existing environment to ensure there are no unknown PSCs or vCenter Servers that are attached to the environment that you may not be aware of, this could include decommissioned PSCs that were not properly removed. To do so, you will need to SSH into each of the PSCs and run the following commands.
You can use the updated dir-cli command to list all nodes (VC and PSCs) within an SSO Domain which is what an ELM is comprised of. Specify the SSO Administrator username as well as the password as shown in the example below:
/usr/lib/vmware-vmafd/bin/dir-cli nodes list --login 'firstname.lastname@example.org' --password 'VMware1!' --server-name localhost
As you can see from the output, we are able to list all nodes (VC and PSC) along with their respective PSC replication partners. What we are looking for is to verify the environment and that PSCs are replicating with the expected systems before we proceed to the next step.
Note: In case you are not seeing the "nodes list" option (new in 6.5 if I recall correctly), you will need to use the vdcrepadmin utility instead.
/usr/lib/vmware-vmdir/bin/vdcrepadmin -f showpartners -h localhost -u administrator -w VMware1!
vdcrepadmin only outputs the replication partners of the PSC, it will not show the VC nodes. However, you can get the list of all VC nodes connected to SSO Domain by simply logging into the vSphere Web Client using any one of the VCs to see the rest (this is basically what ELM provides).
Step 2 - Next, we need to prevent both psc-01 and psc-02 from talking to each other before we break the ELM configuration. This can be done in a variety of ways, but the quickest method is to simply disconnect the vNIC temporarily on psc-02 (only needs to be done on one node, so I chose the second). When a given PSC node is unreachable and when we perform the "decomission" operation, it will automatically comply and allow us to remove the replication partner. A verification step before moving on to the next step is to ensure you can no longer ping psc-02 from psc-01.
Step 3 - Login via SSH to the first PSC (psc-01) and using the cmsso-util to decomission the second PSC (psc-02) by running the following command (replace with your SSO admin credentials):
cmsso-util unregister --node-pnid psc-02.primp-industries.com --username 'email@example.com' --passwd 'VMware1!'
Step 4 - After decommissioning the PSC, we will also need to decommission the other VC (vcenter65-2). We will use the exact same command but now replace it with the second VC by running the following command (replace with your SSO admin credentials):
cmsso-util unregister --node-pnid vcenter65-2.primp-industries.com --username 'firstname.lastname@example.org' --passwd 'VMware1!'
Step 5 - Now, we need to perform a similiar operation for the second PSC (psc-02). Ensure you do NOT re-enable the vNIC on the second PSC yet. Login to the VM Console of psc-02 and then decommission the first PSC (psc-01) and then do the same for the first VC (vcenter65-1) by running the following two commands (replace with your SSO admin credentials):
cmsso-util unregister --node-pnid psc-01.primp-industries.com --username 'email@example.com' --passwd 'VMware1!'
cmsso-util unregister --node-pnid vcenter65-1.primp-industries.com --username 'firstname.lastname@example.org' --passwd 'VMware1!'
Step 6 - Once you have successfully completed Step 5, you can now re-enable the vNIC on psc-02.
At this point, you have now successfully split the second VC and you can confirm by logging into the second VC's vSphere Web Client as shown in the screenshot below.
The instructions above assume that you are using a vCenter Server Appliance (VCSA) but this can also be applied to a Windows-based vCenter Server and PSC. The paths for the respective utilities are as follows:
- C:\Program Files\VMware\vCenter Server\bin\cmsso-util
Note: The techniques described above for breaking up two vCenter Servers configured using ELM should also work for n-vCenter Servers. You simply just need to ensure that all PSCs can not talk to each other while you perform the decommissioning of the other nodes.