How to easily disable vMotion & Cross vCenter vMotion for a particular Virtual Machine?

The question of disabling vMotion for a specific set of Virtual Machine(s) is not a new one. In fact, this topic comes up on some what of a frequent basis and usually driven by arcane change management processes or worse licensing restrictions. Do not get me wrong, there are definitely some valid use cases where you would not want a particular VM to be migrated off. The classic example is a 3rd Party VM solution that provides Anti-Malware, Intrusion Detection & Firewall capabilities for your workload VMs. For this particular use case, VMware provides our partners with an integration hook into the vSphere platform called ESX Agent Manager (EAM) that ensures these "Service VMs" are not allowed to be powered off or migrated to another ESXi host, even in the case of a Maintenance Mode operation. This solutuion even allows you to configure custom icons for your Service VMs!

For all other use cases outside of the "Service VMs", there really is not an easy way of disabling vMotion for a particular VM. There have been many solutions that have been suggested in the past ranging from disabling DRS for a specific VM, DRS Affinity Rules, VM miss-configurations to break vMotion compatibility to using vSphere Permissions to prevent vMotion operations. However, many of these solutions do not work very well or is very difficult to manage at scale. I actually like the idea of using vSphere Permissions to prevent a vMotion, however, I have seen some customers push back on this because the vSphere Administrator still has the ability to perform this operation. For these cases, customers just want to be able to completely disable vMotion for a given VM and prevent anyone from migrating the VM, including the vSphere Administrators.

Given that this topic had recently come up again, I was wondering if there was an easier way in which this could be achieved and made more manageable for our customers. After thinking about about how EAM handles "disabling" certain operations for a VM and recalling an article I wrote last year which leveraged this exact capability to resolve an NSX Controller issue, I thought why not apply it to this use case here?

Disclaimer: The use of internal APIs are not officially supported by VMware and can change at any time. Please use at your own risk.

Each VM has a property called DisableMethod which lists the specific vSphere API methods that are currently disabled. These are not governed by vSphere Permissions but rather the runtime state of the VM. For example, if you have a VM that is currently powered on, then the PowerOnVM_Task API would not be available and would show up in the disabled list.

Here is a quick PowerCLI snippet on how to retrieve the current set of disable methods for a VM:

$vm = Get-Vm -Name TestVM-1
$vm.ExtensionData.DisabledMethod

disable-vmotion-for-vm-1
As mentioned in this article, the ability to enable and disable these methods are only available as an internal vCenter Server API. However, it is possible to access these APIs using the vSphere MOB, but it is not very user friendly nor intuitive. Below is a screenshot of invoking the disableMethods API using the vSphere MOB.

disable-vmotion-for-vm-0
A couple of weeks back I started to investigate on how we might be able to automate against the vSphere MOB. The result of that investigation lead to the creation of a simple PowerCLI script that allows you to automate operations using the vSphere MOB which I had published here. That work became the foundation for the new PowerCLI script that I had created for disabling and enabling the vMotion capability for a particular VM.

You can download the PowerCLI script here called enable-disable-vsphere-api-method.ps1 which includes two functions Enable-vSphereMethod and Disable-vSphereMethod. You will need to edit the script to provide a couple of pieces of information.

  1. Credentials to your vCenter Server
  2. Name of the VM you wish to either disable or enable vMotion capability on
  3. Name of the vSphere API method you wish to disable (by default this is MigrateVM_Task which maps to the vMotion capability)

By default, I have commented out both functions usage, you will need to manually uncomment one of the lines based on the operation you wish to perform.

To Disable the vMotion capability, run the following:

Disable-vSphereMethod -vc_server $vc_server -vc_username $vc_username -vc_password $vc_password -vmmoref $vm_moref -disable_method $method_name

To Enable the vMotion capability if you had disabled it, run the following:

Enable-vSphereMethod -vc_server $vc_server -vc_username $vc_username -vc_password $vc_password -vmmoref $vm_moref -enable_method $method_name

After the script has completed, you can now re-run the command that we ran earlier to see which methods have been disabled and you should see that the MigrateVM_Task is now part of the disable methods.

disable-vmotion-for-vm-5
If we now login to either the vSphere Web/C# Client and right click on the VM that we had disabled vMotion on, you should also see that the Migrate option is now grayed out and unavailable. This behavior will be true for ALL users including those in the vSphere Administrators group.

disable-vmotion-for-vm-3
It is important to note that vMotion is not only disabled from the UI, but it is also disabled from the vSphere API standpoint which the UI is built on top of. Here is an example of trying to perform a vMotion using the PowerCLI Move-VM cmdlet and you can see that an error is thrown immediately stating that the method has been disabled.

disable-vmotion-for-vm-4
Note: The "self" text output from the PowerCLI command is actually something that you can specify as part of disabling the vMotion capability. This might be useful to specify a change control ID or some string to signal to the user who might be trying to perform the operation. Please refer to the script and search for the "self" keyword if you wish to change it.

The really nice thing about this solution is not only is it really easy to enable or disable, but it can also be managed at scale which many of the other solutions mentioned earlier start to break down. The last thing anyone would want is additional operational overhead to manage manage complex DRS rules (which can still be overridden through manual migrations) or additional vSphere Permissions which also runs into the same problem where a vSphere Administrator can still override by performing a manual migration. This solution does prevent both standard vMotion as well as the new Cross vCenter vMotion capability (both between same/different SSO Domain) that was introduced in vSphere 6.0. You do not need to be running vSphere 6.0 to be able to leverage this solution, this should actually work for almost all versions of vSphere. Lastly, enabling or disabling the functionality does not require any type of system restart or impact to your VM other than the ability to vMotion.

Limitations

Beyond artificially limiting what vSphere DRS and HA can do, I did observe an interesting behavior when a Maintenance Mode operation is performed. If you leave the "Move powered-off and suspended virtual machines to other hosts in the cluster" uncheck, then all VMs will be migrated off and the VMs that have vMotion disabled will reside on the ESXi host while it goes into Maintenance Mode. However, if you do check the box, I did find that the system would override setting and actually move the VM to another ESXi host. This is something to be aware of and may not be a bad thing depending on your requirements.

disable-vmotion-vm-6

Disabling Storage vMotion

You may have noticed that if the VM is in a powered off state, that the Migrate option is still available in the UI. The reason for this is that we only disabled vMotion but you are still allowed to perform a Storage vMotion. If you wish to also disable the Storage vMotion capability, then you will need to disable RelocateVM_Task vSphere API method as well.

Auditing vMotion and Storage vMotion Operations

With or without this solution, you may still want another level of confidence that a VM has either not migrated or migrated to authorized set of ESXi hosts. We can easily do so by auditing the VM's Event system and looking for migration events. The name of the vMotion event is called VmMigratedEvent and the name of the Storage vMotion event is called VmRelocatedEvent. Here is a sample script using the vSphere SDK for Perl that exercises this specific vSphere API and provides you with all the ESXi hosts a given VM might have migrated to. For those that rather consume the vSphere API using something like PowerCLI, here is a quick one-liner to extract vMotion events:

$vm = Get-VM TestVM-2
Get-VIEvent -Entity $vm | Where { $_.Gettype().Name -eq "VmMigratedEvent"} | Select CreatedTime, UserName, FullFormattedMessage | ft -wrap -AutoSize

disable-vmotion-for-vm-5

Remotely query an ESXi host without adding it to vCenter Server

I was browsing through the vSphere API Reference the other day and I came across an interesting vSphere API called QueryConnectionInfoViaSpec() which was introduced in vSphere 6.0. This new API is similiar to the existing QueryConnectionInfo() method which also allows you to remotely query an ESXi host without requiring you to add it to your vCenter Server Inventory, which I did not know about before. Some of the information that you can retrieve is whether it is currently being managed by an existing vCenter Server and if so, what is its IP Address, any registered Virtual Machines, networks and datastores that may be configured along with underlying host capabilities.

The main difference between this new API is that instead of accepting a list of arguments to the method, it accepts a HostConnectSpecIt is a minor difference, but what this allows you to do is to be able to re-use that exact same spec in the AddHost_Task() API if you determine that you would like to add the ESXi host to your vCenter Server Inventory after inspecting the remote ESXi host. I think this is a useful API if you want to validate that you are not stealing an ESXi host from another managed vCenter Server and if you are, at least you know which vCenter Server it was being managed by. To demonstrate this vSphere API, I have created a simple PowerCLI script called Get-RemoteESXi.ps1 which requires a vCenter Server login as well as the Hostname/IP Address and credentials of the remote ESXi host you wish to query.

Here is an example output of running the script which stores the returned output to the $results variable.

query-remote-esxi-without-adding-to-vcenter-server
If you want to check whether the ESXi host is being managed by an existing vCenter Server, you can check the following property which will hold the IP Address of the vCenter Server if it is currently being managed:

$result.Host.ManagementServerIp

If you want to check if there are running VMs already on the ESXi host, you can check the following property:

$result.vm

As mentioned earlier, there is a ton of other information returned from the remote ESXi host and I will leave it as an exercise for the reader to explore further.

Quick Tip - Minor changes on how to customize vSphere 6.0 Update 2 Web Client login UI

For those of you interested in customizing the vSphere Web Client login UI in the latest vSphere 6.0 Update 2 release, you may have found that after following the instructions here, the result is an empty background image. The reason this happens is that as of vSphere 6.0 Update 2, new capabilities such as a Logon Banner and 2-Factor Authentication using Smart Card support have been added. All of these capabilities are UI driven and as one would expect, they have updated HTML/CSS pages which from vSphere 6.0 or vSphere 6.0 Update 1. If you simply took the existing sample templates in my Github repo and replaced it on your system, the vSphere Web Client would not be able render them properly.

customize-vsphere-web-client-login-ui-vsphere-60-update-2
The fix is actually quite simple, instead of just copying the sample templates I have, you need to make a copy of the existing vSphere 6.0 Update 2 files (login.css & unpentry.jsp) and make the relevant code changes. Given the number questions I have received on this topic, I figure I would just create a new Github repo that contains vSphere 6.0 Update 2 specific sample themes.

Disclaimer: This is not officially supported by VMware, if you decide to enable this, please use at your own risk and ensure you backup all original files in case you need revert back to the original configurations.

For the full instructions and sample themes for customizing the vSphere Web Client login UI for vSphere 6.0 Update 2, have a look at the new Github repo here https://github.com/lamw/customize-vsphere-web-client-6.0u2

How to automate vSphere MOB operations using PowerShell?

A couple of weeks back I was investigating something that involved the use of the vSphere Managed Object Browser (MOB) and I needed to automate a particular operation. For those of you not familiar with the vSphere MOB, it is a UI debugging tool that allows you to visualize and invoke vSphere APIs using just a web browser. The vSphere MOB is available on both vCenter Server and ESXi and uses the underlying vSphere API. Outside of learning about the vSphere APIs and potentially for troubleshooting purposes, there really is no good reason to be directly interacting with the vSphere MOB on a regular basis. Customers can use any one of the many vSphere SDK/CLIs to easily automate and interact with the vSphere API.

Having said that, there may be cases where you might want to invoke a specific operation using the vSphere MOB, such as private API for example. Obviously, using any private/internal APIs is not officially supported by VMware and their use will be at your own risk. Nonetheless, I had a specific operation that I needed to call from the vSphere MOB and wanted to do so using PowerShell. Several years back I had demonstrated how you could invoke the vSphere MOB using Python but I could not find any PowerShell examples that actually worked. I figured this would be a good learning opportunity for myself and probably something I or others could benefit in the future.

To provide a concrete example, I will be invoking a supported vSphere API called the QueryOptions which is used for accessing either a vCenter Server or ESXi Advanced Settings. For our example, I will be connecting to the vCenter Server's MOB and will be querying for a specific vCenter Server Advanced Setting. If we were to do this manually in the vSphere MOB, we would first open a web browser and login to the following URL: https://[VC-SERVER]/mob/?moid=VpxSettings&method=queryView

automate-vsphere-mob-using-powercli
If you wanted to see all advanced settings, you would leave the "name" parameter blank. For our example, we will query for the VirtualCenter.InstanceName property as shown in the screenshot above and then click on the "Invoke Method" to execute the vSphere API operation. If successful, it should display the results which in our case is the IP Address of my vCenter Server. Pretty straight forward vSphere MOB example. OK, onto automating this from PowerShell. I have created a sample PowerShell script called automate-vsphere-mob.ps1 which you will need to edit and provide your vCenter Server Hostname/IP Address and the vCenter Server credentials (can be read-only) since we are not performing any write operations.

Here are some more specific details on what is happening in the script for those interested:

  1. Authenticate to the vSphere MOB URL that you wish to invoke using an HTTP GET operation and storing the session into a variable named vmware (Line 28-29)
  2. Extract the hidden vmware-session-nonce property used to prevent Cross-Site Request Forgery (CSRF) (Line 31-40)
  3. Create the body request which must include the vmware-session-nounce property along with the vSphere API payload (Line 42-45)
  4. Invoke the actual vSphere API call using an HTTP POST along with the existing session from Step 1 and body payload
  5. Process the results (this will be specific to API call) and profit! (Line 50-59)

If you now run the script, you will see that the result is exactly what we saw from interactively using the vSphere MOB. Hopefully this will be useful for anyone who may have a need to automate against the vSphere MOB. For those curious on what I was investigating with the vSphere MOB, stay tuned for more details 🙂

How to audit vSphere Standalone VMRC or HTML5 VMRC connections?

An interesting question that came in last week from one of our TAMs was how to identify and audit Virtual Machine Remote Console (VMRC) logins from vSphere? The TAM was specifically interested in being able to correlate that a particular user had logged into the VMRC of a VM during a specific period of time. Luckily, this is easily retrievable through vCenter Servers's Event sub-system that stores information about everything that happens in your vSphere environment. The Events can be accessed using either the vSphere Web Client shown below or programmatically using the vSphere API which the UI is built on top of.

audit-standalone-vmrc-and-html5-vmrc-logins-1
You can obviously filter your search in the UI and focus on a particular VM, but often times there can be dozens if not hundreds of "Events" generated for a given VM. I personally prefer to leverage Automation when needing to look for a specific type of Event and more importantly, you can further process the results to either send out reports or hook into other third party systems. Now that we know, "where" to find our data, the next thing is identifying the type of Event that is generated for a VMRC connection.

As of vSphere 5.5 Update 2b, the VMRC in the vSphere Web Client can be accessed in one of two ways: The new HTML5 VMRC by clicking onto the VM screenshot thumbnail or the Standalone VMRC by clicking on the link directly beneath the VM screenshot.

audit-standalone-vmrc-and-html5-vmrc-logins-0
Each VMRC connection method will generate a unique vCenter Server Event. For HTML5 VMRC connections, the Event is called VmAcquiredMksTicketEvent and for Standalone VMRC connections, the Event is called VmAcquiredTicketEvent. As I mentioned earlier, the vCenter Server Event sub-system can be accessed using the vSphere API and you can find the complete list of Events documented here. To demonstrate the use of this particular vSphere API, below is a PowerCLI example using the Get-VIEvent cmdlet. My fellow colleague Alan Renouf has actually blogged about working with Events using PowerCLI which I will be adapting one of his examples for our use case.

We first retrieve the VM that we are interested in by running the following command (specify the name of your VM):

$vm = Get-VM -Name "VCSA-60u2"

To retrieve HTML5 VMRC connections, run the following PowerCLI command:

Get-VIEvent -Entity $vm | Where { $_.Gettype().Name -eq "VmAcquiredMksTicketEvent"} | Select CreatedTime, UserName, FullFormattedMessage | ft -wrap -AutoSize

Here is an example of what the output would look like

audit-standalone-vmrc-and-html5-vmrc-logins-2
To retrieve Standalone VMRC connections, run the following PowerCLI command:

Get-VIEvent -Entity $vm | Where { $_.Gettype().Name -eq "VmAcquiredTicketEvent"} | Select CreatedTime, UserName, UserAgent, FullFormattedMessage | ft -wrap -AutoSize

Here is an example of what the output would look like:

audit-standalone-vmrc-and-html5-vmrc-logins-3