Quick Tip - How to retrieve the ESXi Update Level using the vSphere API?

Using the vSphere API, it is very easy to extract the version and build of all your ESXi hosts. This information is exposed in the Product property of an ESXi host. For example, Product.Version will return something like 6.0.0 and Product.Build will return something like 3029758. However, one thing that is not available in this property is the Update Level information for an ESXi host such as Update 1 or Update 2.

Historically, customers would have to rely on ESXCLI to pull the Update level information using the following command: esxcli system version get and though this can be run remotely or integrated into PowerCLI as shown in the example below, it would be ideal if this information was just available using the vSphere API.

This exact same question was brought up internally again today and Etienne Le Sueur actually shared an awesome tidbit on how to retrieve this information using the vSphere API. You can find the ESXi Update Level information in an ESXi Advanced Setting called Misc.HostAgentUpdateLevel

Below is a quick PowerCLI example which exercises this vSphere API to retrieve the Version, Build and Update Level information:

Here is a screenshot of the output for my ESXi host which is running latest vSphere 6.0 Update 2 (including the recent patch release).

retrieve-esxi-update-level-using-vsphere-api

Its great to hear that the ESXi Update Level information is available through the vSphere API, although I would have liked to have seen it exposed within the Product property. Perhaps its time to file an internal Feature Request? 🙂

Automating the import of vCenter Server 6.x root certificate

In vSphere 6.0, you can now easily import your vCenter Server's trusted root CA certificate onto your client desktop by simply downloading it from the vCenter Server's landing page as shown in the screenshot below. Michael White had also recently wrote about this topic here which includes a step by step walk through.

automate-import-of-vcenter-server-root-certificate-3
Several weeks back I was working on an internal project which required the vCenter Server's root certificate. I was already aware of this interface and had written a quick and dirty script to automate the process of downloading and importing the certificate to the system I was working on. To be honest, I did not think much of the script after I wrote it. It was just recently that Alan Renouf, who was also involved in the project mentioned that it might be worth sharing the script as others might also find it useful. I thought that was a good idea and re-factored the code a bit since it was being used in a slightly different context. While doing so, I also created an equivalent PowerShell sample since the original script was meant to run on either a Mac OS X or Linux platform.

With that, I have created a simple shell script called import-vcrootcertificate.sh which can run on either Mac OS X or Linux system and a PowerShell script called Import-VCRootCertificate.ps1

Both scripts are pretty easy to use, they accept a single command-line argument which is the Hostname/IP Address of the vCenter Server that you wish to import the root certificate from. Both scripts ere able to detect if the vCenter Server is Windows or the VCSA since they have a slightly different URL to the root certificate before performing the import. Since the script will need access to your certificate store, you will need to run the scripts using a privileged account.

Here is a screenshot of running the PowerShell script:

automate-import-of-vcenter-server-root-certificate-0
Here is a screenshot of running the shell script:

automate-import-of-vcenter-server-root-certificate-2

Quick Tip - How to disable the landing page for vCenter Server 5.x & 6.x?

The question of wanting to disable the default landing page for the vCenter Server is one that comes up infrequently. In fact, I probably see this maybe once or twice a year. However, when it does come up, it usually revolves around two topics: some sort of security risk and limiting users from obtaining software provided through these landing pages. In both case, simply disabling these landing pages will not solve either of these perceived issues.

I generally find these landing pages quite useful as they provide links to software downloads such as our legacy vSphere C# Client, SDK documentation as well as links to other interfaces to vCenter Server like the vSphere Web Client login, the datastore browser or the vSphere MOB. All of this information can be obtained through other official channels, so simply disabling this page does not really prevent users from downloading this content or accessing these interfaces.

On the second topic around security (which by no means am I an expert in), some customers feel that simply removing these default landing pages would some how prevent a security risk because a version of the software is no longer listed on that page? This is what some folks would call security through obscurity which just does not work. There are many different ways of identifying a version of vCenter Server and some of its components as well checking if the service is running. Simply removing these pages does little to nothing from stopping someone from retrieving this information using other methods. Instead, users should really be focusing how they are implementing security both in the software as well as the policies and processes they have in place which hopefully are inline with modern security practices.

In fact, by disabling some of these pages, you might even be hurting your overall customer experience depending on their familiarity with vCenter Server.

In any case, for those that are still inclined to disable these pages, below are the instructions on how to disable the various landing pages as I have not really seen this documented anywhere. The solution is actually quite simple which is to just rename the index files to something else which will prevent them from being loaded by the webserver.

Landing page for vCenter Server 5.x 

  • Windows VC: C:\ProgramData\VMware\VMware VirtualCenter\docRoot\index.html
  • VCSA: /etc/vmware-vpx/docRoot/index.html

disable-vcenter-server-landing-splash-page-0
Tomcat landing page for vCenter Server 5.x

  • Windows VC: C:\Program Files\VMware\Infrastructure\tomcat\webapps\ROOT\index.jsp
  • VCSA: /usr/lib/vmware-vpx/tomcat/webapps/tomcat/webapps/ROOT/index.jsp

disable-vcenter-server-landing-splash-page-1
Landing page for vCenter Server 6.x 

  • Windows VC: C:\ProgramData\VMware\VMware VirtualCenter\docRoot\index.html
  • VCSA: /etc/vmware-vpx/docRoot/index.html

disable-vcenter-server-landing-splash-page-2
Landing page for Platform Services Controll (vSphere 6.x)

  • Windows VC: C:\ProgramData\VMware\vCenterServer\runtime\VMwareSTSService\webapps\websso\WEB-INF\views\index.jsp
  • VCSA: /usr/lib/vmware-sso/vmware-sts/webapps/websso/WEB-INF/views/index.jsp

disable-vcenter-server-landing-splash-page-3

How to easily disable vMotion & Cross vCenter vMotion for a particular Virtual Machine?

The question of disabling vMotion for a specific set of Virtual Machine(s) is not a new one. In fact, this topic comes up on some what of a frequent basis and usually driven by arcane change management processes or worse licensing restrictions. Do not get me wrong, there are definitely some valid use cases where you would not want a particular VM to be migrated off. The classic example is a 3rd Party VM solution that provides Anti-Malware, Intrusion Detection & Firewall capabilities for your workload VMs. For this particular use case, VMware provides our partners with an integration hook into the vSphere platform called ESX Agent Manager (EAM) that ensures these "Service VMs" are not allowed to be powered off or migrated to another ESXi host, even in the case of a Maintenance Mode operation. This solutuion even allows you to configure custom icons for your Service VMs!

For all other use cases outside of the "Service VMs", there really is not an easy way of disabling vMotion for a particular VM. There have been many solutions that have been suggested in the past ranging from disabling DRS for a specific VM, DRS Affinity Rules, VM miss-configurations to break vMotion compatibility to using vSphere Permissions to prevent vMotion operations. However, many of these solutions do not work very well or is very difficult to manage at scale. I actually like the idea of using vSphere Permissions to prevent a vMotion, however, I have seen some customers push back on this because the vSphere Administrator still has the ability to perform this operation. For these cases, customers just want to be able to completely disable vMotion for a given VM and prevent anyone from migrating the VM, including the vSphere Administrators.

Given that this topic had recently come up again, I was wondering if there was an easier way in which this could be achieved and made more manageable for our customers. After thinking about about how EAM handles "disabling" certain operations for a VM and recalling an article I wrote last year which leveraged this exact capability to resolve an NSX Controller issue, I thought why not apply it to this use case here?

Disclaimer: The use of internal APIs are not officially supported by VMware and can change at any time. Please use at your own risk.

Each VM has a property called DisableMethod which lists the specific vSphere API methods that are currently disabled. These are not governed by vSphere Permissions but rather the runtime state of the VM. For example, if you have a VM that is currently powered on, then the PowerOnVM_Task API would not be available and would show up in the disabled list.

Here is a quick PowerCLI snippet on how to retrieve the current set of disable methods for a VM:

$vm = Get-Vm -Name TestVM-1
$vm.ExtensionData.DisabledMethod

disable-vmotion-for-vm-1
As mentioned in this article, the ability to enable and disable these methods are only available as an internal vCenter Server API. However, it is possible to access these APIs using the vSphere MOB, but it is not very user friendly nor intuitive. Below is a screenshot of invoking the disableMethods API using the vSphere MOB.

disable-vmotion-for-vm-0
A couple of weeks back I started to investigate on how we might be able to automate against the vSphere MOB. The result of that investigation lead to the creation of a simple PowerCLI script that allows you to automate operations using the vSphere MOB which I had published here. That work became the foundation for the new PowerCLI script that I had created for disabling and enabling the vMotion capability for a particular VM.

You can download the PowerCLI script here called enable-disable-vsphere-api-method.ps1 which includes two functions Enable-vSphereMethod and Disable-vSphereMethod. You will need to edit the script to provide a couple of pieces of information.

  1. Credentials to your vCenter Server
  2. Name of the VM you wish to either disable or enable vMotion capability on
  3. Name of the vSphere API method you wish to disable (by default this is MigrateVM_Task which maps to the vMotion capability)

By default, I have commented out both functions usage, you will need to manually uncomment one of the lines based on the operation you wish to perform.

To Disable the vMotion capability, run the following:

Disable-vSphereMethod -vc_server $vc_server -vc_username $vc_username -vc_password $vc_password -vmmoref $vm_moref -disable_method $method_name

To Enable the vMotion capability if you had disabled it, run the following:

Enable-vSphereMethod -vc_server $vc_server -vc_username $vc_username -vc_password $vc_password -vmmoref $vm_moref -enable_method $method_name

After the script has completed, you can now re-run the command that we ran earlier to see which methods have been disabled and you should see that the MigrateVM_Task is now part of the disable methods.

disable-vmotion-for-vm-5
If we now login to either the vSphere Web/C# Client and right click on the VM that we had disabled vMotion on, you should also see that the Migrate option is now grayed out and unavailable. This behavior will be true for ALL users including those in the vSphere Administrators group.

disable-vmotion-for-vm-3
It is important to note that vMotion is not only disabled from the UI, but it is also disabled from the vSphere API standpoint which the UI is built on top of. Here is an example of trying to perform a vMotion using the PowerCLI Move-VM cmdlet and you can see that an error is thrown immediately stating that the method has been disabled.

disable-vmotion-for-vm-4
Note: The "self" text output from the PowerCLI command is actually something that you can specify as part of disabling the vMotion capability. This might be useful to specify a change control ID or some string to signal to the user who might be trying to perform the operation. Please refer to the script and search for the "self" keyword if you wish to change it.

The really nice thing about this solution is not only is it really easy to enable or disable, but it can also be managed at scale which many of the other solutions mentioned earlier start to break down. The last thing anyone would want is additional operational overhead to manage manage complex DRS rules (which can still be overridden through manual migrations) or additional vSphere Permissions which also runs into the same problem where a vSphere Administrator can still override by performing a manual migration. This solution does prevent both standard vMotion as well as the new Cross vCenter vMotion capability (both between same/different SSO Domain) that was introduced in vSphere 6.0. You do not need to be running vSphere 6.0 to be able to leverage this solution, this should actually work for almost all versions of vSphere. Lastly, enabling or disabling the functionality does not require any type of system restart or impact to your VM other than the ability to vMotion.

Limitations

Beyond artificially limiting what vSphere DRS and HA can do, I did observe an interesting behavior when a Maintenance Mode operation is performed. If you leave the "Move powered-off and suspended virtual machines to other hosts in the cluster" uncheck, then all VMs will be migrated off and the VMs that have vMotion disabled will reside on the ESXi host while it goes into Maintenance Mode. However, if you do check the box, I did find that the system would override setting and actually move the VM to another ESXi host. This is something to be aware of and may not be a bad thing depending on your requirements.

disable-vmotion-vm-6

Disabling Storage vMotion

You may have noticed that if the VM is in a powered off state, that the Migrate option is still available in the UI. The reason for this is that we only disabled vMotion but you are still allowed to perform a Storage vMotion. If you wish to also disable the Storage vMotion capability, then you will need to disable RelocateVM_Task vSphere API method as well.

Auditing vMotion and Storage vMotion Operations

With or without this solution, you may still want another level of confidence that a VM has either not migrated or migrated to authorized set of ESXi hosts. We can easily do so by auditing the VM's Event system and looking for migration events. The name of the vMotion event is called VmMigratedEvent and the name of the Storage vMotion event is called VmRelocatedEvent. Here is a sample script using the vSphere SDK for Perl that exercises this specific vSphere API and provides you with all the ESXi hosts a given VM might have migrated to. For those that rather consume the vSphere API using something like PowerCLI, here is a quick one-liner to extract vMotion events:

$vm = Get-VM TestVM-2
Get-VIEvent -Entity $vm | Where { $_.Gettype().Name -eq "VmMigratedEvent"} | Select CreatedTime, UserName, FullFormattedMessage | ft -wrap -AutoSize

disable-vmotion-for-vm-5

Remotely query an ESXi host without adding it to vCenter Server

I was browsing through the vSphere API Reference the other day and I came across an interesting vSphere API called QueryConnectionInfoViaSpec() which was introduced in vSphere 6.0. This new API is similiar to the existing QueryConnectionInfo() method which also allows you to remotely query an ESXi host without requiring you to add it to your vCenter Server Inventory, which I did not know about before. Some of the information that you can retrieve is whether it is currently being managed by an existing vCenter Server and if so, what is its IP Address, any registered Virtual Machines, networks and datastores that may be configured along with underlying host capabilities.

The main difference between this new API is that instead of accepting a list of arguments to the method, it accepts a HostConnectSpecIt is a minor difference, but what this allows you to do is to be able to re-use that exact same spec in the AddHost_Task() API if you determine that you would like to add the ESXi host to your vCenter Server Inventory after inspecting the remote ESXi host. I think this is a useful API if you want to validate that you are not stealing an ESXi host from another managed vCenter Server and if you are, at least you know which vCenter Server it was being managed by. To demonstrate this vSphere API, I have created a simple PowerCLI script called Get-RemoteESXi.ps1 which requires a vCenter Server login as well as the Hostname/IP Address and credentials of the remote ESXi host you wish to query.

Here is an example output of running the script which stores the returned output to the $results variable.

query-remote-esxi-without-adding-to-vcenter-server
If you want to check whether the ESXi host is being managed by an existing vCenter Server, you can check the following property which will hold the IP Address of the vCenter Server if it is currently being managed:

$result.Host.ManagementServerIp

If you want to check if there are running VMs already on the ESXi host, you can check the following property:

$result.vm

As mentioned earlier, there is a ton of other information returned from the remote ESXi host and I will leave it as an exercise for the reader to explore further.