Potential ESXi Host Preparation issues with NSX 6.3

While working on updating my vGhetto Automated vSphere Lab Deployment script to add support for NSX 6.3 with vSphere 6.5, I ran into an issue with the Host Preparation step. Although the resolution turned out to be quite simple, it was very difficult to diagnose the problem. I suspect this scenario could easily be encountered by others, so I wanted to make folks aware of what I ran into. There is also another potential gotcha for host preparation that I did not encounter myself, but it was brought to my attention that I thought was also worth sharing as well.

Scenario 1 - Attempted Host Preparation and all "Install agent" tasks fails with "Cannot complete the operation. See the event log for details" and below is a screenshot of the error. There was nothing useful when looking at the event logs for either NSX or ESXi using the vSphere Web Client.


There was also nothing useful in the ESXi log /var/log/esxupdate.log that gave insights to why the NSX VIBs failed to install:

2017-02-16T12:38:53Z esxupdate: 73899: Transaction: DEBUG: Populating VIB list from all VIBs in metadata https://vcenter65-1.primp-industries.com:443/eam/vib?id=d4917629-51d1-4da9-82d6-8da54815447d; depots:
2017-02-16T12:38:54Z esxupdate: 73899: downloader: DEBUG: Downloading https://vcenter65-1.primp-industries.com:443/eam/vib?id=d4917629-51d1-4da9-82d6-8da54815447d to /tmp/tmpdfcbr23q...
2017-02-16T12:38:54Z esxupdate: 73899: Metadata.pyc: INFO: Unrecognized file vendor-index.xml in Metadata file
2017-02-16T12:38:54Z esxupdate: 73899: imageprofile: INFO: Adding VIB VMware_locker_tools-light_6.5.0-0.0.4564106 to ImageProfile (Updated) ESXi-6.5.0-4564106-standard
2017-02-16T12:38:54Z esxupdate: 73899: imageprofile: INFO: Adding VIB VMware_bootbank_esx-vsip_6.5.0-0.0.4987428 to ImageProfile (Updated) ESXi-6.5.0-4564106-standard
2017-02-16T12:38:54Z esxupdate: 73899: imageprofile: INFO: Adding VIB VMware_bootbank_esx-vxlan_6.5.0-0.0.4987428 to ImageProfile (Updated) ESXi-6.5.0-4564106-standard
2017-02-16T12:38:54Z esxupdate: 73899: vmware.runcommand: INFO: runcommand called with: args = '['/bin/localcli', 'system', 'maintenanceMode', 'get']', outfile = 'None', returnoutput = 'True', timeout = '0.0'.
2017-02-16T12:38:54Z esxupdate: 73899: HostInfo: INFO: localcli system returned status (0) Output: Disabled Error:

Continue reading

Update on Intel NUC 7th Gen (Kaby Lake) & ESXi 6.x

Intel just started shipping their i3 7th Gen (Kaby Lake) Intel NUCs and no surprise, Florian of virten.net has already gotten his hands on a unit for testing. The Intel NUCs is a very popular platform for running vSphere/vSAN-based home labs, especially for their price point and footprint. Last week, Florian discovered from his testing that the built-in network adapter on the 7th Gen NUC was not being detected by any of the ESXi installers and had published his findings here.

Since the Intel NUC is not an officially supported platform for ESXi, it does not surprise me that these sort of things happen, even if the NUC had a pretty good track record going back to the 5th Gen releases. Nonetheless, I reached out to Florian to see if he can provide me with a vm-support bundle and see if there was anything I could do to help.

A couple of Engineers took a look and quickly identified the issue with the Kabylake NIC (8086:15d8) but before getting to the solution, I did want to clarify something. The Kabylake NIC is actually NOT an officially supported NIC for ESXi and although it currently shows up in the VMware HCL, it is a mistake. I have been told the VMware HCL will be updated shortly to reflect this, apologies for any confusion that this may have caused.

Ok, so the great news is that we do have a solution for getting ESXi to recognize the built-in NIC on the 7th Gen Intel NUCs. The semi-bad news is that we currently do not have a solution in the short term for any released version of ESXi, as the fix will require an updated version of the e1000e Native Driver which will only be available in a future update of ESXi. I can not provide any timelines, but keep an eye on this blog and I will publish more details once they are available.

In the meantime, if you already own a 7th Gen NUC, there is a workaround which Florian has already blogged about here which uses the USB Ethernet Adapter VIB for the initial ESXi installation. If you are planning to purchase the 7th Gen NUC and would like to wait for folks to confirm the fix, then I would recommend holding off or potentially looking at the 6th Gen if you can not wait. Thanks to Florian and others who shared their experiences with the 7th Gen NUC and also to the VMware Engineers who found a quick resolution to the problem.

Using the vSphere API in vCenter Server to collect ESXTOP & vscsiStats metrics

Back in 2013, I wrote two articles here and here on how to use the Service Manager API that was introduced in vSphere 5.1 to remotely collect ESXTOP and vscsiStats metrics. At the time, I was told that this API was only available when connecting directly to an ESXi host using the vSphere API. This was still a huge improvement over the previous method which basically required customers to run the commands locally. For ESXTOP, there was resxtop but it was not available for all platforms and some customers still prefer to use the ESXi Shell's version. When I had learned about this API, I was really hoping I could collect both ESXTOP & vscsiStats metrics using vCenter Server which would remove the need to have direct connectivity to each ESXi host.

Last week, an Engineer came across one of my blog posts related to the Service Manager APIs which helped him with a problem he was trying to solve. In the email conversation, I then came to learn from the Engineer that the the Service Manager API can be used from vCenter Server and going directly to the ESXi host was not necessary. It turns out that the QueryServiceList() method which accepts an array of "location" expects a special keyword prefix appended to the list of ESXi hosts that you wish to use the local Service Manager instances on.

The special keyword prefix is "vmware.host." and this is appended to either the Hostname or IP Address of the ESXi hosts being managed by the vCenter Server. For example, in my environment I have an ESXi host (192.168.1.50) that is managed by my VC and so the location string for that host should be "vmware.host.192.168.1.50". If the method was successfully called, you should get back the two service instances for ESXTOP and vscsiStats for each of the ESXi hosts where you can then perform the metric collection.

I have created two sample pyvmomi scripts which exercises the Service Manager API for ESXTOP and vscsiStats:

Note: For more details on how to use Service Manager API to collect ESXTOP and vscsiStats, please refer to this post here and here.

Here is an example of running the ESXTOP sample script and executing the "CounterInfo" operation:


Here is an example of running the vScsiStats sample script and executing the "FetchAllHistograms" operation:


I had initially tried to create the two sample scripts using PowerCLI, but it looks like there may be a bug with PowerCLI when calling the Service Manager instance.

If you try to run the following snippet which retrieves the view for the Service Manager

Get-View $global:DefaultVIServer.ExtensionData.Content.ServiceManager

You will find that it throws the following error. I have already filed an internal bug and hopefully this get resolved in a future PowerCLI update so that customers who wish to retrieve ESXTOP and vScsiStats can do so using PowerCLI. For all other vSphere SDKs, it should work as expected.

Exploring new VCSA VAMI API w/PowerCLI: Part 8

In Part 8, we are going to take a look how services are managed in a VCSA or PSC node which is provided by the vCenter Server Lifecycle Management system also referred to internally as vMon. You can interact with the vMon service using either the service-control utility which is only available via SSH or the VAMI APIs which are available remotely. As you probably have guessed, we will be using the VAMI APIs 🙂

VAMI UI Area of Focus

There is not a service view in the VAMI UI (https://[VCSA]:5480) for either the VCSA or PSC. However, this information is available as part of the VAMI information when logged into the vSphere Web Client by navigating to System Configuration->Nodes->Related Objects or System Configuration->Services.

VAMI APIs Used

  • GET /appliance/vmon/service
  • POST /appliance/vmon/service/start
  • POST /appliance/vmon/service/stop

PowerCLI Function

Sample Output

The Get-VAMIService will lists all available services for the given VCSA or PSC node that you are connected to. It provides the exact same output that you would see in the vSphere Web Client such as the name of the service, the current state, the health and whether the service is disabled or configured to start up automatically or manually.


The function also accepts a name parameter if you know the specific service you wish to query, for example here is the syntax for checking the Auto Deploy service which is named rbd:

Get-VAMIService -Name rbd

We can use the Start-VAMISerivce function and given a service name, we can start it as shown in the screenshot below.


Similialy, we can use the Stop-VAMISerivce function and given a service name to stop the service as shown in the screenshot below.

How to forward other VCSA 6.5 logs to remote syslog server?

As mentioned in my previous article (which I strongly recommend you review before continuing further), the VCSA 6.5 no longer uses syslog-ng as the syslog client and it has been replaced with rsyslog. This means the instructions outlined in my old article here is no longer valid on forwarding logs from a VCSA 6.5 system to a remote syslog server. Luckily, the process to forward logs within VCSA 6.5 is also pretty straight forward using rsyslog.

Disclaimer: This is not officially supported by VMware, please use at your own risk. For very large environments, forwarding additional logs can potentially impact the vCenter Server service, so please take caution in the logs you decide on forwarding and test in a lab environment before applying this across your environment.

To help provide a concrete example, I will be using a real world scenario that often comes up from customers on auditing failed vSphere Web Client login success/failures as well as SSO user creation, deletion and password changes. The following two log files provides us with this information which we will forward to our syslog server:

  • /var/log/vmware/sso/ssoAdminServer.log - Auditing SSO logins
  • /var/log/vmware/sso/vmware-identity-sts.log - Auditing SSO user changes

We will be making using of rsyslog Text File Input Module (imfile) which will allow us to process local log files in the VCSA.

Continue reading