VMXNET3 driver now included in Mac OS X 10.11 (El Capitan)+

Yesterday I received a pretty interesting comment from one of my Twitter followers @NTmatter who wrote:

This is a pretty neat find because currently today, the only network adapter that is functional with an Apple Mac OS X guest running on either VMware vSphere or Fusion is the e1000{e} driver. This update was definitely news to me and after sharing it internally to see if I could find some more details, it turns out this news also came as surprise to the folks internally. Darius, one of the Engineers who I frequently reach out to on Apple related topics did some digging and found out that Apple started to bundle this VMXNET3 driver starting with Mac OS X 10.11 (El Capitan) release. You can find the driver located in /System/Library/Extensions/IONetworkingFamily.kext/Contents/PlugIns/AppleVmxnet3Ethernet.kext

Disclaimer: Given that this VMXNET3 Mac OS X driver was not developed by VMware nor has it been tested by VMware, it currently would not be officially supported by VMware.

If you wish to try out the VMXNET3 driver, you will need to install Mac OS X 10.11 or newer on a VM running on vSphere or Fusion. By default, the only available network adapter type is e1000{e}. To add a VMXNET3 network adapter, you can either manually tweak the .VMX file or you can easily add it by using either the vSphere Web/C# Client or ESXi Embedded Host Client. Below are the instructions on configuring the VMXNET3 network adapter for your Mac OS X guests.

Step 1 - Remove the existing network adapter and then temporarily change the GuestOS type to "Other" (no need to save setting, just update it in VM reconfigure wizard) so that you will be allowed to add a VMXNET3 network adapter. Once you have added it to the VM reconfigure wizard, go ahead and toggle back the GuestOS type to Mac OS X 10.10 and then save the settings as shown in the screenshots below.

mac-os-x-el-capitan-10-11-vmxnet3-driver-0
mac-os-x-el-capitan-10-11-vmxnet3-driver-1
Step 2 - Open a terminal inside of the Mac OS X guest and run the following command to load the VMXNET3 driver:

sudo kextload /System/Library/Extensions/IONetworkingFamily.kext/Contents/PlugIns/AppleVmxnet3Ethernet.kext

Step 3 - You can verify that the VMXNET3 driver was successfully loaded by running the following command:

kextstat | grep -i vmxnet3

mac-os-x-el-capitan-10-11-vmxnet3-driver-2
Once the driver has been loaded, you should now have networking connectivity to your Mac OS X VM using the VMXNET3 network adapter. Below is a screenshot of the system info showing the VMXNET3 network adapter.

mac-os-x-el-capitan-10-11-vmxnet3-driver-3

In addition to having an optimized networking when using the VMXNET3 driver, the other benefit is being able to get a link speed of 10GbE which is something customers have been inquiring about when virtualizing Mac OS X guests. Below is a screenshot of the media link shown in this Mac OS X 10.11 guests.

mac-os-x-el-capitan-10-11-vmxnet3-driver-4

Although this a great development for Apple customers who uses VMware vSphere and Fusion, it also does raise an interesting question on whether Apple would be officially supporting this VMXNET3 driver going forward? If I do receive any more details on this, I will update the article. Until then, you can play with this new capability if you are running Mac OS X 10.11 or greater on VMware. Big thanks to Thomas for this great find and sharing it with the VMware Community!

How to check the size of your Config + Stats, Events, Alarms & Tasks (SEAT) data in the VCDB?

I think many of you know that I am not a fan of anyone poking around in the vCenter Server Database (VCDB) and having to manually craft SQL queries to retrieve information about their vSphere environment. This is especially true when you can easily and painlessly retrieve all of this information by simply using the vSphere API.

Having said that, there is one use case that is currently not available today in the API, yet. The use case that I am referring to is having better visibility into the storage utilization of our VCDB for things like the Core inventory configuration as well as the Stats, Events, Alarms and Tasks (SEAT) data which generally makes up the bulk of the VCDB data. Some of the benefits to having this information includes understanding the size of your VCDB given your current inventory size + data retention policy, whether or not you should consider reducing/truncating your dataset and even ensuring that vCenter Server rollup jobs have properly ran by simply getting visibility into the current storage footprint of your VCDB.

The other really nice benefit of having this information for those looking to use the recently released VCSA Migration Tool (migrating from Windows vCenter Server to the vCenter Server Appliance) is that it can be used to help calculate the estimated amount of downtime that is required for the migration to complete. The process is currently outlined in the following VMware KB 2146420 which requires customers to manually run a specific SQL query to retrieve the specific tables within the VCDB, perform some basic arithmetic with the results and then plugging them into an excel spreadsheet to provide the time estimations for migration.

Note: The migration time estimates from VMware are just that, estimates. There are many other factors such as source and destination hardware capabilities, network and storage bandwidth that may influence the amount of time a migration may take. It is recommended that customers use the estimates as guidance and still add a time buffer to their maintenance window.

To help simplify the consumption of the KB, I have created a small PowerShell script called Get-VCDBUsage.ps1 which will allow you to remotely connect to your VCDB (assuming you have enabled remote connectivity) to execute the correct SQL query based on your database platform and provide you with the results. The script also includes an optional parameter which will automatically take the results and calculate the estimated amount of downtime required for migrating from your Windows based vCenter Server to the VCSA. This makes gathering the information about your VCDB quite easy without having to manually go through the KB which can be challenging if you have a large amount of vCenter Servers.

The script supports the following 3 modes:

  • Running "locally" on the Microsoft SQL Server DB (requires Windows PowerShell Extensions for SQL Server as I rely on the Invoke-Sqlcmd cmdlet)
  • Running "remotely" connected to the Microsoft SQL Server DB
  • Running "remotely" connected to the Oracle DB (requires Oracle ODAC Client to be installed on the Windows system running the script)

For the first mode, you only need to specify the dbType, connectionType and dbInstance parameters as it will use the existing local ODBC connection so you do not have to provide any DB credentials. Here is an example command:

Get-VCDBUsage -dbType mssql -connectionType local -dbInstance VCDB

For the second mode, you will need to specify the dbType, connectionType, dbServer, dbPortdbInstance, dbUsername and dbPassword parameters as you will be connecting remotely and the additional DB information will be needed. Here is an example command:

Get-VCDBUsage -dbType mssql -connectionType local -dbServer sql.primp-industries.com -dbPort 1433 -dbInstance VCDB -dbUsername sa -dbPassword VMware1!

Here is a screenshot of what the output would look like whether you run this against a VCDB running on either Microsoft SQL Server or Oracle system. As you can see, you get a nice break down of the 4 more interesting tables: Core configuration, Alarm, Events and Stats data.

how-to-check-size-of-vcenter-server-database-0
If you wish to also calculate the estimated VCSA migration time, you simply just need to append the -migration_type parameter which accepts a value of option1 or option2. When performing the Windows vCenter Server to VCSA Migration, customers have the option of either only migrating the Configuration + Alarm data which I am referring to as Option 1 (default) or you can migrate all data which includes Configuration + Alarm + Event + Stats which I am referring to as Option 2. By simply changing the parameter in the script, you can get an idea of the time estimate as well as the amount of data (in GB) that would be migrated. Here is an example command:

Get-VCDBUsage -dbType mssql -connectionType local -dbServer sql.primp-industries.com -dbPort 1433 -dbInstance VCDB -dbUsername sa -dbPassword VMware1! -migration_type option1

Here is a screenshot of what the output would look like with the additional parameter.

how-to-check-size-of-vcenter-server-database-1
As you can see, you can easily run this script non-disruptively against your VCDB and assess the amount of data that could potentially be migrated as well as the amount of downtime required for a given migration scenario. This is also a great time to consider whether or not you need all of this data, especially when it comes to the Performance Stats. For Tasks/Events, this data is generally useful for auditing purposes and some of our customers must retain a certain amount for compliance purposes. However, for the Performance Stats, this information may not be as useful as some of you may think. As vCenter Server performs its daily, weekly and monthly rollup jobs, the statistics are continuously averaged out to the point where the granularity of the original data points are pretty lost. This means that you end up storing a ton of data that is really not all that useful. For fine grain historical stats, solutions like vRealize Operations Manager should be considered and vCenter Server should really be used for short term historical stats and quick ease of access for troubleshooting purposes. For more details on calculating the estimated amount of downtime for migration, please refer to VMware KB 2146420.

One last note, as you may have noticed from the screenshot or running the script that at the end of the output there is a question asking if you would like to compare your VCDB stats with others. If you do decide to share  the information(completely optional) which only includes the size for the each of the tables and number of rows that will be sent off to a public github repository https://github.com/migrate2vcsa. If we get enough submissions, we may do some fun things with the data and report back to the community. The data is anonymous and it might be interesting to see how your data set compares to others.

How to tell if an ESXi host is a VSAN Witness Virtual Appliance programmatically?

I had received this question awhile back but I was only able to get to it recently. If you are not familiar with the VSAN Witness Virtual Appliance and its purpose, Cormac Hogan did an excellent write-up on the topic which you can find it here.

how-to-tell-if-esxi-is-vsan-witness-vm-0
The reason this question came up was that if you were to simply iterate over all ESXi hosts within your vSphere Inventory from an Automation standpoint, you might find a mix of regular ESXi hosts and potentially this new VSAN Witness Virtual Appliance which is basically an ESXi host that runs in a VM (e.g. Nested ESXi). Although, it may look and feel like a regular ESXi host, it is not and the question was how might you go about distinguishing between the two? You can of course setup specific naming standards, folder structure or separate datacenter objects, but you still may accidentally retrieve a VSAN Witness host without even realizing it.

One quick solution is to check for a specific ESXi Advanced Setting called Misc.vsanWitnessVirtualAppliance which will return a value of 1 if it is the VSAN Witness Appliance. Here is a quick PowerCLI snippet which demonstrates how you can access this property:

$vmhost = Get-VMHost -Name 192.168.1.115
Get-AdvancedSetting -Entity $vmhost -Name Misc.vsanWitnessVirtualAppliance

how-to-tell-if-esxi-is-vsan-witness-vm-1
Although the method described above is one quick way to easily identify whether an ESXi host is a VSAN Witness Appliance, it is also limited in the information that it provides you. Another approach is to actually use the new VSAN 6.2 Management API and specifically the Stretched Clustering System APIs to retrieve the associated VSAN Witness host for a given VSAN Cluster. Not only will you get more information about the specific ESXi host providing the VSAN Witness functionality which will allow you to correlate back to your vSphere Inventory, but you will also get additional VSAN Witness configuration such as the preferred Fault Domain, Node UUID and the VSAN Cluster that it is associated with for example.

Here is a quick VSAN Management SDK for Python sample script that I had created called vsan-stretched-cluster-system-sample.py which implements the VSANVcGetWitnessHosts() API method. The script prints out a few of the WitnessHostInfo properties as shown in the screenshot below.

how-to-tell-if-esxi-is-vsan-witness-vm-2
One other option is if you simply just want to know if a given ESXI host is a VSAN Witness host or not, there is also the VSANVcIsWitnessHost() API that simply returns a boolean value. This might useful if you just have a list of ESXi hosts retrieved through the vSphere API and no knowledge of the underlying VSAN Clusters.

How to tell if your vCenter Server Appliance (VCSA) was migrated from a Windows vCenter Server?

In case you had not heard, last week VMware had officially released the VCSA Migration Tool which is included in the new vSphere 6.0 Update 2m release. Customers can now easily migrate from a Windows based vCenter Server over to the vCenter Server Appliance (VCSA) all while preserving their existing vCenter Server configurations and integrations. For more details, I highly recommend you check out all the links and resources here related to the VCSA Migration Tool.

One interesting question that came up over the weekend from a troubleshooting standpoint was how do you tell if your VCSA was migrated from a Windows vCenter Server? Besides remembering 😉 there is actually a pretty simple way to check by looking at the install parameters as I have previously written about here. To do so, you will need to SSH to your VCSA and enable the Bash Shell first. Once that has been done, go ahead and run the following command:

install-parameter upgrade.source.platform

If your VCSA was migrated from a Windows based vCenter Server using the new VCSA Migration Tool, you should see a value of windows. If you do not get any results, then it means the VCSA was not migrated and it was freshly deployed as an appliance.

In addition, you can also check whether or not you had migrated over the original vCenter Server's Stats, Events and Tasks (SET) data. To do so, run the following command:

install-parameter upgrade.user.options

You should get back a value of either yes or no for migrating over the SET data.

Lastly, if your VCSA was migrated from a Windows based vCenter Server, you can even tell if the migration was done so using the UI or CLI. To do so, run the following command:

install-parameter upgrade.silent

You should get back a value of either True for a CLI-based migration or False for a UI-based migration.

Here is a quick screenshot of running the three commands on a VCSA that was migrated.
how-to-check-if-vcsa-was-migrated-from-windows