apple

Using vSphere Guest Operations API on macOS Guests?

07/05/2017 by William Lam 2 Comments

I have written a number of articles exploring the usage and some of the cool tricks that the vSphere Guest Operations (GuestOps) feature provides which you can be found here, here, here and here. I have been a huge fan and supporter of GuestOps since the early days where it was formally known as the VIX API. Having used GuestOps across many different GuestOS types including Nested ESXi, I have to admit, I had never tried it against an Apple macOS guests. I recently had a customer reach out who was looking to use the GuestOps API via PowerCLI (Invoke-VMScript) to automate updates against his guestOS templates that span across Windows, Linux and macOS (from 10.7 to latest). The customer was able to get all guestOSes working except for macOS.

Since I had never tried this before, I spun up my Apple Mac Mini which happen to have a macOS 10.11 (El Capitan) guests running. I tried using the vSphere API GuestOps directly to see if this was a PowerCLI and/or API issue. I too ran into issues and after enabling VMware Tools debugging on the guests (which you can find more details below), I found that it hit the following error:

[Jun 28 06:35:42.805] [ debug] [vix] >VixToolsImpersonateUser

[Jun 28 06:35:42.925] [ warning] [vmsvc] Failed to set gid for user root

Reaching out to Engineering regarding the problem, I came to learn that this particular issue was due to a syscall change made by Apple starting with macOS 10.10.3 and newer. Although the change was a positive thing from a security standpoint, it did break the GuestOps functionality. The good news was that this was already resolved with VMware Tools 10.1 or later. When I had initially provisioned the macOS guests, the latest VMware Tools at the time was 9.10.5. After I applied the latest version which is currently 10.1.7, the issue went away and I was able to successfully use the GuestOps API on my macOS guests.

Below are examples of running the system_profiler SPSoftwareDataType command using both the Invoke-VMScript cmdlet as well as the vSphere API and PowerCLI to consume the GuestOps APIs. Both approaches delivers the exact same outcome, the one benefit of using Invoke-VMScript is that if you want to easily return output from a given command, the cmdlet already does the heavy lifting. If you notice in the native vSphere API case, you do not get output but rather just the PID ID. If you want to return the output, you need to first save it into a file and then download the file to your client system, which may not be ideal for interactive usage but it all depends on your use case.

Using vSphere Auto Deploy to Netboot ESXi onto Apple Mac Hardware

01/17/2017 by William Lam 4 Comments

Last week I published an article that demonstrated for the first time on how to netboot an ESXi installation onto Apple Mac Hardware. As you can imagine, this was very exciting news for our VMware/Apple customers, who historically have not had this capability before. Customers can now automate and install ESXi over the network onto their Apple Mac Hardware just like you would for other non-Apple hardware.

With the ability to boot ESXi over the network for Apple Mac Hardware, it is now also possible for customers to take advantage of the vSphere Auto Deploy feature. Auto Deploy allows customers to easily and quickly provision ESXi hosts at scale and integrates directly with vCenter Server to automatically join and apply specific defined host configuration policies. This is a great time to check out Auto Deploy, especially with all the new enhancements that were introduced in vSphere 6.5 like custom script bundles for example.

Below are the instructions on how to setup Auto Deploy to work with Apple Mac Hardware.

How to Netboot install ESXi onto Apple Mac Hardware?

01/13/2017 by William Lam 11 Comments

The ability to perform an ESXi Scripted Installation over the network has been a basic capability for non-Apple hardware customers since the initial release of classic ESX. However, for customers who run ESXi on Apple Mac Hardware (first introduced in vSphere 5.0), being able to remotely boot and install ESXi over the network has not been possible and customers could only dream of this capability which many of us have probably taken for granted.

Unlike traditional scripted network installations which commonly uses Preboot eXecution Environment (PXE), Apple Mac Hardware actually uses its own developed Boot Service Discover Protocol (BSDP) which ESXi and other OSses do not support. In addition, there are very few DHCP servers that even support BSDP (at least this may have been true 4 years ago when I had initially inquired about this topic). It was expected that if you were going to Netboot (equivalent of PXE/Kickstart in the Apple world) a server that you would be running a Mac OS X system. Even if you had set this up, a Netboot installation was wildly different from a traditional PXE installation and it would be pretty difficult to near impossible to get it working with an ESXi image. With no real viable solution over the years, it was believed that a Netboot installation of ESXi onto Mac Hardware just may not be possible.

tl;dr - If you are interested in the background to the eventual solution, continue reading. If not and you just want the goods, jump down a bit further. Though, I do think it is pretty interesting and worth getting the full context 🙂

ESXi 6.5 support for Apple Mac Pro 6,1

01/09/2017 by William Lam 56 Comments

I know several of you have reached out asking about the support for ESXi 6.5 on the Apple Mac Pro 6,1 but as of right now, the Mac Pro 6,1 is currently not supported with ESXi 6.5. I know this is not ideal especially for customers who wish to take advantage of the latest vSphere release. The good news is that VMware is in the process of testing the Apple Mac Pro 6,1 for ESXi 6.5, however there is not an ETA on when this will be completed by.

Some of you might be wondering why this did not happen earlier? The primary reason is that hardware certification for ESXi is actually performed by the hardware vendors. Once a vendor completes the certification for a particular hardware platform or component, they submit the results to VMware and the VMware HCL is updated. If there is a piece of hardware that is not on the VMware HCL today, it is definitely worth reaching out to your hardware vendor to inquire about its status.

In Apple's case, it unfortunate as they do not participate in VMware's Hardware Certification program for ESXi which makes certification challenging. VMware intends to continue to support customers who require the use of Mac OS X Virtualization and will work towards getting the Mac Pro's certified for latest version of vSphere as mentioned earlier. Historically, testing and certifying ESXi for Apple hardware does take an additional amount of time and in some cases, code changes may even be required due to unexpected hardware changes from Apple.

I hope this gives customers some additional insights into how Apple hardware is certified for ESXi. If you would like to see this improved in the future, you may want to reach out to Apple and provide them with your feedback.

Now ... before you close this blog post thinking it is going to take awhile before there is going to be an update regarding ESXi 6.5 and Mac Pro 6,1, please continue reading further 🙂

UPDATE (07/28/2017) - ESXi 6.5 Update 1 just GA'ed yesterday and is fully supported with all current Apple Mac Pro 6,1 (as you can see on the HCL here) and the workaround mentioned below is no longer required. This means you can install ESXi without any modification to the image.

UPDATE (03/25/2017) - VMware has just published the following VMware KB 2149537 which outlines the officially recommended workaround to install ESXi 6.5 onto the Apple Mac Pro 6,1. The VMware HCL has also been updated to include the Apple Mac Pro 6,1 4-Core, 6-Core, 8-Core & 12-Core systems. In a future release of ESXi, the workaround will not be required and ESXi will just install out of the box. This temporarily workaround is to enable customers who wish to run the current version of ESXi 6.5 which includes GA release, 6.5a and 6.5p01.

VMXNET3 driver now included in Mac OS X 10.11 (El Capitan)+

10/01/2016 by William Lam 14 Comments

Yesterday I received a pretty interesting comment from one of my Twitter followers @NTmatter who wrote:

@lamw Just noticed that OSX has a VMXNET3 driver. Have to edit the vmx file to actually use it, but it's there! AppleVmxnet3Ethernet.kext

— Thomas Johnson (@NTmatter) September 30, 2016

This is a pretty neat find because currently today, the only network adapter that is functional with an Apple Mac OS X guest running on either VMware vSphere or Fusion is the e1000{e} driver. This update was definitely news to me and after sharing it internally to see if I could find some more details, it turns out this news also came as surprise to the folks internally. Darius, one of the Engineers who I frequently reach out to on Apple related topics did some digging and found out that Apple started to bundle this VMXNET3 driver starting with Mac OS X 10.11 (El Capitan) release. You can find the driver located in /System/Library/Extensions/IONetworkingFamily.kext/Contents/PlugIns/AppleVmxnet3Ethernet.kext

Disclaimer: Given that this VMXNET3 Mac OS X driver was not developed by VMware nor has it been tested by VMware, it currently would not be officially supported by VMware.

If you wish to try out the VMXNET3 driver, you will need to install Mac OS X 10.11 or newer on a VM running on vSphere or Fusion. By default, the only available network adapter type is e1000{e}. To add a VMXNET3 network adapter, you can either manually tweak the .VMX file or you can easily add it by using either the vSphere Web/C# Client or ESXi Embedded Host Client. Below are the instructions on configuring the VMXNET3 network adapter for your Mac OS X guests.

Step 1 - Remove the existing network adapter and then temporarily change the GuestOS type to "Other" (no need to save setting, just update it in VM reconfigure wizard) so that you will be allowed to add a VMXNET3 network adapter. Once you have added it to the VM reconfigure wizard, go ahead and toggle back the GuestOS type to Mac OS X 10.10 and then save the settings as shown in the screenshots below.

Step 2 - Open a terminal inside of the Mac OS X guest and run the following command to load the VMXNET3 driver:

sudo kextload /System/Library/Extensions/IONetworkingFamily.kext/Contents/PlugIns/AppleVmxnet3Ethernet.kext

Step 3 - You can verify that the VMXNET3 driver was successfully loaded by running the following command:

kextstat | grep -i vmxnet3

Once the driver has been loaded, you should now have networking connectivity to your Mac OS X VM using the VMXNET3 network adapter. Below is a screenshot of the system info showing the VMXNET3 network adapter.

In addition to having an optimized networking when using the VMXNET3 driver, the other benefit is being able to get a link speed of 10GbE which is something customers have been inquiring about when virtualizing Mac OS X guests. Below is a screenshot of the media link shown in this Mac OS X 10.11 guests.

Although this a great development for Apple customers who uses VMware vSphere and Fusion, it also does raise an interesting question on whether Apple would be officially supporting this VMXNET3 driver going forward? If I do receive any more details on this, I will update the article. Until then, you can play with this new capability if you are running Mac OS X 10.11 or greater on VMware. Big thanks to Thomas for this great find and sharing it with the VMware Community!

ESXi Thunderbolt Driver to Fibre Channel Storage from ATTO

09/12/2016 by William Lam 6 Comments

One of the things I always enjoy doing at VMworld, when I am not running around and I have a few minutes to myself, is to check out the VMware Solutions Exchange. This is where you can learn and interact with hundreds of our VMware Certified Partners showcasing their new solutions and innovations that they have built on top of VMware's products.

UPDATE (08/22/17) - ATTO's ESXi Thunderbolt Driver is now officially on the VMware HCL, please see this blog post here for more details.

While walking through the show floor, I had stopped by the ATTO Technology booth who has been a long time partner of VMware in the storage and networking connectivity space. What caught my eye was that they had just released a Beta of an ESXi Thunderbolt Driver in the form of an ESXi VIB that would allow customers to connect their Apple Mac Pro 6,1 using the Thunderbolt 2 interface to an external Fibre Channel storage array. I believe ATTO might be the first vendor ever to produce a Thunderbolt Driver for ESXi. This is really exciting news if you ask me, especially as more and more of our customers are looking to virtualize Mac OS X guests in their Datacenters using vSphere.

Historically, the only option to connect a Mac Pro 6,1 to an external Fibre Channel array was to use something like a Sonnet Chassis. Now, you can potentially connect up to 6 of the built-in Thunderbolt 2 interfaces on the Mac Pro's to your external storage array using this new solution from ATTO. Before I go into some of the details, ATTO did want me to mention that this solution is currently not officially supported by VMware nor is it on VMware's HCL. ATTO will be providing full support on their software as well as VMware's software stack during the duration of the beta program. In terms of official certification on VMware's HCL, I suspect that it will most likely depend on customer demand which would influence whether ATTO applies for an official certification, which again, would be the first of its kind for Thunderbolt.

The way in which this solution works is that you install the ATTO Thunderbolt Driver on your ESXi host and this will allow it to communicate with an ATTO ThunderLink device which provides the Thunderbolt 2 to Fibre Channel connectivity. You have the option of using either the FC2082 which provides 20Gb/s Thunderbolt 2 (2-port) to 8Gb/s FC (2-Port) Device or the FC2182 which provides 20Gb/s Thunderbolt 2 (2-port) to 16Gb/s FC (2-Port) Device. Below is a diagram from the ATTO digital solution brief on Thunderbolt Driver for ESXi which outlines the configuration.

If you are interested in taking part in ATTO's ESXi Thunderbolt Driver Beta program or would like to learn more about the solution, you can reach out directly to Carllene Mowry (*protected email*) who is running the program. For more information be sure to check out the ATTO digital brief on Thunderbolt Driver for ESXi.

Lastly, I was also fortunate to have a quick chat with Carllene and team to get a few additional exclusive tidbits on some of the things the ATTO team is working on next. The first of which is support for the Thunderbolt 3 (aka USB-C) interface to Fibre Channel which will be quite nice for newer platforms that include that interface, including home lab setups such as the Intel NUC. Speaking of Intel NUC, this is just one of the many other platforms that include either Thunderbolt 2 or 3 interfaces. Although the solution today is specifically supporting the Mac Pro, I know ATTO folks are interested to hear from customers on other systems with Thunderbolt interface and providing similiar capabilities.

The other really exciting development that is currently being investigated is support for Thunderbolt 2 or 3 to 10GbE connectivity on ESXi. As you can imagine, this is really going to open up some really cool new use cases, especially around things like VSAN which can easily benefit from this. It is still in early development but from my understanding, ATTO is already seeing a lot of interest in this area as well as how this might work with VSAN. I am hoping I will be able to share more details as this further develops. If any of these updates sounds interesting, do leave a comment to let the ATTO folks know and I will make sure they monitor the thread.