mac mini

Apple NVMe driver for ESXi using new Community NVMe Driver for ESXi Fling

02/23/2021 by William Lam 10 Comments

VMware has been making steady progress on enabling both the Apple 2018 Mac Mini 8,1 and the Apple 2019 Mac Pro 7,1 for our customers over the past couple of years. These enablement efforts have had its challenges, including the lack of direct hardware access for our developers and supporting teams due to the global pandemic but also the lack of participation from Apple has certainly not made this easier.

Today, I am happy to share that we have made some progress on enabling ESXi to see and consume the local Apple NVMe storage device found in the recent Apple T2-based mac systems such as the 2018 Mac Mini and 2019 Mac Pro. There were a number of technical challenges the team had to overcome, especially since the Apple NVMe was not just a consumer grade device but it also did not follow the standard NVMe specification that you normally would see in most typical NVMe devices.

This meant there was a lot of poking and prodding to reverse engineer the behavior of the Apple NVMe to better understand how this device works, which often leads to sudden reboot or PSODs. With the Apple NVMe being a consumer device, it also meant there were a number of workarounds that the team had to come up with to enable ESXi to consume the device. The implementation is not perfect, for example we do not have native 4kn support for SSD devices within ESXi and we had to fake/emulate a non-SSD flag to work around some of the issues. From our limited testing, we have also not observed any significant impact to workloads when utilizing this driver and we also had had several internal VMware teams who have already been using this driver for a couple of months now without reporting any issues.

A huge thanks goes out to Wenchao and Yibo from the VMkernel I/O team who developed the initial prototype which has now been incorporated into the new Community NVMe Driver for ESXi Fling.

Caveats

Before folks rush out to grab and install the driver, it is important to be aware of a couple of constraints that we have not been able to work around yet.

ESXi versions newer then ESXi 6.7 Patch 03 (Build 16713306) is currently NOT supported and will cause ESXi to PSOD during boot up.
The onboard Thunderbolt 3 ports does NOT function when using the Community NVMe driver and can cause ESXi to PSOD if activated.

Note: For detailed ESXi version and build numbers, please refer to VMware KB 2143832

VMware Engineering has not been able to pin point why the ESXi PSOD is happening. For now, this is a constraint to be aware of which may impact anyone who requires the use of the Thunderbolt 3 ports for additional networking or storage connectivity.

With that out of the way, customers can either incorporate the Community NVMe Driver for ESXi offline bundle into a new ESXi Image Profile (using vSphere Image Builder UI/CLI) and then exporting image as an ISO and then installing that on either a Mac Mini or Mac Pro or you can manually install the offline bundle after ESXi has been installed over USB and upon reboot, the local Apple NVME will then be visible for VMFS formatting.

Here is a screenshot of ESXi 6.7 Patch 03 installed on my 2018 Mac Mini with the Apple NVMe formatted with VMFS and running macOS VM

Passthrough of Integrated GPU (iGPU) for Apple Mac Mini 2018

06/25/2020 by William Lam 28 Comments

After successfully enabling and persisting the passthrough of the iGPU for the latest Intel NUC 10 (Frost Canyon), I thought it was worth experimenting with the Apple Mac Mini 2018 to see if the same could be accomplished with its iGPU, which is an Intel UHD 630. The biggest benefit in addition to unlocking the iGPU for general use is support for Apple's Metal API which gives developers access to the underlying GPU when building and testing MacOS and iOS applications. This is also quite timely as the Apple Mac Mini 2018 was just added to the VMware HCL!

My initial attempt failed when using the latest ESXi 6.7 Update 3 release. After enabling passthrough of the iGPU and rebooting the ESXi host for the change to take affect, the system would get stuck during boot up when loading the dma_iommu_mapper module. After speaking with Engineering, the issue is probably not related to dma_iommu_mapper module but some other module shortly after but without serial console output or ability to see terminal screen, it would be very difficult to debug the issue.

About to give up, my last attempt was try ESXi 7.0 and to my surprise the ESXi host fully booted up after enabling passthrough of the iGPU. It is still not clear on what might be causing the problem for 6.7 but at least 7.0 works!

Note: To be able to successfully power on a MacOS VM running on ESXi 7.0, ensure you have applied the recent ESXi 7.0b patch. You will need to go to the VMware Patch Portal siteto download and apply the update.

Step 1- Enable passthrough of the iGPU using the vSphere UI and then reboot for changes to take affect.

Step 2 - Navigate to Configure->Hardware->Graphics->Host Graphics and change the default graphics type to "Shared Direct"

Virtually Speaking Podcast: MacOS Virtualization and MacStadium

05/11/2020 by William Lam Leave a Comment

Last week I had the pleasure to be on the Virtually Speaking Podcast (#1 Virtualization Podcast) to talk a little about the history and the use cases driving MacOS Virtualization in the Enterprise. In fact, this affects most if not every single organization that develops either an Apple MacOS and/or iOS application which includes VMware.

We also had a very special guest, Preston Lasebikan, a Systems Architect for MacStadium who gave us some insights into how they are supporting major Enterprise customers such as Dropbox, Capital One, Shopify, Box and many others using their Apple Mac Infrastructure which runs on VMware vSphere. If you never heard of MacStadium before, they are the largest service provider of Apple Mac Infrastructure as a Service in the world and there is a high probability your organization is already using them with you even knowing.

Click on the image below to listen 👇

Update on ESXi on Apple Mac Mini 2018 & Mac Pro 2019

02/28/2020 by William Lam 58 Comments

Although there has not been any news in some time regarding the support for ESXi on the latest Apple Mac Mini 2018 and the recently released Apple Mac Pro 2019, there has definitely been work happening behind the scenes at VMware. Today, I would like to share a pretty significant update as a result of some of these efforts.

MacOS Guest

One of the biggest issue which I had observed when using a T2-based Apple system with ESXi is that it would fail to boot a MacOS Guest and just keep rebooting the VM. I am very happy to announce that this issue has been resolved and ESXi can now properly recognize the Apple System Management Controller (SMC) device which is used as part of the MacOS Guest start up process. This now means a MacOS Guest will be able to properly boot on a T2-based Apple system.

Thunderbolt 3

Another impact of a T2-based Apple system with ESXi is that storage and networking devices connected to the Thunderbolt 3 ports are not visible. I am also happy to announce that this issue has been resolved and ESXi can now see PCIe devices that are attached to the Thunderbolt 3 ports.

An ESXi Advanced Setting change is required for Thunderbolt 3 to work correctly and the following command will need to be executed after installing ESXi:

esxcli system settings kernel set -s pciExperimentalFlags -v 16

Once the setting has been applied, a system reboot will be required and your PCIe devices will show up properly. In future, this additional configuration may not be required and can be detected based on the underlying hardware.

Both of the fixes mentioned above are included in the latest ESXi 6.7 Patch 02 (ESXi670-202004002) release which is available today! Hopefully this was the news that many of you have been waiting for 😀

UPDATE (02/23/21) - The Community NVMe Driver for ESXi Fling now enables access to the local Apple NVMe device.

UPDATE (08/27/20) - The Apple 2018 Mac Mini 8,1 is now officially supported with ESXi 6.7 Update 3 which requires the latest ESXi 6.7 Patch 03 which also incorporates automatically setting the ESXi Advanced Setting for Thunderbolt 3 access.

UPDATE (06/25/20) - The Apple 2018 Mac Mini 8,1 is now officially on the VMware HCL and is fully supported with ESXi 7.0b, which contains the fixes mentioned above. See note below on 06/23 for more information.

UPDATE (06/23/20) - ESXi 7.0b has just been released and contains fixes for both the MacOS guest boot issue support for Thunderbolt 3 devices which now enables support for the vSphere 7 release. One additional enhancement, customers no longer need to configure the ESXi Advanced Setting to enable Thunderbolt 3 support, this is now automatically configured based on detecting an Apple hardware system such as an Apple Mac Mini 2018 or Apple Mac Pro 2019. This is a patch release and you will need to go to the VMware Patch Portal site to download and apply the update.

Now, before you rush out to start deploying MacOS Guests on either the Mac Mini or Mac Pro, I do have to mention that neither the Mac Mini 2018 or the Mac Pro 2019 will be officially supported by VMware. Due to the current situation that we are all in with COVID-19, personnel access to VMware facilities like many other organizations has been severely restricted and/or prohibited. In fact, much of the early validation was done by yours truly using a Mac Mini 2018 which I had access to (Thanks Michael Roy) as Engineering did not have access to hardware during the shelter in place orders. This also means that certifications of these platforms is still on-going and until these systems are officially listed on VMware's HCL, they will not be officially supported by VMware.

Disclaimer: VMware currently does not officially support the Apple 2019 Mac Pro7,1

Apple Mac Mini on VMware HCL!

08/01/2019 by William Lam 13 Comments

For the past 6 years, the Apple Mac Mini has been one of the most popular hardware platforms for Virtualizing MacOS running on VMware vSphere enabling our customers to develop and build iOS and MacOS applications. With that said, VMware has historically only supported two Apple hardware platforms: Xserve (now EOL'd) and the Mac Pro (6,1) which is officially listed on VMware's Hardware Compatibility list and this has been officially supported by VMware since 2012 when we first introduced support for MacOS Virtualization with the vSphere 5.0 release.

As many of you know, I have been a huge advocate of this platform for a number of years now and I have been working with various Engineers over the years to ensure that we have the exact same user experience when working with ESXi on the Mac Mini as you do with the Mac Pro. I still recall in the early days where it took several "hacks" to get ESXi to successfully boot and install.

Today, ESXi installs on the Mac Mini just like any other x86 platform. It runs amazing well for our customers, especially for a consumer device, who have deployed them in their datacenters ranging from a couple hundred to several thousands for some of our larger Enterprise customers, one such example is MacStadium, the largest Apple Infrastructure-as-a-service provider which many of the Fortune 100/500 companies are leveraging to provide them with a platform to build and develop for the Apple eco-system.

UPDATE (08/27/20) - Apple 2018 Mac Mini 8,1 has been added to VMware HCL which supports both ESXi 6.7 Update 3 (Patch 03) & ESXi 7.0b

Aquantia 10GbE ESXi Driver for Apple 2018 Mac Mini

04/10/2019 by William Lam 23 Comments

I know many of you have been anxiously waiting for ESXi to be fully functional on the latest Apple Mac Mini (2018), unfortunately I do not have any news to share with you on that front. Without help from Apple, we are still challenged with Apple's new T2 chip, which prevents us from accessing the underlying NVMe device.

UPDATE (04/27/20) - Marvell (formally Aquantia) has just released an official Native ESXi Driver for their AQtion based network adaptors which you can find here and for the complete list of supported devices using this driver, please refer to the VMware HCL here.

Having said that, I do have some exciting news regarding the built-in 10Gigabit Ethernet adaptor on the 2018 Mac Mini. The 10GbE adaptor uses an Aquantia chipset, this is also the same chipset used in Apple's high end iMac Pro which was released earlier in the year. Over the past few weeks, I have been working with the Aquantia team and they have successfully ported over their open source Atlantic driver to a VMKlinux driver for ESXi, which they have published here. Although the local NVMe device can not be used to run any VMs, with the network fully enabled, customers could take advantage of this model and connect to IP-based storage to start leveraging the new Mac Mini platform.

The easiest way to incorporate the driver into the latest ESXi release is to use Image Builder within the vSphere H5 Client UI, below are the step-by-step instructions.