VMware has been making steady progress on enabling both the Apple 2018 Mac Mini 8,1 and the Apple 2019 Mac Pro 7,1 for our customers over the past couple of years. These enablement efforts have had its challenges, including the lack of direct hardware access for our developers and supporting teams due to the global pandemic but also the lack of participation from Apple has certainly not made this easier.
Today, I am happy to share that we have made some progress on enabling ESXi to see and consume the local Apple NVMe storage device found in the recent Apple T2-based mac systems such as the 2018 Mac Mini and 2019 Mac Pro. There were a number of technical challenges the team had to overcome, especially since the Apple NVMe was not just a consumer grade device but it also did not follow the standard NVMe specification that you normally would see in most typical NVMe devices.
This meant there was a lot of poking and prodding to reverse engineer the behavior of the Apple NVMe to better understand how this device works, which often leads to sudden reboot or PSODs. With the Apple NVMe being a consumer device, it also meant there were a number of workarounds that the team had to come up with to enable ESXi to consume the device. The implementation is not perfect, for example we do not have native 4kn support for SSD devices within ESXi and we had to fake/emulate a non-SSD flag to work around some of the issues. From our limited testing, we have also not observed any significant impact to workloads when utilizing this driver and we also had had several internal VMware teams who have already been using this driver for a couple of months now without reporting any issues.
A huge thanks goes out to Wenchao and Yibo from the VMkernel I/O team who developed the initial prototype which has now been incorporated into the new Community NVMe Driver for ESXi Fling.
Caveats
Before folks rush out to grab and install the driver, it is important to be aware of a couple of constraints that we have not been able to work around yet.
-
ESXi versions newer then ESXi 6.7 Patch 03 (Build 16713306) is currently NOT supported and will cause ESXi to PSOD during boot up.
-
The onboard Thunderbolt 3 ports does NOT function when using the Community NVMe driver and can cause ESXi to PSOD if activated.
Note: For detailed ESXi version and build numbers, please refer to VMware KB 2143832
VMware Engineering has not been able to pin point why the ESXi PSOD is happening. For now, this is a constraint to be aware of which may impact anyone who requires the use of the Thunderbolt 3 ports for additional networking or storage connectivity.
With that out of the way, customers can either incorporate the Community NVMe Driver for ESXi offline bundle into a new ESXi Image Profile (using vSphere Image Builder UI/CLI) and then exporting image as an ISO and then installing that on either a Mac Mini or Mac Pro or you can manually install the offline bundle after ESXi has been installed over USB and upon reboot, the local Apple NVME will then be visible for VMFS formatting.
Here is a screenshot of ESXi 6.7 Patch 03 installed on my 2018 Mac Mini with the Apple NVMe formatted with VMFS and running macOS VM
Great job. Looking forward to see the caveats fixed.
I’m really hoping someday this gets fixed so I can use current hardware without external thunderbolt. I’m currently using 2012 Mac Mini’s.
James,
This Fling only applies to T2-based Macs and with this driver, you can take advantage of the local NVMe device without using Thunderbolt. For non T2-based Macs, you can already use the onboard storage devices without any caveats, which many of our customers have been doing for number of years now. Just wanted to make sure that was clear
So I’ve tried this. I could see the local Apple SSD but when I attempted to clear the partition table ESXi hard rebooted.
What version/build of ESXi are you using? If reboot is happening suddenly during the install, then it sounds like you might be using something newer than 6.7p03
Build is 16713306. I installed ESXi 6.7p03 to a USB drive, installed the fling, rebooted then went to wipe the local NVMe Apple SSD. Do I need to install ESXi to the Apple NVMe instead of run off of a USB?
I just tried this again:
1. Used vSphere ESXi Image Builder to build an ISO based on ESXi 6.7 Image Profile-ESXi-6.7.0-20200804001-standard + the NVMe Community fling
2. Flashed the above ISO to a USB drive
3. Attempted to install to internal SSD using USB drive
During the install ESXi doesn’t PSOD but instead just hard reboots. Have you guys tested with 1TB drives? Maybe something is different with them? I’m happy to help collaborate in any way.
Screenshot of install screen: https://f000.backblazeb2.com/file/dropshares/CleanShot-2021-02-24-at-00.29.25.png
Can you grab boot rom / smc version? https://www.virtuallyghetto.com/2018/10/retrieving-apple-hardware-details-model-serial-board-efi-boot-rom-smc-version-from-esxi.html
Boot it up, don’t install and enable SSH via Alt+F1 to get IP. User is root and blank pass
Ryan – Can you try ESXi 6.7 P02 (Build 16075168)? I have some suspicions but I’d also want to capture boot room before and after applying latest update and trying this again
Model Identifier: Macmini8,1
Board ID: Mac-7BA5B2DFE22DDD8C
Boot ROM Version: 1554.3.0 (1554.80.3.0.0 (iBridge: 18.16.14346.0.0,0))
I am guessing SMC number is now iBridge?
I also have this info if you need it:
Controller Information:
Model Name: Apple T2 Security Chip
Firmware Version: 18P4346
Apple T2 Bus:
Host Controller Driver: AppleUSBVHCIBCE
Headset:
Product ID: 0x8103
Vendor ID: 0x05ac (Apple Inc.)
Version: 2.05
Apple T2 Controller:
Product ID: 0x8233
Vendor ID: 0x05ac (Apple Inc.)
Version: 2.01
Creating a new installer and testing with ESXi 6.7 P02 (Build 16075168)
So I tried ESXi 6.7 P02 (Build 16075168) and it’s working as expected. So fast!
Btw here is some more reading on Mac firmware. It doesn’t explain what happened to the SMC versions that no longer show in System Information in macOS but it’s interesting none the less. https://eclecticlight.co/2020/11/16/which-firmware-should-your-mac-be-using-version-5/
I’m going to guess that this has to do with a recent update to the T2 security chip by Apple. Have you or anyone else on the team that worked on this fling installed macOS Big Sur directly on the Mac mini and then installed all updates? I’m betting that will flash the T2 security chip in a way that you are not seeing. Oh well, at least I have my 1TB Thunderbolt 3 drive arriving tomorrow that I can use for ESXi storage. 😉
Will do this tomorrow and see if there are updates. Will grab before / after boot rom and smc version. https://www.virtuallyghetto.com/2018/10/retrieving-apple-hardware-details-model-serial-board-efi-boot-rom-smc-version-from-esxi.html
If folks can share if they’re hitting dame, useful for Engr
Hello,
I ran into a similar issue as the previous users. I executed the following:
Created an ESXi image using the following:
– ESXi 6.7 P03
– nvme-community 1.0.1.0-1vmw.670.0.0.8169922
– atlantic 1.0.2.0-1OEM.670.0.0.8169922
Was able to successfully install ESXi. However, when I tried to add the internal drive as datastore, the host rebooted and the ESXi configuration was wiped, meaning it lost its IP, DNS, etc. It was similar to removing T3 from the host when ESXi is running.
Additionally I tried installing the ESXi on the internal drive and the same thing happened. Meaning the installation progress Bar got to 5% and then it rebooted.
Both hosts that I was testing this on had the following BIOS Version
Macmini8,1
BIOS version 1037.100.362.0.0 (iBridge: 17.16.14281.0.0,0)
I will test ESXi 6.7 P02 (Build 16075168) tomorrow in hopes of making it work.
Lastly, I tried the installation with T3 VMNIC for data. It was successful, but when I tried mounting the drive as a datastore, it went through a reboot again. However this time the ESXi configuration was retained, unlike the time when the 10gb on board nic was used.
Hope this helps.
Thank you,
-Alex
Hi William,
Any chance this could be a way to get other consumer nvme devices working 6.7+ without having to swap the driver with one from a previous release?
https://www.virtuallyghetto.com/2019/05/quick-tip-crucial-nvme-ssd-not-recognized-by-esxi-6-7.html
I note the driver will only enable on mac hardware, is there a way to override this setting?
It’s possible, but would have to know exact device, DID/VID and ultimately there needs to be fair amount of folks asking for the same and team can consider
Could the DID/VID be configurable in a settings file? Then it would be very flexible.
It’s not just putting those IDs and things magically work 🙂 There’s actual code behind it, the ID just helps us understand vendor/device
Wait,what isn’t this all just magic 🙂
Hmm, how about a wrapper driver for nvme devices that can spoof the full 1.3 specification to allow them to work with later exsi nvme drivers that need to see those missing attributes.
Anybody successfully install this combination on a 2019 Mac Pro? I’m trying the:
– ESXi 6.7 P03
– nvme-community 1.0.1.0-1vmw.670.0.0.8169922
– atlantic 1.0.2.0-1OEM.670.0.0.8169922
combination and getting a hard crash during install… no thunderbolt devices plugged in.
Thanks!
Also got error on ESXi 6.7U3, here are the goodies:
Build 16713306:
https://www.audiocircle.com/index.php?action=gallery;area=browse;image=221270;size=huge
Build 14320388:
https://www.audiocircle.com/index.php?action=gallery;area=browse;image=221269;size=huge
Mac Details:
https://www.audiocircle.com/index.php?action=gallery;area=browse;image=221271;size=huge
Can you please try ESXi 6.7 P02 (Build 16075168) and see if the results are the same. I’ve seen some inconsistencies across several customer setup, some having no issues with p03 but others having and using p02 seems to be more stable.