ESXi

Removable M.2 NVMe SSD PCIe enclosure by Icy Dock

07/27/2020 by William Lam 5 Comments

My homelab is a constant experiment and hardware components are moved around for various testing, especially when it comes to networking and storage. When needing to move around an M.2 NVMe SSD, complexity of taking apart a system will vary on the platform but generally it is inconvenience. When I came to learn that Icy Dock, a manufacturer of storage enclosures, will be releasing a removable M.2 NVMe SSD tray that is connected to PCIe expansion slot, I knew I had to get my hands on it.

The good folks over at Icy Dock were kind enough to send me an early evaluational unit of the upcoming MB840M2P-B which is now available for $69. ~~will be released in August and should retail for around $80 USD (final prices are still TBD)~~. The use case above may not apply to most folks and is probably unique to my specific hardware usage but I think this is still a very interesting solution that is still useful to be aware of if you are your own homelab whitebox and have a spare PCIe slot. Icy Dock also produces many other types of storage enclosures that you might find interesting based your own needs.

For my setup, I installed the MB840M2P-B into my Intel NUC 9 Pro, which is definitely not easy to take apart. This is especially true for the two M.2 which is attached to the NUC Element but even more painful to get to the 3rd M.2 which is located under the baseboard. For my specific use case, this was well worth using up one of the PCIe slots on the NUC 9 Pro! This enclosure can also be added to the new 2019 Mac Pro which is another platform that Icy Dock sees benefiting from this solution.

Configuring dnsmasq as PXE Server for ESXi

07/09/2020 by William Lam Leave a Comment

One really cool thing that I came to learn while setting up the infrastructure to network boot the latest Raspberry Pi 4 was the use of dnsmasq, which I have used in the past but I did not realize it could do so much more. In addition to providing DNS services, it can also be configured to run TFTP and provide DHCP capabilities which can then be used to support PXE installations.

Another neat feature of dnsmasq is ability to proxy to an existing DHCP server which is extremely useful for anyone with an existing DHCP infrastructure. Given the simplicity of dnsmasq and having already set this up for the rPI, I figure it would also be useful to take folks through in setting up dnsmasq to also support ESXi installations over PXE, since this still comes up from new folks just getting started with ESXi kickstart automation.

For more details about PXE installation of ESXi, I highly recommend this whitepaper and although it states 6.0, the concepts and configurations are still applicable to the latest ESXi 7.0 release.

How to patch Intel NUC 10 with latest ESXi 7.0 update?

06/28/2020 by William Lam 23 Comments

vSphere 7.0b was just released last week and one of the important fixes was to resolve an issue where Nested ESXi VMs were crashing upon powering on an inner-guest VM. This looks to have also affect newer generations of CPUs including Intel's 10th Gen Comet Lake which is also found in the latest 10th Gen Intel NUCs (Frost Canyon).

A number of folks quickly found that if you simply applied the ESXi 7.0b patch, an unexpected behavior occurred on the 10th Gen Intel NUCs and the onboard networking was lost upon a reboot. This occurs as the original ne1000 driver which had been replaced with a newer version found within ESXi 7.0b no longer recognizes the onboard Intel NIC. The solution is quite simple, create a new Image Profile that contains the Intel NUC NIC Driver.

Several of you have asked for instructions and although this is a pretty common vSphere workflow, I have documented the two supported options using the vSphere Image Builder utility but there are definitely other methods which will have the same results. If you have access to a vCenter Server 6.7 or newer, I recommend using the Image Builder UI. If vCenter Server access is not available, then you can use Image Builder with PowerCLI, however you will need to have access to a Windows machine as the Image Builder cmdlet is not supported with PowerCLI Core.

Note: There is currently a known bug with the Image Builder UI when using vSphere 7 which will prevent you from authoring a new Image Profile. A workaround would be to deploy a VCSA 6.7 which does not have this issue when looking to use the Image Builder UI.

Passthrough of Integrated GPU (iGPU) for Apple Mac Mini 2018

06/25/2020 by William Lam 28 Comments

After successfully enabling and persisting the passthrough of the iGPU for the latest Intel NUC 10 (Frost Canyon), I thought it was worth experimenting with the Apple Mac Mini 2018 to see if the same could be accomplished with its iGPU, which is an Intel UHD 630. The biggest benefit in addition to unlocking the iGPU for general use is support for Apple's Metal API which gives developers access to the underlying GPU when building and testing MacOS and iOS applications. This is also quite timely as the Apple Mac Mini 2018 was just added to the VMware HCL!

My initial attempt failed when using the latest ESXi 6.7 Update 3 release. After enabling passthrough of the iGPU and rebooting the ESXi host for the change to take affect, the system would get stuck during boot up when loading the dma_iommu_mapper module. After speaking with Engineering, the issue is probably not related to dma_iommu_mapper module but some other module shortly after but without serial console output or ability to see terminal screen, it would be very difficult to debug the issue.

About to give up, my last attempt was try ESXi 7.0 and to my surprise the ESXi host fully booted up after enabling passthrough of the iGPU. It is still not clear on what might be causing the problem for 6.7 but at least 7.0 works!

Note: To be able to successfully power on a MacOS VM running on ESXi 7.0, ensure you have applied the recent ESXi 7.0b patch. You will need to go to the VMware Patch Portal siteto download and apply the update.

Step 1- Enable passthrough of the iGPU using the vSphere UI and then reboot for changes to take affect.

Step 2 - Navigate to Configure->Hardware->Graphics->Host Graphics and change the default graphics type to "Shared Direct"

MacOS 11 (Big Sur) Beta 1 on ESXi

06/24/2020 by William Lam 14 Comments

The first Beta of Apple MacOS 11 (Big Sur) was just released a couple of days ago and I know folks are excited to start kicking the tires. Some folks have noticed when to installing Big Sur running on VMware Fusion, the following error is observed:

BIErrorDomain error 3

From the suggested workarounds, it looks like the MacOS installer was somehow unable to detect that the underlying hardware was Apple which causes this generic error to be thrown. Interestingly, this was the same error I came across when attempting to install Big Sur on ESXi 7.0. Instead of having to lookup your physical Apple hardware IDs and specify several VM Advanced Settings, you can simply add the following setting which will accomplish the same behavior:

smbios.reflectHost = "TRUE"

After the setting has been applied, the error should go away and you should be able to upgrade from an existing MacOS deployment to Big Sur. This issue has already been reported internally at VMware and I have also shared with the teams the quick workaround.

Here is Big Sur on ESXi 7.0 running on an Apple Mac Mini 2018 (requires ESXi 7.0b patch VMware-ESXi-7.0b-16324942)

Here is Big Sur on ESXi 6.7 Update 3 running on an Apple Mac Mini 2018 (requires ESXi 6.7 Patch 02 ESXi670-202004002)

Passthrough of Integrated GPU (iGPU) for standard Intel NUC

06/18/2020 by William Lam 20 Comments

Earlier this week I found out that it was possible to passthrough the Integrated GPU (iGPU) for standard Intel NUC which was motivated by a question I had saw on the VMware Subreddit. I have written about iGPU passthrough for Intel NUCs before but only for the higher end models which were the Hades Canyon NUC at the time.

Neat! Just found out you can actually passthrough the iGPU of standard Intel NUC. The trick looks to be enabling passthrough using ESXi Embedded Host Client UI & then you can assign it using vSphere UI#Homelab pic.twitter.com/NwuxbXwUMj

— William Lam (@lamw) June 15, 2020

To be honest, I never thought about trying this out with a standard NUC as I figured the iGPU may not be powerful enough or warrant any interests. After sharing the news on Twitter, I came to learn from the community that not only is this desirable for various use cases but some folks have also been doing this for some time now and have shared some the benefits it brings for certain types of workloads.

Can’t take credit. It was one of our collegaues that pointet me to it. Hw transcoding went up a factor of almost x 20. So for specefic workloads the nuc is suddently a lot more capable than before.

— Robert Jensen (@rhjensen) June 15, 2020

I’ve been doing this forever, when I need to crack passwords but don’t need the full 7 gpu rig - all Supermicro and 1080ti GPUs these dayshttps://t.co/GJGRV5eu8f

— Rob VandenBrink (@rvandenbrink) June 15, 2020

seems like this would be great for ESXi + Plex hardware transcoding

— Will Beers (@willbeers) June 15, 2020

Below are the instructions I used to enable iGPU passthrough on an Intel NUC 10 (Frost Canyon) with vSphere 7.0. These instructions should also be applicable for other NUC models and earlier versions of vSphere including details around passthrough configuration persistency which I know some folks have ran into which I was able to figure out as part of this experiment.