Home Lab

Other Intel and AMD small form factor (SFF) systems for vSphere Homelabs

08/06/2020 by William Lam 27 Comments

When it comes to selecting a platform for a vSphere Homelab, there are many options which include building your own "whitebox" system. For the large majority of folks, the preference is to purchase a ready to use kit such as an Intel NUC or Supermicro which both extremely popular. These systems not only work well but their form factor is also ideal for home offices where space is always at a premium.

With that said, there are many other small form factor (SFF) platforms that exists out in the market and not just Intel-based systems, but also AMD SFF kits which are being introduced and have been getting many inquiries about. As someone who keeps a close eye on this market for new and interesting platforms, I have been re(sharing) some of these new updates on Twitter.

Although Twitter is great way to share and discuss news, it is not the best place to consolidate this type of information that can easily be searched. This was one the motivation for putting together this post for both informational awareness but also something that can be updated over time. This was certainly a challenge when asked about other SFF options, especially in the AMD space where I was not able to easily point folks to. Below is a collection of SFF for both Intel and AMD that I have come across, some of which are currently being used for vSphere Homelabs and others having the potential given their specification. In addition, I suspect many of the kits below which report 32GB of memory as their max should be able to go up 64GB as I have shown in the past with NUC platform.

If folks have other SFF kits they would like to share or confirm that works with latest versions of vSphere, feel free to leave a comment which can help others in the VMware Community.

Enhancements to the community ne1000 VIB for Intel NUC 10

08/03/2020 by William Lam 11 Comments

As many of you know, the onboard Intel NIC (8086:0d4f) found in the 10th generation of the Intel NUC (Frost Canyon) is not automatically recognized by ESXi and requires an updated ne1000 VIB which was released earlier this year. An unfortunate side affect after patching or upgrading an ESXi host which contains this modified ne1000 VIB is that it will be replaced by a newer version of the VIB and causes the NIC to no longer be recognized again.

A quick workaround is to simply re-install the modified ne1000 VIB and network connectivity will be restored which is less than ideal. A new vSphere Image Profile can also be created that contains both the patch/upgrade you intend to apply along with the modified ne1000 VIB, ensuring that you remove the newer version which may not be ideal as well. In speaking with Songtao, a VMware Engineer who I worked with on the USB Network Native Driver for ESXi, about this issue and he came up with a very simple solution. Lets choose a different name for the VIB module which removes all the complexity mentioned above. This solution would allow for both drivers to coexists and more importantly, it is persistent across patching and upgrades of ESXi.

Removable M.2 NVMe SSD PCIe enclosure by Icy Dock

07/27/2020 by William Lam 2 Comments

My homelab is a constant experiment and hardware components are moved around for various testing, especially when it comes to networking and storage. When needing to move around an M.2 NVMe SSD, complexity of taking apart a system will vary on the platform but generally it is inconvenience. When I came to learn that Icy Dock, a manufacturer of storage enclosures, will be releasing a removable M.2 NVMe SSD tray that is connected to PCIe expansion slot, I knew I had to get my hands on it.

The good folks over at Icy Dock were kind enough to send me an early evaluational unit of the upcoming MB840M2P-B which is now available for $69. ~~will be released in August and should retail for around $80 USD (final prices are still TBD)~~. The use case above may not apply to most folks and is probably unique to my specific hardware usage but I think this is still a very interesting solution that is still useful to be aware of if you are your own homelab whitebox and have a spare PCIe slot. Icy Dock also produces many other types of storage enclosures that you might find interesting based your own needs.

For my setup, I installed the MB840M2P-B into my Intel NUC 9 Pro, which is definitely not easy to take apart. This is especially true for the two M.2 which is attached to the NUC Element but even more painful to get to the 3rd M.2 which is located under the baseboard. For my specific use case, this was well worth using up one of the PCIe slots on the NUC 9 Pro! This enclosure can also be added to the new 2019 Mac Pro which is another platform that Icy Dock sees benefiting from this solution.

Two methods to network boot Raspberry Pi 4

07/07/2020 by William Lam 3 Comments

My Raspberry Pi 4 (8GB) model just came last week and after completing my RADIO (VMware's R&D Innovation Offsite) session recording, I wanted to setup my new rPI so I can start playing with it when I had some spare time. I also have the 4GB model but it was running quite hot as I was using the default case (do not recommend) and decided to put that aside for now. I ended up purchasing the 8GB model from Canakit which includes additional heatsinks and nice built-in fan with their custom case.

Look who just arrived to join the rest of 🥧 family! This will be my reward for tomorrow after I finish my #RADIO session recording pic.twitter.com/h9kxWRpM8S

— William Lam (@lamw) July 1, 2020

After spending some time setting up the rPI OS and applying the various updates, I was not too keen on using the SD Card, especially as some folks on forums mentioned that it can easily be worn out depending on the type of workload. While researching online and some references pointed out by colleagues, I came to learn that in addition to booting from USB which is now possible with rPI, you can also network boot the rPI without an SD Card or any storage for that matter! This immediately resonated with me, especially as I am a huge fan of scripted installations such as Kickstart/Jumpstart.

Here are all the resources that I had used that provided all the heavy lifting. I have optimized some of the commands to make it easy for anyone to simply copy and past who is new to setting up an rPI. The authors below did a fantastic job in outlining all the details, so I highly recommend a read if you would like to get more information. I also will be going over an alternative method at the end for booting the rPI over the network which is not really talked about.

One really cool thing that I came to learn while setting up the infrastructure to network boot an rPI was the use of dnsmasq, which I have used in the past but I did not realize it could do so much more. I may have spent more time playing with dnsmasq than with the rPI itself and I will probably cover this in another blog post on how you can easily setup a PXE/gPXE/iPXE system to enable automated OS installation (e.g. Kickstart) that can be used with ESXi or any other OS for that support network installations via BIOS/UEFI.

Configure NSX-T Edge to run on AMD Ryzen CPU

05/06/2020 by William Lam 3 Comments

The vast majority of VMware Homelabs is still Intel-based today but I have been seeing a slow rise of AMD-based kits being adopted, especially with AMD's desktop line of CPUs known as Ryzen. One of the considerations on whether you could use an AMD processor was whether you were planning to deploy NSX-T and in earlier releases, only Intel was supported as the NSX-T Edge required support for Data Plane Development Kit (DPDK) and this was only supported with Intel-based processors.

With the latest NSX-T 3.0 release, AMD-based processors are now supported and per the release notes, the following CPUs can be used:

AMD EPYC 7xx1 Series (Naples)
AMD EPYC 3000 Embedded Family and newer
AMD EPYC 7xx2 Series (Rome)

You will notice that only AMD's server line of CPUs known as EPYC are currently supported, which makes sense for running Production workloads. If you attempt to deploy an NSX-T Edge Node running on a non-EPYC platform, you will get an error message stating the CPU is not supported and I figured this was probably due to the lack of DPDK support in the consumer CPUs.

Yesterday, in our internal "Homelab" Slack channel, I came across an interesting tidbit from Andrea Spagnolo, a Sr. Field Engineer in our Cloud Native Business Unit who shared a pretty neat trick on how to get latest NSX-T 3.0 release to work with a Ryzen-based CPU.

Disclaimer: This is not officially supported by VMware. The behaviors described here can change in the future

First off, I want to thank Andrea for sharing but also credit to Beniamino Guarnaschelli and his blog post here which actually gave Andrea the idea to take a closer look as he was trying to get this setup in his own personal homelab.

Changing the default size of the ESX-OSData volume in ESXi 7.0

05/02/2020 by William Lam 8 Comments

In ESXi 7.0, a new partition scheme was introduced which also brings along a new set of storage requirements. These changes are explained in the official documentation here and the following VMware KB 77009 also contains some additional info which can be helpful. Storage changes are not easy but this was necessary to not only better support some of the current capabilities but more importantly, it setups the foundation for future ESXi capabilities.

The biggest change to the partition layout is the consolidation of VMware Tools Locker, Core Dump and Scratch partitions into a new ESX-OSData volume (based on VMFS-L). This new volume can vary in size (up to 138GB) depending on a number of factors including the current ESXi boot media (USB SD-Card, Local Disk) but also the size of the device itself, which is explained in the official documentation.

From some of the comments on Twitter, Reddit and the direct inquiries that I have received, this new behavior seems to be most impactful to smaller homelabs where a fresh install of ESXi 7.0 has been performed. Folks have shared that their ESX-OSData volume has taken up 120GB which can be quite significant if you have a smaller disk which can be quite common. I normally install ESXi on a USB device and I also use vSAN, which has a different behavior and I have also not upgraded my physical ESXi host (E200-8D) to 7.0 yet.

I performed a fresh installation of ESXi 7.0 (running as Nested ESXi VM) that was configured with 1TB of storage and here is what the filesystem layout now looks:

We can see that the ESX-OSData volume takes up ~119.75GB, which is not too bad for 1TB volume but I can understand this may not be ideal if you have something smaller such as 250GB to 512GB disk. Due to the size of the local device, the boot options mentioned in the KB would not be helpful and I was curious myself if this ESX-OSData volume size could be configurable. In doing some research it looks like the size of the ESX-OSData can be specified using the following ESXi boot option (SHIFT+O during the initial boot) called autoPartitionOSDataSize

Disclaimer: This may not be officially supported by VMware as it deviates from the system defaults and can have other unintended behaviors. Use at your own risk.