Intel NUC

ESXi on 11th Gen Intel NUC (Panther Canyon) & (Tiger Canyon)

01/13/2021 by William Lam 16 Comments

The highly anticipated 11th Generation Intel NUCs based on the new Tiger Lake processors has just been announced by Intel and I am excited to share my first hand experience with this new NUC platform. There are currently two models in the new 11th Gen lineup: the Intel NUC 11 Performance codenamed Panther Canyon (pictured on the left) which is the successor to the 10th Gen (Frost Canyon) NUC and the Intel NUC 11 Pro codenamed Tiger Canyon (pictured on the right) which is the successor to the 8th Gen (Provo Canyon) NUC.

There are a number of new improvements and capabilities that will make these new NUCs quite popular for anyone looking to build or upgrade their vSphere environment in 2021.

Before diving right in, I must say I love the new aesthetic look of the NUC chassis. In previous versions, the lid had a glossy and shiny finish, which easily left hand prints. These new models now have a clean matte finish. The NUC 11 Performance has a smoother feel compared to the NUC 11 Pro which has more of a texture to the finish, which I personally prefer. The other noticeable change is the power adapter, which is now half the size now which is nice for those looking to have several of these new kits sitting next to each other.

Intel NUC with 512GB memory

12/03/2020 by William Lam 7 Comments

Yes, you read that correctly. 512 gigabytes of memory on an Intel NUC. Not only is this pretty 🤯 but this is actually possible today with an already released Intel NUC!

A few months back, I was made aware of some really cool technology from Intel called Intel Memory Drive Technology (IMDT) which leverages Intel Optane SSDs to extend memory of a system beyond its physical memory (DRAM) capacity. This technology is made possible with their IMDT software, which is a purpose built Hypervisor whose sole purpose is to just manage memory and this Hypervisor runs on top of the Intel Optane SSD. You can think of this like a Software-Defined Memory (SDM) solution. In fact, SDM was actually coined in this performance white paper evaluating IMDT with scientific-based applications back in 2018.

Note: This should not be confused with Intel Optane and its Datacenter Persistent Memory (PMEM) solution which vSphere already supports today.

The target use case for this type of technology is for memory intensive applications such as SAP HANA, Oracle, Redis, Memcache and Apache Spark to just name a few. These workloads can easily gobble up 10's of terabytes of memory that can bring a number of challenges when needing to scale up these solutions. High capacity memory DIMMS are not only expensive, but once you exhaust the number of physical DIMM slots, your only option for scale up is to add additional servers which is very costly.

Using IMDT, customers can expand their physical DRAM capacity from 8x to 15x, which can significantly improve cost, performance but also the operational overhead in managing additional systems. Putting aside the in-memory based workloads, I think there is also huge potential for general purpose workloads that can also get the exact same benefits, especially when you think about constraints like power, cooling and location such as Edge or ROBO locations. Since this solution works on an Intel NUC, a really interesting use case for this technology that immediately came to mind was for a vSphere/NSX/vSAN homelab environment.

Complete vSphere with Tanzu homelab with just 32GB of memory!

11/09/2020 by William Lam 26 Comments

Since the release of vSphere 7.0 Update 1, the demand and interests from the community on getting hands on with vSphere with Tanzu and the new simplified networking solution, has been non-stop. Most folks are either upgrading their existing homelab or looking to purchase new hardware that can better support the new features of the vSphere 7.0 release.

Although vSphere with Tanzu now has a flavor that does not require NSX-T which helps reduces the barrier on getting started, it still has some networking requirements which may not be easily met in for all lab environments. In fact, this was actually the primary reason I had started to look into this since my personal homelab network is very basic and I do not have nor want a switch that can support multiple VLANs, which is one of the requirements for vSphere with Tanzu.

While investigating for a potential solution, which included way too MANY hours of debugging and troubleshooting, I also thought about the absolute minimal amount of resources I could get away with after put everything together. To be clear, my homelab is comprised of a single Supermicro E200-8D which has 128GB of memory and that has served me well over the years and I highly recommend it for anyone that can fit that into their budget. With that said, I did set out with a pretty aggressive goal of using something that is pretty common in VMware homelabs which is an Intel NUC and with just 32GB of memory.

Here is the hardware BOM (similar hardware should also work):

Intel NUC 10i7FNH
32GB memory
Single 250GB M.2 NVMe SSD
- NUC can support two SSD (M.2 + SATA), you can always go larger

Here is the software BOM:

Note: The Intel NUCs (Gen 6 to 10) can all support up to 64GB of memory and this is one of the best upgrades you can give yourself, but if you only have 32GB of memory, this will also work.

The final solution will comprise of the following:

1 x vCenter Server Appliance (VCSA) running on the Intel NUC self-managing the ESXi host
VMFS storage will be used instead of vSAN to reduce memory footprint (If you have 64GB of memory, recommend using vSAN)
Onboard NIC will be used for all traffic and will be attached to a Distributed Virtual Switch (VDS)
- 3 x Distributed Portgroups will be configured on top of your existing LAN network, the latter two will be routed through our Photon OS Router VM
  - Management - Existing LAN network
  - Frontend - 10.10.0/24
  - Workload - 10.20.0.0/24
1 x vSphere with Tanzu Cluster enabled with Workload Management
1 x HAProxy VM deployed using 3-NIC configuration
1 x Photon OS Linux VM used as a Router for IP forwarding and optionally, a DNS server if you do not already have one
9 x IP Addresses in total will be required from your local LAN network
- 4 x IP Addresses which should map to following hostnames or similiar
  - esxi-01.tanzu.local
  - vcsa.tanzu.local
  - router.tanzu.local
  - haproxy.tanzu.local
- 5 x IP Addresses in a consecutive block (e.g. 192.168.30.20-192.168.30.25) will be needed for the Supervisor Control Plane VMs

As part of this solution, I have automated as much of the tasks as possible and all scripts used for this solution can be found at https://github.com/lamw/vsphere-with-tanzu-homelab-scripts which I will be referencing throughout the instructions. There are also a number of techniques and tricks I am using to be able to reduce the overall memory footprint for setting up vSphere with Tanzu, obviously these should not be used in a Production grade environment.

I also want to give a huge thanks to Timo Sugliani for all of his help with the networking question/challenges and Mayank B. from the vSphere with Tanzu Engineering team who helped with the debugging and ultimately making this solution a possibility.
[Read more...] about Complete vSphere with Tanzu homelab with just 32GB of memory!

ESXi 7.0 Update 1 now includes NIC driver for Intel NUC 10

09/21/2020 by William Lam 14 Comments

With the upcoming release of vSphere 7.0 Update 1 and specifically ESXi 7.0 Update 1, support for the onboard NIC of the Intel NUC 10 (Frost Canyon) is now included and the community ne1000 VIB driver is no longer needed. If you had previously installed the community driver, you can uninstall the VIB after successfully upgrading to ESXi 7.0 Update 1.

Disabling TPM 2.0 connection cannot be established message in ESXi for Intel NUC 10

08/21/2020 by William Lam Leave a Comment

For Intel NUC 10 (Frost Canyon) owners who have installed ESXi may have noticed that even after disabling Intel's Trusted Platform Module (TPM), the following warning message "TPM 2.0 device detected but a connection cannot be established." is still being displayed in the vSphere UI as shown in the screenshot below.

Thanks to Reddit member mscaff and casperette who recently discovered and confirmed that the latest BIOS (FN0044) resolves an issue where disabling TPM in the BIOS was not actually working which would explain the behavior observed above. The really interesting thing is that I had initially ran into this problem several months back and after speaking with some internal VMware folks, I was able to get rid of this message without this update. This involved installing Windows 10 and clear the TPM keys which may have still been cache but since then, it has not been reproducible by other folks. In any case, it is always recommended to check and update to latest BIOS to ensure you have all the latest bug fixes.

Lastly, Intel states support for TPM 2.0 for these NUCs, so why is ESXi complaining? Well, it has to do with the interface type and not with SHA1 vs SHA256 which are both supported on the NUC 10. The NUC only supports CRB but proper compliant TPM 2.0 chip must support FIFO which is not configurable the last time I had checked. For more detail requirements and configuration of TPM 2.0 on ESXi, please refer to this blog post.

Enhancements to the community ne1000 VIB for Intel NUC 10

08/03/2020 by William Lam 17 Comments

As many of you know, the onboard Intel NIC (8086:0d4f) found in the 10th generation of the Intel NUC (Frost Canyon) is not automatically recognized by ESXi and requires an updated ne1000 VIB which was released earlier this year. An unfortunate side affect after patching or upgrading an ESXi host which contains this modified ne1000 VIB is that it will be replaced by a newer version of the VIB and causes the NIC to no longer be recognized again.

A quick workaround is to simply re-install the modified ne1000 VIB and network connectivity will be restored which is less than ideal. A new vSphere Image Profile can also be created that contains both the patch/upgrade you intend to apply along with the modified ne1000 VIB, ensuring that you remove the newer version which may not be ideal as well. In speaking with Songtao, a VMware Engineer who I worked with on the USB Network Native Driver for ESXi, about this issue and he came up with a very simple solution. Lets choose a different name for the VIB module which removes all the complexity mentioned above. This solution would allow for both drivers to coexists and more importantly, it is persistent across patching and upgrades of ESXi.