Home Lab

Running Nested ESXi, NSX-V or NSX-T on top of NSX-T

11/22/2019 by William Lam 5 Comments

Nested Virtualization is an extremely useful tool that helps customers easily test and try out new VMware products and solutions before rolling that into a proper development environment for further validation. This is especially handy for those wanting to setup an NSX-based environment and simulating their actual deployment topology, configuration and upgrade workflows.

In this past year, I have seen a 10x increase in the number of NSX-T based questions that have come up from customers and our field, the adoption of NSX-T is definitely in full swing. As expected, questions about running Nested ESXi on top of a physical NSX-T deployment has come up and there has actually been several variations that have been asked about whether that is Nested ESXi using VSS, VDS, NSX-V or even NSX-T running on top of an N-VDS, which is the virtual switch that NSX-T uses.

Luckily all of these combinations work and just require some basic configuration changes within NSX-T. However, before I continue, let me remind folks again that VMware does NOT officially support Nested Virtualization.

Hands on with new AMD “NUC” platform from ASRock

11/16/2019 by William Lam 6 Comments

It has been rumored for some time now that AMD would be re-entering the embedded/small form factor (SFF) market and with their latest generation of Ryzen CPUs, timing could not be more perfect. With the rise of Edge and IoT workloads, new constraints like cost, footprint, low to no tech-touch and power consumption is now top of mind more than ever for customers.

At VMware, we certainly see interests from customers across this spectrum whether it is a traditional "ROBO" environment running a handful of Virtual Machines for a supermarket or retail chain to an industrial site such as an oil rig or factory to mobile infrastructure like a car, ship or even a submarine. On top of these use cases, personal and development labs continue to be extremely popular amongst our customer base for learning and training purposes.

Today, the Intel NUC is still by far the most popular platform due to its size, portability, cost and low power consumption for running VMware vSphere, vSAN & NSX. The idea of an AMD "NUC" kit which includes a chassis, motherboard and CPU is something that that many folks have been asking for. With the market mostly dominated by Intel, I certainly welcome AMD re-entering this market as this will certainly drive new solutions and innovations in this market. With that said, AMD has re-entered into this market and this is initially through partnerships with several OEMs/manufacturers.

USB Native Driver Fling for ESXi adds support for Multi-Gig (1G/2.5G/5G) Adapter

09/27/2019 by William Lam 3 Comments

Today, we have an exciting update to give on our USB Network Native Driver for ESXi Fling which has had two updates since releasing earlier this year and has been extremely well received by the VMware community. As many of you know, I am always on the look out for new and innovative tech that can help enable our customers, especially when it comes to building home labs to learn about the latest and greatest VMware software.

Several months back, I came to learn about a really cool USB-based Multi-Gigabit Network Adapter (QNA-UC5G1T) from QNAP which can negotiate with speeds up to 1Gbps, 2.5Gbps and 5Gbps. I was not familiar with the multi-gig specification but it it looks like it was created as a standard back in 2016 as IEEE 802.3bz. This initially evolved from advancements in wireless technology but more recently it started to make its way into ethernet-based devices.

Although this particular device is from QNAP, the underlying chipset is actually from Aquantia, now part of Marvell. If the name sounds familiar, it should as Aquantia is also the vendor to Apple for their 10GbE NICs in both the 2018 Mac Mini and new iMac Pros. In fact, their chipsets are also used in a number of Thunderbolt 3 to 10GbE NICs which also works with ESXi. Access to 10GbE is certainly more common these days but it certainly is not for everyone and not all platforms can be expanded to support it.

The QNA-UC5G1T device is not only small but because it is USB-based, you are more likely to have spare USB ports on your system than say a traditional PCIe slot or Thunderbolt 3 port. From a cost standpoint, this device is about half the cost of the 10GbE Thunderbolt adapter coming in at $79 USD and can be ordered from Amazon. As far as I know, QNAP is the only vendor who has produced a multi-gig USB adapter, but perhaps in the future, there will be other vendors.

New hardware support & enhancements to USB Network Native Driver for ESXi

06/17/2019 by William Lam 1 Comment

Since releasing the USB Network Native Driver for ESXi Fling back in February, the feedback and support for this Fling has simply been amazing. I especially enjoy hearing from customers, almost on a weekly basis, on how they are taking advantage of this solution in their vSphere/vSAN/NSX home lab and enabling for more complex networking topologies.

When we released the fling, we had focused on the most popular USB 3.0 network adaptors that we had accessed to which were based on AX8817a, AX88179 & RTL8153 chipsets. The initial list started out with just 5 devices, but we knew there were going to be others. In fact, Songtao and I have received plenty of feedback from our customers on what they would like to see supported, including USB 2.0 network adaptors. Another highly requested feature that customers have been asking for is support for Jumbo Frames and this is usually in the context of setting up either an NSX-V or NSX-T environment.

Thunderbolt 3 enclosures with (Single, Dual & Quad) M.2 NVMe SSDs for ESXi

06/03/2019 by William Lam 14 Comments

Thunderbolt 3 (TB3) and eventually USB 4 is a really fascinating technology and I believe it still has so much untapped potential, especially when looking at Remote/Branch Office (ROBO), Edge and IoT types of deployments. TB3 was initially limited to Apple-based platforms, but in the last couple of years, adoption has been picking up across a number of PC desktop/laptops including the latest generations of Intel NUCs which are quite popular for vSphere/vSAN/NSX Home Labs. My hope with USB 4 is that in the near future, we will start to see servers with this interface show up in the datacenter 🙂

In the mean time, I have been doing some work with TB3 from a home lab standpoint. Some of you may have noticed my recent work on enabling Thunderbolt 3 to 10GbE for ESXi and it should be no surprise that the next logical step was TB3 storage. Using a Thunderbolt interface to connect to external storage, usually Fibre Channel is something many of our customers have been doing for quite some time. In fact, I have a blog post from a few years back which goes over some of the solutions customers have implemented, the majority use case being Virtualizing MacOS on ESXi for iOS/MacOS development. These solutions were usually not cheap and involved a sizable amount of infrastructure (e.g. storage arrays, network switches, etc) but worked very well for large vSphere/MacOS based environments.

Quick Tip – Crucial NVMe SSD not recognized by ESXi 6.7

05/19/2019 by William Lam 19 Comments

If you own or have recently purchased Crucial NVMe SSD such as CT1000P1SSD8 (1TB M.2 NVMe SSD) or CT500P1SSD8 (500GB M.2 NVMe SSD), please be aware that these devices may no be recognized by ESXi after upgrading to the latest release. Thanks to Pete Lindley, (OCTO for End-User Computing), who reached out last week regarding the observation as well as a workaround for the problem. This was also quite timely as I recently purchased a Crucial M.2 NVMe SSD and would have also ran into this problem.

It turns out these Crucial devices were working fine while running on ESXi 6.5 Update 2 but was no longer recognized in latest release of ESXi 6.7 Update 2. It is unclear whether support for these SSDs were removed intentionally or unintentionally, but in either case, these devices are not officially on VMware's Hardware Compatibility List (HCL).

UPDATE (05/23/19) - After speaking with a few folks who took a closer look, the issue is due to the fact that we added support for NVMe 1.3 spec in latest ESXi 6.7 Update 2 release, but because these are "consumer" devices, they did not conform to the latest specification and hence the driver is unable to claim the device. This is another good reminder when using components not on VMware HCL, this is always a risk from a home lab perspective. In general, I know Samsung and Intel NVMe SSD usually works quite well without issues but always good to do some research. I think Engineering is looking to see if there are other workarounds for the future, but for now, you can use the workaround below.

The easy workaround that Pete found was to simply replace the NVMe driver from ESXi 6.7 Update 2 (1.2.2.27-1vmw.670.2.48.13006603) with one found in ESXi 6.5 Update 2 (1.2.1.34-1vmw.650.2.50.8294253). To so do, simply copy nvme.v00 to /bootbank from either an existing ESXi 6.5 Update 2 system or directly from the ISO. Please note, any future updates or patches to the ESXi host will most likely override the updated driver.