virtuallyGhetto

Apple NVMe driver for ESXi using new Community NVMe Driver for ESXi Fling 

by 23 Comments

VMware has been making steady progress on enabling both the Apple 2018 Mac Mini 8,1 and the Apple 2019 Mac Pro 7,1 for our customers over the past couple of years. These enablement efforts have had its challenges, including the lack of direct hardware access for our developers and supporting teams due to the global pandemic but also the lack of participation from Apple has certainly not made this easier.

Today, I am happy to share that we have made some progress on enabling ESXi to see and consume the local Apple NVMe storage device found in the recent Apple T2-based mac systems such as the 2018 Mac Mini and 2019 Mac Pro. There were a number of technical challenges the team had to overcome, especially since the Apple NVMe was not just a consumer grade device but it also did not follow the standard NVMe specification that you normally would see in most typical NVMe devices.

This meant there was a lot of poking and prodding to reverse engineer the behavior of the Apple NVMe to better understand how this device works, which often leads to sudden reboot or PSODs. With the Apple NVMe being a consumer device, it also meant there were a number of workarounds that the team had to come up with to enable ESXi to consume the device. The implementation is not perfect, for example we do not have native 4kn support for SSD devices within ESXi and we had to fake/emulate a non-SSD flag to work around some of the issues. From our limited testing, we have also not observed any significant impact to workloads when utilizing this driver and we also had had several internal VMware teams who have already been using this driver for a couple of months now without reporting any issues.

A huge thanks goes out to Wenchao and Yibo from the VMkernel I/O team who developed the initial prototype which has now been incorporated into the new Community NVMe Driver for ESXi Fling.

Caveats

Before folks rush out to grab and install the driver, it is important to be aware of a couple of constraints that we have not been able to work around yet.

  1. ESXi versions newer then ESXi 6.7 Patch 03 (Build 16713306) is currently NOT supported and will cause ESXi to PSOD during boot up.
  2. The onboard Thunderbolt 3 ports does NOT function when using the Community NVMe driver and can cause ESXi to PSOD if activated.

Note: For detailed ESXi version and build numbers, please refer to VMware KB 2143832

VMware Engineering has not been able to pin point why the ESXi PSOD is happening. For now, this is a constraint to be aware of which may impact anyone who requires the use of the Thunderbolt 3 ports for additional networking or storage connectivity.

With that out of the way, customers can either incorporate the Community NVMe Driver for ESXi offline bundle into a new ESXi Image Profile (using vSphere Image Builder UI/CLI) and then exporting image as an ISO and then installing that on either a Mac Mini or Mac Pro or you can manually install the offline bundle after ESXi has been installed over USB and upon reboot, the local Apple NVME will then be visible for VMFS formatting.

Here is a screenshot of ESXi 6.7 Patch 03 installed on my 2018 Mac Mini with the Apple NVMe formatted with VMFS and running macOS VM

VMware customer production use cases for Intel NUC 

by 3 Comments

The Intel NUC also known as the Next Unit of Computing is a very popular platform for running VMware based homelabs. I have been working with the Intel NUCs since 2016 with their 6th Generation model when I decided to rebuild my personal home lab. Since then I have continued my efforts to ensure that vSphere continues to run extremely well on this amazing little platform even if it is not officially supported by VMware, which now also includes the latest 11th Generation (Tiger and Panther Canyon NUCs).

At the end of last year, I came across this fascinating Intel NUC documentary that was put together by Robtech, which I highly recommend a watch.

While listening to some of the use cases that SimplyNUC had observed over the years which has spanned land ⛰️, air 🛫, sea 🛳️ and space 🚀, it got me thinking about some of the use cases that I had come across while talking to our VMware customers.

Disclaimer: The Intel NUC is not officially supported by VMware and therefore they are not listed on the VMware HCL

A common misconception is that Intel NUCs are only useful for homelab purposes and has no place for running production workloads, which is just simply not true. Here are some of the common use cases that I have seen over the years, most of which are deployed at the Edge/ROBO:

  • vSphere Development/Testing, Education and Training
  • Retail, Grocery, Industrial Factories and Ships
  • Build Automation (CI/CD)
  • Telco/NFV (e.g. Network/Hardware monitoring)
  • Virtual Desktop Infrastructure (VDI)

I also wanted to take this opportunity and to share some of the stories on how some of our customers have taken advantage of this platform, even though it is not officially supported by VMware and some of the underlying business drivers. Hopefully these stories will educate, resonate and perhaps even inspire other customers to explore different computing platforms, especially at the Edge where constraints and requirements will differ quite significantly when compared to a typical Enterprise Datacenter.

If you would like to share your story of how you are using Intel NUC and VMware for production, feel free to reach out using the contact page.

[Read more...] about VMware customer production use cases for Intel NUC 

New Community Networking Driver for ESXi Fling

by 12 Comments

I am super excited to announce the release of a new Community Networking Driver for ESXi Fling! The idea behind this project started about a year ago when we released an enhancement to the ne1000 driver as a community update which enabled ESXi to recognize the onboard network adapter for the Intel 10th Gen (Frost Canyon) NUC. Although the Intel NUC is not an officially supported VMware platform, it is extremely popular amongst the VMware Community. In working with the awesome Songtao, we were able to release this driver early last year for customers to take advantage of the latest Intel NUC release.

At the time, I knew that this would not be the last occurrence dealing with driver compatibility. We definitely wanted an easier way to distribute various community networking drivers that is packaged into a single deliverable for customers to easily consume and hence this project was born. In fact, it was quite timely as I had just received engineering samples of the new Intel NUC 11 Pro and Performance (Panther Canyon and Tiger Canyon) at the end of 2020 and work needed to be done before we could enable the onboard 2.5GbE (multi-gigabit) network adapter which is a default component of the new Intel Tiger Lake architecture. As reported back in early Jan, Songtao and colleague Shu were successful in getting ESXi to recognize the new 2.5GbE network adapter and has also been incorporated into this new Fling. In addition, we also started to receive reports from customers that after upgrading to a newer ESXi 7.0 releases, the onboard network adapters for the Intel 8th Gen NUC was no longer functioning. In an effort to help customers with this older platform, we have also updated the original community ne1000e driver to include the relevant PCI IDs within this Fling.


The new Community Networking Driver for ESXi is for PCIe-based network adapters and currently contains the following two driver modules:

  • igc-community - which adds support for Intel 11th Gen NUCs and any other hardware platform that uses the same 2.5GbE devices
  • e1000-community - which adds support for Intel 8th Gen NUC and any other hardware platform that uses same 1GbE devices

For a complete list of supported devices (VendorID/ProductID), please take a look at the Requirements tab on the Fling website. As with any Fling, this is being developed and supported in our spare time. In the future, we may consider adding other types of devices based on feedback from the broader community. I know Realtek-based PCIe NICs is something that many have been asking about and as mentioned back in this blog post, I have been in engaged with the Realtek team and hopefully in the near future, we may see an ESXi driver that can support some of the more popular devices in the community. If there are other PCIe-based networking adapters that could fit the Fling model, feel free to leave a comment on the Fling website and we can evaluate as time permits.

Useful Kubernetes Tricks/Tools

by 5 Comments

As you can see from my weekly Tweets, I am usually doing something that involves Kubernetes (K8s), whether that it is playing with Knative and our VMware Event Broker Application solution or doing some deep R&D research. I wanted to share some of the tips and tools that I have been using which has been helpful for me to better learn and interact with K8s whether that is using vSphere with Tanzu, Tanzu Kubernetes Grid (TKG), KinD or other K8s distributions like k3s.

By no means is this a comprehensive list and I know there are many other collections including the curated Awesome Kubernetes project. If there are other useful K8s tricks or tools that you would like to share, feel free to leave a comment below.
[Read more...] about Useful Kubernetes Tricks/Tools

Retro DOS Games on Kubernetes

by 14 Comments

Over the weekend I discovered this really cool Github project by Paolo Mainardi called additronk8s which is a retro DOS game engine (using DOSBox) built as a Kubernetes (K8s) custom controller and implemented in Javascript. Having spent quite a bit of time last year building out our VMworld 2019 demo which ran a number of MSDOS games on ESX 3.0 running on VMware Cloud on AWS, I definitely had to give this project a try!

In addition to having some fun playing with K8s, this solution was also quite interesting from the techniques that were used, here is a description from Paolo's own words:

One of the goal of this project was to use just Kubernetes API without any external dependency (neither the storage), in fact is noteworthy that ConfigMaps are (ab)used as a persistent storage layer, using a simple technique of split/merge parts of files to save the games.

AdditronK8S Scheme
After poking around the repository, I found that it was not very intuitive to get started. In fact, it took me some time to figure out everything and lots of trial/error. I eventually got everything working and successfully deployed several DOS games to my Tanzu Kubernetes Grid (TKG) Cluster which I had running in my homelab. Below are the detailed instructions on how to quickly get this solution stood up which just requires a vanilla K8s deployment.

[Read more...] about Retro DOS Games on Kubernetes

VEBA + Knative + k3s on ESXi-Arm

by Leave a Comment

In response to a customer request to add Arm64 support for our VMware Event Router, I have been spending some more time playing with k3s (lightweight Kubernetes distribution for Arm) running on ESXi-Arm using a Raspberry Pi. Not only was this a good learning experience that exposed to me to the broader Arm ecosystem, which is still maturing but it also took me down several 🐰🕳️ which got me exploring new tools that I had never used before such as Buildpacks and Docker buildx to name a few.

This past weekend, I was finally successful in setting up our VMware Event Router for Arm using the Knative processor on a k3s cluster using ESXi-Arm running on a Raspberry Pi 4b 8GB model! As of writing this, the following versions were used:

  • Knative Serving v0.20.0
  • Knative Net Contour v0.20.0
  • Knative Eventing v0.20.1
  • RabbitMQ Cluster Operator v0.4.0

In addition, I was able to also convert the Knative python echo function that was originally created by my colleague Michael Gasch and build an Arm64 version of the Knative python echo function which demonstrates the integration of VEBA with the Knative processor connected to a vCenter Server as my event source.

 

For those interested in just the VMware Event Router Arm64 image, you can access it here and we plan to make that an official image shortly. For those interested in setting up a fully functional Arm deployment of VEBA and Knative processor, you can find the detailed instructions below.

[Read more...] about VEBA + Knative + k3s on ESXi-Arm