esxi

Kubernetes on ESXi-Arm using k3s

10/16/2020 by William Lam 9 Comments

The tiny form factor of a Raspberry Pi (rPI) is a fantastic hardware platform to start playing with the ESXi-Arm Fling. You can already do a bunch of fun VMware things like running a lightweight vSAN Witness Node to setting up basic automation environment for PowerCLI, Terraform and Packer to running rPI OS as VM, enabling some neat use cases like consolidating your physical rPI assets which might be running RetroPi and Pi-Hole which many home labbers are doing.

In addition to VMware solutions, its is also a great platform to learn and tinker with new technologies like Kubernetes (K8s) which I am sure many of you have been hearing about 🙂 Although our vSphere with Tanzu and Tanzu Kubernetes Grid (TKG) does not currently work with the ESXi-Arm Fling, I have actually been meaning to try out a super lightweight K8s distribution designed for IoT/Edge called k3s (pronounced k-3-s) which also recently joined Cloud Native Computing Foundation (CNCF) Sandbox level.

k3s is supported on rPI and you normally would have multiple rPI devices to represent the number of nodes, for example if you want a basic 3-Node cluster, you would need three physical rPI devices. With ESXi-Arm, you can now create these nodes as VM, using just a single rPI. This opens up the door for all sorts of explorations, you can create HA cluster or try out more advanced features which might be more difficult if you needed several physical devices. If you mess up, you can simply re-deploy the VM without much pain or simply clone the VM.

In my setup, I am using 3 x Photon OS VMs. One for the primary node and two for k3s worker nodes. You can certainly install k3s on any other Arm-based OS including rPI OS (which can now run as a VM as mentioned earlier).

How to copy the Raspberry Pi UEFI files and boot ESXi-Arm using just USB device?

10/14/2020 by William Lam 2 Comments

There are actually a number of ways to boot and install the ESXi-Arm Fling, but the easiest method is already outlined in the official ESXi-Arm Raspberry Pi (rPI) documentation and the PDF can be downloaded from the Fling website. As a quick refresher, you only need to have two storage devices.

SD Card - Used to store the rPI UEFI files which is required to boot ESXi-Arm Installer
- ESXi-Arm can not run on SD Card and for these reasons, you do not need a large capacity SD Card
USB Device - Contains ESXi-Arm Installer / Installation
- After the ESXi-Arm installer boots, you can actually re-use the exact same USB device for the installation of ESXi-Arm itself. A separate USB device is not required unless that is your goal or if the capacity is not enough for running VMs

The boot process after the ESXi-Arm installation is that the UEFI firmware will first load on the rPI and then it will boot up the ESXi-Arm from the USB device. As mentioned, there are other variations but this is the most basic option. The other nice behavior is that if you need to re-install ESXi-Arm, you simply create a bootable USB device with the ESXi-Arm installer and then install that right on the same USB device without having to mess with UEFI image. This also allows you to perform scripted installation also known as Kickstart, which is something I will be covering in the future that takes UEFI image into consideration.

I have seen a few questions asked whether it is possible to have everything run off of the SD Card and/or USB Device and the answer is yes to certain degree.

It is possible to put the ESXi-Arm installer + UEFI on SD Card but ESXi-Arm will NOT be able to use it as installation media, so there is not a whole ton of value there.
It is possible to have both the UEFI image and ESXi Installation on the same USB device, especially if you do not have spare SD Cards which apparently has come up a few times

In this blog post, I will outline the instructions for booting an installed ESXi-Arm installation completely off of the USB device without the needing an SD Card containing the UEFI image.

Quick Tip – vmware-iso builder for Packer now supported with ESXi 7.0

10/12/2020 by William Lam 2 Comments

When vSphere 7.0 GA'ed earlier this year, one of the changes that I had noticed while going through the release notes was the removal of the VNC Server on ESXi. By default, this is disabled but users could enable it on a per-VM basis and connect to a specific VM using VNC. Not many customers used this feature and it made sense on why it was removed.

However, one implication is that if you use HashiCorp Packer and the vmware-iso builder to created automated images with ESXi, it will no longer work after upgrading to ESXi 7.0 as Packer relies on this VNC interface to send automated keystrokes to a VM as part of its automation. After learning about this change with vSphere 7.0, I filed a Packer Github Enhanacement to see if someone would be open to re-implementing the keystrokes functionality by leveraging the vSphere HTML5 Console SDK which would then allow for the use of VNC over websockets. The PR was closed about a month ago and while recently working on the vCenter Event Broker Appliance (VEBA) project, I finally got a chance to verify the feature after upgrading my physical ESXi host to latest 7.0 Update 1 and happy to share that the vmware-iso builder now functions as before.

The following two lines should be added to your Packer template:

"vnc_over_websocket": true
"insecure_connection": true

For reference, you can also refer to the VEBA Packer template

An alternative workaround is to use the vsphere-iso builder which leverages the vSphere USB scan codes API to send keystrokes into a VM without having to rely on the VNC interface. One downside is that you do need have a vCenter Server as the vsphere-iso builder interacts with the vSphere API on vCenter Server rather than directly going to ESXi and this would also impact anyone using Free ESXi to build their Packer images.

The primary reason that I had not switched over to the vsphere-iso builder was that I had quite a few Packer templates using the vmware-iso builder and the syntax was not portable between the two. For this reason alone, I decided to hold off upgrading my physical ESXi host to 7.0 until now.

Workaround for ESXi-Arm in vSphere 7.0 Update 1

10/12/2020 by William Lam 2 Comments

In vSphere 7.0 Update 1, a new capability was introduced called the vCenter Cluster Services (vCLS) which provides a new framework for decoupling and managing distributing control plane services for vSphere. To learn more, I highly recommend the detailed blog post linked above by Niels. In addition, Duncan also has a great blog post about common question/answers and considerations for vCLS, which is definitely worth a read as well.

vSphere DRS is one of the vSphere features which relies on this new vCLS service and this is made possible by the vCLS VMs which are deployed automatically when it detects there are ESXi hosts within a vSphere Cluster (regardless if vSphere DRS is enabled or not). For customers who may be using the ESXi-Arm Fling with a vSphere 7.0 Update 1 environment, you may have noticed continuous "Delete File" tasks within vCenter that seems to loop forever.

This occurs because the vCLS service will first test to see if it can upload a file to the datastore, once it can, it will delete it. The issue is that the vCLS VMs are x86 and can not be deployed to an ESXi-Arm Cluster as the CPU architecture is not supported. There is a workaround to disable vCLS for the ESXi-Arm Cluster, which I will go into shortly. However, because vCLS can not properly deploy, it means vSphere DRS capabilities will not be possible when using vSphere 7.0 Update 1 with ESXi-Arm hosts. If this is desirable, it is recommended that to use either vSphere 7.0c or vSphere 7.0d if you wish to use vSphere DRS.

Note: vSAN does not rely on vCLS to function but to be able to use it, you must place your ESXi-Arm hosts into a vSphere Cluster and hence applying this workaround would be desirable for that use case.

Using ESXi-Arm Fling as a lightweight vSphere Automation environment for PowerCLI and Terraform

10/09/2020 by William Lam Leave a Comment

A set of use cases that I was really excited for when I first heard about ESXi-Arm a few years ago was around the topic of vSphere Automation and Development. I speak with many customers who are just starting out on their Automation journey whether that is using PowerCLI, one of our many vSphere Automation SDK or even directly to the new vCenter REST API which all new features are being exposed through these days.

One of the biggest challenge for new comers is simply getting access to hardware that they can start playing around with and although there are is plethora of vSphere Homelab choices, it does require some amount of investment, which is definitely worth it in the long run. However, if you are just getting started and maybe you want something that is a bit more lighter weight, there are not too many options outside of an Intel NUC. I know many consultants actually carry around an Intel NUC that contains several VM images that they use to with their clients, including demos.

With the small form factor, low cost and reduced power consumption of the Raspberry Pi, I think this really opens up the door for some interesting creative solutions:

Basic vSphere footprint that can be used for work or learning purposes
Easy way to learn and explore the vSphere API with an actual host and enabling real VM deployments
Trying out Infrastructure-as-Code (IaC) tools such as Terraform and Ansible
Quick way to run through basic demos in front of customers
On-demand and self-contained lab environment for small Hackathon at your local VMUG or even at VMworld

Something I was really interested in early on was to be able to use ESXi-Arm with the Raspberry Pi to not only have a basic ESXi environment but also have PowerCLI environment up and running in an Arm VM. My first thought was to get this setup using Photon OS, which not only has Arm distribution but also has support for Powershell and PowerCLI. I was hoping with some tinkering, I could easily get Powershell for Arm to run on PhotonOS (which it did) but I then ran into issues installing PowerCLI itself.

I decided to give up for now and take a look at Ubuntu which also supports Powershell for Arm, but the Microsoft documentation only listed instructions for 32-bit and ESXi-Arm requires a 64-bit. Taking a look at the Powershell release files, I noticed there was 64-bit package and with a few minor adjustments to the commands, I got PowerCLI installed and connected back to my rPI which was attached to my x86 vCenter Server!

vSAN Witness using Raspberry Pi 4 & ESXi-Arm Fling

10/08/2020 by William Lam 14 Comments

As hinted in my earlier blog post, you can indeed setup a vSAN Witness using the ESXi-Arm Fling running on a Raspberry Pi (rPI) 4b (8GB) model. In fact, you can even setup a standard 2-Node or 3-Node vSAN Cluster using the exact same technique. For those familiar with vSAN and the vSAN Witness, we will need to have at least two storage devices for the caching and capacity tier.

For the rPI, this means we are limited to using USB storage devices and luckily, vSAN can actually claim and consume USB storage devices. For a basic homelab, this is probably okay but if you want something a bit more reliable, you can look into using a USB 3.0 to M.2 NVMe chassis. The ability to use an M.2 NVMe device should definitely provide more resiliency compared to a typical USB stick you might have lying around. From a capacity point of view, I had two 32GB USB keys that I ended up using which should be plenty for a small setup but you can always look at purchasing large capacity given how cheap USB devices are.

Disclaimer: ESXi-Arm is a VMware Fling which means it is not a product and therefore it is not officially supported. Please do not use it in Production.

With the disclaimer out of the way, I think this is a fantastic use case for an inexpensive vSAN Witness which could be running at a ROBO/Edge location or simply supporting your homelab. The possibilities are certainly endless and I think this is where the ESXi-Arm team would love to hear whether this is something customers would even be interested in and please share your feedback to help with priorities for both the ESXi-Arm and vSAN team.

In my setup, I have two Intel NUC 9th Pro which make up my 2-Node vSAN Cluster and then an rPI as my vSAN Witness. Detailed instructions can be found below including a video for those wanting to see vSAN Witness in action by actually powering on an actual workload 😀