ESXi-Arm

Packer reference for PhotonOS Arm NFS Virtual Appliance using OVF properties for ESXi-Arm

10/21/2020 by William Lam 4 Comments

In case the title was not descriptive enough, I was curious if I could build an Arm Virtual Appliance (OVA) using OVF properties that would allow for all sorts of interesting guest customizations which I have blogged about before here, here, here and here using x86 PhotonOS as a reference implementation. My idea for this actually pre-dated the release of the ESXi on Arm Fling, but it was only until recently with support for VMware Tools for Photon OS Arm, was I able to finally piece together this solution.

It was also neat to see that I could build an Arm OVA using x86 tooling (Packer and OVFTool) which ran on my desktop and you simply needed an ESXi-Arm host. This really goes to show the level of compatibility from a management and vSphere API point of view that an ESXi-Arm host behaves just like a standard x86 ESXi host!

and successfully deployed maybe the “1st” OVA on #ESXionARM?

Just confirmed all guest customization via OVF properties executed correctly! Will be publishing reference Packer image in case you wish to build your own pic.twitter.com/PiSpceXtFF

— William Lam (@lamw) October 15, 2020

To demonstrate a more interesting use case than just basic network customization for the ?first? Arm OVA, I thought it would be useful to setup a simple NFS appliance that would take input from the user such the size of the exported volume (default 60GB) and then the name of the mount point. Upon first boot up, there is a guest customization script that would read in the OVF properties and configure the networking, OS password and NFS server configuration which you can certainly use to host your Arm VMs.

Kubernetes on ESXi-Arm using k3s

10/16/2020 by William Lam 11 Comments

The tiny form factor of a Raspberry Pi (rPI) is a fantastic hardware platform to start playing with the ESXi-Arm Fling. You can already do a bunch of fun VMware things like running a lightweight vSAN Witness Node to setting up basic automation environment for PowerCLI, Terraform and Packer to running rPI OS as VM, enabling some neat use cases like consolidating your physical rPI assets which might be running RetroPi and Pi-Hole which many home labbers are doing.

In addition to VMware solutions, its is also a great platform to learn and tinker with new technologies like Kubernetes (K8s) which I am sure many of you have been hearing about 🙂 Although our vSphere with Tanzu and Tanzu Kubernetes Grid (TKG) does not currently work with the ESXi-Arm Fling, I have actually been meaning to try out a super lightweight K8s distribution designed for IoT/Edge called k3s (pronounced k-3-s) which also recently joined Cloud Native Computing Foundation (CNCF) Sandbox level.

k3s is supported on rPI and you normally would have multiple rPI devices to represent the number of nodes, for example if you want a basic 3-Node cluster, you would need three physical rPI devices. With ESXi-Arm, you can now create these nodes as VM, using just a single rPI. This opens up the door for all sorts of explorations, you can create HA cluster or try out more advanced features which might be more difficult if you needed several physical devices. If you mess up, you can simply re-deploy the VM without much pain or simply clone the VM.

In my setup, I am using 3 x Photon OS VMs. One for the primary node and two for k3s worker nodes. You can certainly install k3s on any other Arm-based OS including rPI OS (which can now run as a VM as mentioned earlier).

Installing VMware Tools on Raspberry Pi OS for ESXi-Arm

10/15/2020 by William Lam 4 Comments

Now that you have Raspberry Pi (rPI) OS running as a VM on ESXi-Arm, the next thing you will probably want to install is VMware Tools, especially useful to see the IP Address of your Guest if you are using DHCP and for enabling guest "soft" shutdown using the vSphere UI. Below are the instructions

How to copy the Raspberry Pi UEFI files and boot ESXi-Arm using just USB device?

10/14/2020 by William Lam 2 Comments

There are actually a number of ways to boot and install the ESXi-Arm Fling, but the easiest method is already outlined in the official ESXi-Arm Raspberry Pi (rPI) documentation and the PDF can be downloaded from the Fling website. As a quick refresher, you only need to have two storage devices.

SD Card - Used to store the rPI UEFI files which is required to boot ESXi-Arm Installer
- ESXi-Arm can not run on SD Card and for these reasons, you do not need a large capacity SD Card
USB Device - Contains ESXi-Arm Installer / Installation
- After the ESXi-Arm installer boots, you can actually re-use the exact same USB device for the installation of ESXi-Arm itself. A separate USB device is not required unless that is your goal or if the capacity is not enough for running VMs

The boot process after the ESXi-Arm installation is that the UEFI firmware will first load on the rPI and then it will boot up the ESXi-Arm from the USB device. As mentioned, there are other variations but this is the most basic option. The other nice behavior is that if you need to re-install ESXi-Arm, you simply create a bootable USB device with the ESXi-Arm installer and then install that right on the same USB device without having to mess with UEFI image. This also allows you to perform scripted installation also known as Kickstart, which is something I will be covering in the future that takes UEFI image into consideration.

I have seen a few questions asked whether it is possible to have everything run off of the SD Card and/or USB Device and the answer is yes to certain degree.

It is possible to put the ESXi-Arm installer + UEFI on SD Card but ESXi-Arm will NOT be able to use it as installation media, so there is not a whole ton of value there.
It is possible to have both the UEFI image and ESXi Installation on the same USB device, especially if you do not have spare SD Cards which apparently has come up a few times

In this blog post, I will outline the instructions for booting an installed ESXi-Arm installation completely off of the USB device without the needing an SD Card containing the UEFI image.

How to run Raspberry Pi OS as a VM on ESXi-Arm

10/13/2020 by William Lam 48 Comments

It has only been a week since the ESXi-Arm Fling was released, but the amount of experimentation and frankly cool s*** that people have been able to do in such a short period of time has been pretty mind blowing. For example, did you know you could run ESXi-Arm on Nintendo Switch!?🤯

Andrei recently published a blog post on the official ESXi-Arm blog showcasing some of the really cool stuff the community has shared on Twitter, definitely worth checking digest post #1 it out if you have not seen it.

Something that really caught my eye which I did not see mentioned in Andre's blog post was from Twitter user Joakim Korhonen who shared that he was able to run Raspberry Pi's (rPI) OS (formally Raspbian OS) as a Virtual Machine running on top of the ESXi-Arm Fling!

running raspberry pi os on esxi on raspberry pi. nice.
needs uefi grub and debian kernel#raspberrypi #esxionarm pic.twitter.com/QcOxMAiSuC

— Joakim Korhonen (@korhojoa) October 8, 2020

This is pretty interesting because rPI OS was designed to run on a physical rPI and there are no installers other than the image file which you download and copy onto the SD Card to boot. What is really exciting about this news is that you can now run any of the popular rPI applications such as RetroPi or Pi-hole which traditionally may have required several rPI to host.

In addition, this can also benefit the rPI OS development community by making it easier to build and test applications on top of rPI OS as you can now spin these up as VMs and get all the benefits of vSphere and ESXi such as snapshots, cloning, etc. The possibilities are endless and wanted to give a huge thanks to Joakim for sharing his hack on getting this to work on ESXi-Arm. For those interested, I have documented the detailed instructions below.

UPDATE (11/106/20) - For those familiar with VMware Virtual Appliances (OVA) and using custom OVF properties for guest/application customization, be sure to check out this complimentary blog post on how to build your own rPI OS OVA that can allow you to easily deploy additional rPI OS VMs with ease, especially useful for testing and development purposes.

Workaround for ESXi-Arm in vSphere 7.0 Update 1

10/12/2020 by William Lam 4 Comments

In vSphere 7.0 Update 1, a new capability was introduced called the vCenter Cluster Services (vCLS) which provides a new framework for decoupling and managing distributing control plane services for vSphere. To learn more, I highly recommend the detailed blog post linked above by Niels. In addition, Duncan also has a great blog post about common question/answers and considerations for vCLS, which is definitely worth a read as well.

vSphere DRS is one of the vSphere features which relies on this new vCLS service and this is made possible by the vCLS VMs which are deployed automatically when it detects there are ESXi hosts within a vSphere Cluster (regardless if vSphere DRS is enabled or not). For customers who may be using the ESXi-Arm Fling with a vSphere 7.0 Update 1 environment, you may have noticed continuous "Delete File" tasks within vCenter that seems to loop forever.

This occurs because the vCLS service will first test to see if it can upload a file to the datastore, once it can, it will delete it. The issue is that the vCLS VMs are x86 and can not be deployed to an ESXi-Arm Cluster as the CPU architecture is not supported. There is a workaround to disable vCLS for the ESXi-Arm Cluster, which I will go into shortly. However, because vCLS can not properly deploy, it means vSphere DRS capabilities will not be possible when using vSphere 7.0 Update 1 with ESXi-Arm hosts. If this is desirable, it is recommended that to use either vSphere 7.0c or vSphere 7.0d if you wish to use vSphere DRS.

Note: vSAN does not rely on vCLS to function but to be able to use it, you must place your ESXi-Arm hosts into a vSphere Cluster and hence applying this workaround would be desirable for that use case.