witness

vSAN Witness using Raspberry Pi 4 & ESXi-Arm Fling

10/08/2020 by William Lam 29 Comments

As hinted in my earlier blog post, you can indeed setup a vSAN Witness using the ESXi-Arm Fling running on a Raspberry Pi (rPI) 4b (8GB) model. In fact, you can even setup a standard 2-Node or 3-Node vSAN Cluster using the exact same technique. For those familiar with vSAN and the vSAN Witness, we will need to have at least two storage devices for the caching and capacity tier.

For the rPI, this means we are limited to using USB storage devices and luckily, vSAN can actually claim and consume USB storage devices. For a basic homelab, this is probably okay but if you want something a bit more reliable, you can look into using a USB 3.0 to M.2 NVMe chassis. The ability to use an M.2 NVMe device should definitely provide more resiliency compared to a typical USB stick you might have lying around. From a capacity point of view, I had two 32GB USB keys that I ended up using which should be plenty for a small setup but you can always look at purchasing large capacity given how cheap USB devices are.

Disclaimer: ESXi-Arm is a VMware Fling which means it is not a product and therefore it is not officially supported. Please do not use it in Production.

With the disclaimer out of the way, I think this is a fantastic use case for an inexpensive vSAN Witness which could be running at a ROBO/Edge location or simply supporting your homelab. The possibilities are certainly endless and I think this is where the ESXi-Arm team would love to hear whether this is something customers would even be interested in and please share your feedback to help with priorities for both the ESXi-Arm and vSAN team.

In my setup, I have two Intel NUC 9th Pro which make up my 2-Node vSAN Cluster and then an rPI as my vSAN Witness. Detailed instructions can be found below including a video for those wanting to see vSAN Witness in action by actually powering on an actual workload 😀

ESXi on Arm Fling is LIVE!

10/06/2020 by William Lam 11 Comments

The highly anticipated ESXi on Arm Fling has just been announced and is NOW generally available as a new VMware Fling! Head over to https://flings.vmware.com/esxi-arm-edition and be sure to carefully read through the Requirements and documentation before try out the bits.

History

Although ESXi-Arm was publicly demo'ed at VMworld Europe 2018 during the closing keynote by Ray O'Farrell (former CTO of VMware), the reality was there was a ton more to do before ESXi-Arm could be a reality for VMware customers. The newly formed ESXi-Arm team at VMware has been hard at work these last couple of years working with both Arm and its eco-system in extending hardware standards, firmware standards (open contribution to UEFI), and certification beyond the existing Arm server ecosystem, which enabled us to support platforms like SmartNICs and the ubiquitous Raspberry Pi. This is just a glimpse into what it took to get where we are at today.

I am also excited to share that the Virtually Speaking Podcast crew has invited us back for an exclusive episode featuring both Andrei Warkentin and myself to dive deeper into the development of ESXi-Arm project at VMware. This is an episode you will not want to miss!

Hardware

The ESXi-Arm Fling supports a number of different Arm platforms ranging from a traditional Datacenter form-factor to both Near and Far Edge systems including the highly requested Raspberry Pi (rPI)! With the rPI, only the 4b model will be supported and although both the 4GB and 8GB memory model works with ESXi-Arm. We highly recommend folks invest in the 8GB model to be able to take advantage of more vSphere features and be able to run more workloads.

For a complete list of supported Arm hardware platforms, please refer to the Requirements section of the Fling website. If there are other platforms you would like to see get added, do not hesitate to either leave a comment here and/or post directly on the ESXi-Arm Fling page.

vCenter Support

For customers with an existing x86 vCenter Server or those that would like to deploy a new vCenter Server, you will be able to attach and manage ESXi-Arm hosts just like you normally would as long as you are using vCenter Server 7.0 or greater.

We expect the majority of vSphere platform features to "just work" like vMotion but there may be some features that may not work or have additional requirements.

For example, to enable vSphere HA and/or vSphere FT, the Fault Domain Manager (FDM) Client VIB must be installed on an ESXi-Arm host. Today, this VIB is distributed as part of vCenter Server and only x86 version of the client is available. We do provide FDM Client VIBs for ESXi-Arm as part of the ESXi-Arm Fling, but support will be limited to vCenter Server 7.0c and 7.0d. For detailed instructions, please refer to the ESXi-Arm documentation.

VMware Tools

VMware Tools for ESXi-Arm GuestOS is not bundled as part of ESXi-Arm Fling, but can be installed. To do so, you will need to compile open-vm-tools for your respective GuestOS. Instructions can be found in the ESXi-Arm Fling documentation and below, you can see a screenshot of VMware Tools for Arm successfully running on Ubuntu 20.04 GuestOS running on ESXi-Arm on the rPI 4.

vSAN Witness

Lastly, a popular use case that has been brought up when ESXi-Arm was initially demo'ed was the use of the rPI as an inexpensive vSAN Witness, which is a fantastic use case for ROBO & Edge locations. I am very happy to share that using an rPI 8GB as a vSAN Witness works! As you can see from the screenshot below, I have two physical Intel NUC 9th Pro configured in a 2-Node vSAN Cluster and I am using the rPI as vSAN Witness 😀

In case this was not clear, this is NOT officially supported but it does demonstrate the viability of this concept and and feedback from the our users would help drive the priority and the potential support for such a configuration. More details will be shared in a future blog post outlining the instructions on using rPI as vSAN Witness. Stay tuned!

As you can see, this is just a small taste of what can be done with the ESXi-Arm Fling and the possibilities are truly endless! The ESXi-Arm team is very excited to see what the community will do with the ESXi-Arm Fling, what type of use cases are you solving or workloads that you are running. Below are a few ways in how you can engage with the ESXi-Arm team and community.

ESXi-Arm Engagement

For general questions/issues, please leave a Comment and/or file Bug on the ESXi-Arm Fling site
Follow ESXi-Arm on Twitter: @esxi_arm
Follow the official ESXi-Arm Blog: https://blogs.vmware.com/arm
Chat with the ESXi-Arm team and community on Slack: #esxi-arm-fling on VMware {code}
For other inquiries or engaging with ESXi on ARM Product Team, you can send an email to esxionarm [at] vmware [dot] com

How to tell if an ESXi host is a VSAN Witness Virtual Appliance programmatically?

09/26/2016 by William Lam Leave a Comment

I had received this question awhile back but I was only able to get to it recently. If you are not familiar with the VSAN Witness Virtual Appliance and its purpose, Cormac Hogan did an excellent write-up on the topic which you can find it here.

The reason this question came up was that if you were to simply iterate over all ESXi hosts within your vSphere Inventory from an Automation standpoint, you might find a mix of regular ESXi hosts and potentially this new VSAN Witness Virtual Appliance which is basically an ESXi host that runs in a VM (e.g. Nested ESXi). Although, it may look and feel like a regular ESXi host, it is not and the question was how might you go about distinguishing between the two? You can of course setup specific naming standards, folder structure or separate datacenter objects, but you still may accidentally retrieve a VSAN Witness host without even realizing it.

One quick solution is to check for a specific ESXi Advanced Setting called Misc.vsanWitnessVirtualAppliance which will return a value of 1 if it is the VSAN Witness Appliance. Here is a quick PowerCLI snippet which demonstrates how you can access this property:

$vmhost = Get-VMHost -Name 192.168.1.115
Get-AdvancedSetting -Entity $vmhost -Name Misc.vsanWitnessVirtualAppliance

Although the method described above is one quick way to easily identify whether an ESXi host is a VSAN Witness Appliance, it is also limited in the information that it provides you. Another approach is to actually use the new VSAN 6.2 Management API and specifically the Stretched Clustering System APIs to retrieve the associated VSAN Witness host for a given VSAN Cluster. Not only will you get more information about the specific ESXi host providing the VSAN Witness functionality which will allow you to correlate back to your vSphere Inventory, but you will also get additional VSAN Witness configuration such as the preferred Fault Domain, Node UUID and the VSAN Cluster that it is associated with for example.

Here is a quick VSAN Management SDK for Python sample script that I had created called vsan-stretched-cluster-system-sample.py which implements the VSANVcGetWitnessHosts() API method. The script prints out a few of the WitnessHostInfo properties as shown in the screenshot below.

One other option is if you simply just want to know if a given ESXI host is a VSAN Witness host or not, there is also the VSANVcIsWitnessHost() API that simply returns a boolean value. This might useful if you just have a list of ESXi hosts retrieved through the vSphere API and no knowledge of the underlying VSAN Clusters.

How to deploy and run the VSAN 6.1 Witness Virtual Appliance on VMware Fusion & Workstation?

09/11/2015 by William Lam 17 Comments

One of the most exciting new feature in VSAN 6.1 is the new Stretched Clustering capability which also provides support for a 2-Node ROBO deployment. If you are interested in learning more about the new VSAN 6.1 capabilities, be sure to check out Duncan's blog post here as well as a video on how to configure the new VSAN Stretched Clustering here. Like many of you, I am sure you are looking forward to giving both vSphere 6.0 Update 1 as well as the new VSAN 6.1 capabilities a spin in your home lab or development environment. By now, you probably know how easy it is to run Nested ESXi on top of your existing vSphere environment. However, not everyone has access to a vSphere environment. The next best thing is using VMware Fusion and Workstation which also supports Nested ESXi and for many of our customers and field, it is a great solution as it allows you to easily play with all the VMware goodies while you are on the go, especially useful if you travel frequently.

I was interested in setting up a 2-Node VSAN configuration and as part of the setup, you will also need to deploy the new VSAN Witness Virtual Appliance. I wanted to see what it would take to deploy the VSAN Witness Appliance onto VMware Fusion and Workstation and after a bit of exploration, hair pulling and OVF hackery, I was able to finally work out the process as shown in the steps below.

Disclaimer: This is not officially supported by VMware, please use this only for evaluational and testing purposes.

Note: If you plan on deploying the VSAN Witness Appliance directly to an ESXi host, you can use the injectOvfEnv method shown here.

Step 1 - Download the VSAN 6.1 Witness Virtual Appliance OVA from here

Step 2 - We need to make a few minor adjustments to the OVF file before we can import it into VMware Fusion/Workstation. Before we do so, we need to first convert the OVA to an OVF using ovftool.

Here is an example of running this on Mac OS X system:

/Applications/VMware\ OVF\ Tool/ovftool VMware-VirtualSAN-Witness-6.0.0.update01-3029758.ova VMware-VirtualSAN-Witness-6.0.0.update01-3029758.ovf

Once the conversion has completed, you should see a total of 8 files (6 VMDK files, 1 manifest & 1 OVF file). Before moving onto the next step, you will need to delete the manifest file (extension ending in .mf). We need to do this because the checksum will no longer be valid after we edit the OVF and the upload will fail.

Step 3 - The first edit that we need to make in the OVF is to change the required OVF parameter for specifying the password to the VSAN Witness Appliance from "true" to "false" since both Fusion/Workstation do not support OVF properties.

Change the following from

Step 4 - The final edit that we need to make in the OVF is to specify the deployment size (tiny, medium or large) for the Witness VM. By default, it will use the medium option. Below is a quick table of the deployment size and for evaluation/testing purposes, I suspect most you will want to stick with the "tiny" configuration.

Deployment Size	vCPU	vMEM	vDISKs	Components
tiny	2	8GB	1x8GB 1x15GB 1x10GB	750
medium	2	16GB	1x8GB 1x350GB 1x10GB	22K
large	2	32GB	1x8GB 1x350GB 1x10GB	45K

The way this is specified in the OVF is by adding ovf:default="true" to the specific deployment size. As mentioned, by default this is set to the "medium" deployment size which looks like the following:

If you wish to change this to some other deployment size, you will need to move the ovf:default="true" entry to the deployment size you wish to use. In our case, we will move it to "tiny" size by adding the following

Step 5 - Now that we are done with the OVF surgery, we can now import our VSAN Witness OVF into VMware Fusion or Workstation using the "Import" option. Ensure you select "Customize" option after the import has completed to prevent the VM from automatically powering on. This is very important because we still need to add one final configuration to the VMX file for the appliance to be properly configured.

Step 6 - The last and final step is to add a VMX entry which will configure the root password for VSAN Witness Appliance. This is not necessary when deploying to vCenter Server which has OVF support, but is a problem when deploying VMware Fusion, Workstation and even ESXi. You will need to add the following entry to the VMX file and be sure to replace the password of your choice which is highlighted below in blue.

guestinfo.ovfEnv = "<?xml version='1.0' encoding='UTF-8'?><Environment xmlns='http://schemas.dmtf.org/ovf/environment/1' xmlns:oe='http://schemas.dmtf.org/ovf/environment/1'><PropertySection><Property oe:key='vsan.witness.root.passwd' oe:value='vmware123'/></PropertySection></Environment>"

Step 7 - You are now ready to power on your VSAN Witness and if everything was configured correctly, you should be able to login to DCUI of the VSAN Witness which as you probably have guessed by now is running Nested ESXi 🙂

Here is a quick screenshot of configuring my very first 2-Node / Stretched VSAN Cluster which is comprised of 2 Nested ESXi VMs running on my Mac Mini which is running vSphere 6.0 Update 1 and my VSAN Witness Appliance running on my iMac using VMware Fusion. If you need additional instructions on configuring either a 2-Node / Stretched VSAN Cluster, be sure to check out the how-to video here. I plan to share this feedback internally with Engineering and hopefully deploying VSAN Witness in the future can be much more easier when it comes to running it on VMware Fusion and Workstation.