Virtual SAN

Useful M.2 NVMe accessories for vSphere (VSAN/VMFS) Home Labs

10/01/2018 by William Lam 6 Comments

I recently acquired a new toy for the home lab thanks to Timo Sugliani who shared an article on Twitter a few weeks back for a new USB-based enclosure that supports an NVMe SSD device using the M.2 form factor.

Trying to see if I can get this new toy working 😁 pic.twitter.com/0o4jLng72M

— William Lam (@lamw) September 27, 2018

I was excited to give the accessory a try, especially as the M.2 devices are used regularly for vSphere home labs running on either the Intel NUCs or Supermicro E200-8D. Most of these platforms only support a single M.2 slot and this is an easy way to add additional high performance storage capacity with a small footprint. The other benefit with an external enclosure is that you now have a portable and reliable storage solution that can easily be moved from system to system, especially for those that have asked about running VMFS on USB-based device.

Leveraging Instant Clone in vSphere 6.7 for extremely fast Nested ESXi provisioning

05/17/2018 by William Lam 18 Comments

The idea of "Instant Cloning" a Nested ESXi VM (running ESXi in a VM) is not a new concept. In fact, I had shared a solution back in 2015 using the private VMFork APIs. However, what has changed is the ease of consumption, primarily due to the re-architecture of Instant Clone in vSphere 6.7 (more details here and here) which resulted in a public and simplified API. Some of you might ask, why not simply clone a Nested ESXi VM or create a Link Clone? What benefit would I get by using Instant Clone?

The answer is not only speed, but the fact that the instantiated VM is fully operational and ready to start executing where as a traditional full clone or linked clone requires a full OS boot up that can take up to several minutes to deploy and configure. This may not sound like much for a small number of Nested ESXi VMs, but as you increase the number of instances, Instant Clone really shines while still maintaining speed and the instant availability of the VM. As you can imagine, this definitely opens up for some interesting use cases whether it be for personal home lab or educational purposes like VMware HOL. In addition, we also have customers who deploy Nested ESXi not only at high scale but also with a high churn rate for development purposes, think CI/CD type of a workload who can also benefit from Instant Clone.

So how fast are we talking about? Lets say you wanted to test out the latest version of VSAN in vSphere 6.7, you would normally deploy 3 Nested ESXi VMs, power them up and wait for them to be ready on the network. With Instant Clone, you can deploy three fully functional Nested ESXi VMs in just 30seconds! As the VMs are instantly available for consumption, you can start the VSAN enablement workflow immediately and even parts of that can be baked into the Instant Clone workflow. With the ease of provisioning Nested ESXi VMs, you can simply maintain a catalog of ESXi templates which are in "frozen" states and then leverage Instant Clone to deploy just-in-time Nested ESXi environments and discard them once you are done. Pretty slick if you ask me! and something I plan on using going forward.

Disclaimer: Nested ESXi is still not officially supported by VMware. Please use at your own risk.

Translating vSAN VM Object IDs (UUID to VM and VM to UUID)

11/21/2017 by William Lam 2 Comments

I was working on one of my vSAN Clusters a few weeks back and I had noticed a bunch of vSAN Objects being listed under the "Other" category within the vSAN Virtual Objects Health view as shown in the screenshot below.

I could not figure out what files or VMs these vSAN objects were actually associated to and it was especially strange since all VMs that were deployed on my vSAN Cluster were already properly showing up under this view and I could not account for these "Other" vSAN Objects. I had reached out to a few folks to see if anyone knew how to identify these objects and the only suggestion I had received back was try to run this python vSAN Health Status script located on one of the ESXi hosts participating in the vSAN Cluster to see if it provided what I needed.

The script is located at /usr/lib/vmware/vsan/bin/vsan-health-status.pyc and you run it like the following:

python /usr/lib/vmware/vsan/bin/vsan-health-status.pyc > /tmp/output

The above command just runs the script and stores its output (which is quite extensive) to /tmp/output. Once the script finishes, you can then open up the file using vi and search for the specific vSAN Object UUID in question. I was able to eventually identify what these vSAN Object UUIDs were mapped to (more on this later), but the overall experience was not ideal and it required SSH access to ESXi host which most customers disable by default. In addition, the process was pretty manual and tedious if you wanted to check multiple vSAN Object UUIDs.

So what did I do, well I looked for a better way of course! It turns out the output produced by vsan-health-status.pyc is actually all available using the vSAN Management API. Not only can you obtain this information programmatically and remotely but you can also retrieve this information by simply going to vCenter Server rather than having to directly connect to an ESXI host which was huge negative for me regarding the previous solution.

How to convert vSAN RVC commands into PowerCLI and/or other vSphere SDKs?

06/27/2017 by William Lam 1 Comment

A common request that I see come up from our field and customers is getting specific vSAN Ruby vSphere Console (RVC) commands to be made more generally available in other vSphere CLI/SDKs like PowerCLI for example. Funny enough, many folks do not realize that this functionality has been there since vSAN 6.2 and specifically with the release of the vSAN Management APIs which exposes all vSAN functionality programmatically whether you are consuming it from the vSphere Web Client, Embedded Host Client or from RVC. All of these tools have been built using the vSAN Management APIs.

Although we have supported a variety of vSAN Management SDKs (language bindings) since its first release, I will say that PowerCLI consumption of the vSAN Management API has only been made available recently with PowerCLI 6.5.1 and it supports the latest release of vSAN 6.6 and can go all the way back to vSAN 6.2. Even with PowerCLI support, I still continue to see vSAN RVC requests come up time after time and it seems like folks still have not made the connection that RVC is just simply using the vSAN Management API just like UI does.

What is even more interesting is that the source code of RVC can be viewed by anyone to see how each command is implemented and which APIs are being used. RVC is built using rbvmomi (vSphere SDK for Ruby) which provides access to both the vSphere and vSAN Management APIs. Given the number of requests that I have seen, I am going to assume that this is not common knowledge and I figured the best way to show how this work is with a real world example. I decided to take the vsan.check_limits RVC command and create an equilvenet PowerCLI script that uses the vSAN Management API to provide the exact same information.

Note: You will need to know how to use the vSphere/vSAN Management APIs and knowing a little of Ruby can also help. If you are new to vSAN Management APIs, have a look at this blog post on how to get started.

Here is a screenshot of running the vsan.check_limits RVC command:

Here is a screenshot of running the PowerCLI script that I have created:

As you would expect, the data is exactly the same since they both consume the same underlying vSAN Management API.

So, how do we get started?

Managing & silencing vSAN Health Checks using PowerCLI

04/24/2017 by William Lam 3 Comments

One neat capability that was introduced with vSAN 6.6 is the ability to "silence" and disable specific vSAN Health Checks. A recent use for this came up on Duncan's blog where the vMotion health check would fail if you are using a vMotion network stack. As you can imagine, this feature can also come in handy for vSphere Home Labs where your hardware may not be on the official VMware HCL and wish to disable those specific vSAN Health Checks.

How to tell if an ESXi host is a VSAN Witness Virtual Appliance programmatically?

09/26/2016 by William Lam Leave a Comment

I had received this question awhile back but I was only able to get to it recently. If you are not familiar with the VSAN Witness Virtual Appliance and its purpose, Cormac Hogan did an excellent write-up on the topic which you can find it here.

The reason this question came up was that if you were to simply iterate over all ESXi hosts within your vSphere Inventory from an Automation standpoint, you might find a mix of regular ESXi hosts and potentially this new VSAN Witness Virtual Appliance which is basically an ESXi host that runs in a VM (e.g. Nested ESXi). Although, it may look and feel like a regular ESXi host, it is not and the question was how might you go about distinguishing between the two? You can of course setup specific naming standards, folder structure or separate datacenter objects, but you still may accidentally retrieve a VSAN Witness host without even realizing it.

One quick solution is to check for a specific ESXi Advanced Setting called Misc.vsanWitnessVirtualAppliance which will return a value of 1 if it is the VSAN Witness Appliance. Here is a quick PowerCLI snippet which demonstrates how you can access this property:

$vmhost = Get-VMHost -Name 192.168.1.115
Get-AdvancedSetting -Entity $vmhost -Name Misc.vsanWitnessVirtualAppliance

Although the method described above is one quick way to easily identify whether an ESXi host is a VSAN Witness Appliance, it is also limited in the information that it provides you. Another approach is to actually use the new VSAN 6.2 Management API and specifically the Stretched Clustering System APIs to retrieve the associated VSAN Witness host for a given VSAN Cluster. Not only will you get more information about the specific ESXi host providing the VSAN Witness functionality which will allow you to correlate back to your vSphere Inventory, but you will also get additional VSAN Witness configuration such as the preferred Fault Domain, Node UUID and the VSAN Cluster that it is associated with for example.

Here is a quick VSAN Management SDK for Python sample script that I had created called vsan-stretched-cluster-system-sample.py which implements the VSANVcGetWitnessHosts() API method. The script prints out a few of the WitnessHostInfo properties as shown in the screenshot below.

One other option is if you simply just want to know if a given ESXI host is a VSAN Witness host or not, there is also the VSANVcIsWitnessHost() API that simply returns a boolean value. This might useful if you just have a list of ESXi hosts retrieved through the vSphere API and no knowledge of the underlying VSAN Clusters.