Correlating vSAN perf metrics from vSphere Web Client to both PowerCLI & vSAN Mgmt API

While going through the PowerCLI 6.5.1 release notes last week, in addition to the new Get-VsanView cmdlet which exposes the complete vSAN Management API through PowerCLI, I had learned that we had also released a new Get-VsanStat cmdlet. This new cmdlet allows customers to easily retrieve the various vSAN Performance Metrics provided by the vSAN Performance Service.


The really nice thing about vSAN Performance Service is that all vSAN stats are now available directly in the vSphere Web Client, where as before, this information was only available as part of the vSAN Observer which was a completely different interface. The other huge benefit to using the vSAN Performance Service is that it also stores the historical stats compared to the vSAN Observer which was primarily used for real time troubleshooting. As of vSAN 6.6, the vSAN Observer has been deprecated and will only be used by GSS in limited scenarios, everything you were able to do with the vSAN Observer is now possible with the vSAN Performance Service.

While trying out the new Get-VsanStat cmdlet, I had found it to be quite difficult to easily map the vSAN metrics I saw in the UI to the specific PowerCLI query required to extract that information. The documentation was also quite light and only included a single sample and although some of the metrics could easily be deduced, there were many others that I was just unsure of. I had also tried using the vSAN Management API directly, thinking that I might have more luck but it was also challenging to use for other reasons and I still ran into the same problem which was how do I easily map what I saw from the UI down to the API or even associating that back to PowerCLI.

After spending a few days with BOTH PowerCLI and the vSAN Management API and with a bit of frustration, I think I have finally figured out how to map what I saw from the UI back to the both CLI and API. This was not an easy task as and I had to cross-reference multiple datasources to build up this mapping and I thought I would put together a reference which outlines this mapping so that others would have to go through the same pain. IMHO, this should be a pretty straight forward task. In addition, I have also provided a PowerCLI sample for each of the metric types as well as the associated vSAN Management API mapping as those differ in name as well. This hopefully should make it easy for anyone to start using either of these interfaces for collecting vSAN metrics from an Automation standpoint. As part of this exercise, I also ran into a variety of bugs which I have already filed internally and all this information has been feedback to Engineering teams to hopefully improve both our CLI and API in future updates.

Continue reading

VMware Validated Design (VVD) & VMware Cloud Foundation (VCF) Quick References

The other day, I had a customer ask about the latest version of the Ports and Protocols document that was put together by the VMware Validated Design (VVD) team which does a fantastic job of outlining all the connectivity between the products used within the VVD SDDC. It actually took me awhile to find after realizing it was posted on the VMTN Community rather than the official documentation page.

I also came across other VVD content that I thought would also be useful for customers and decided to create a VVD "Quick Reference" that organizes all of this content into a single place. In addition, I also created a nice short URL to the quick reference so you only have to remember that. After sharing this on Twitter yesterday, Ryan Johnson (TMM for VVD/VCF) got inspired and also created VMware Cloud Foundation Quick Reference which I also built a short URL for. If there is other content that you would like to see, feel free to drop me or Ryan a comment or better yet, submit a pull request on the Github page!

To access the Quick References, use the following short URLs:

Below is a quick screenshot of both the VVD and VCF Quick References:

Quick Tip - Creating a multiline Dockerfile using heredoc w/variable substitution

I was helping out a fellow colleague yesterday who was having some troubles handling a multiline echo statement within his Dockerfile. There are multiple ways in which you can create multiline Dockerfiles, the web is full of examples from using multiple echo statements (pretty ugly) to using heredocs which is easier to read and manage. The challenge was that he also wanted to substitute some variables into his multiline statement which apparently there were no examples online, at least neither of us could find.

Taking a closer look, I found that we can just leverage Bash's ANSI-C Quoting syntax $'string' to do what we want, which was actually something new to me as well. You can then pass in the variable like you normally would between the strings and that would give you the readability of heredocs and still be able to use Docker variables. I am sure there are other methods with more extensive escapes with single-ticks, but I also prefer a solution that is easy to read and use in case others need to manage it.

Here is a quick sample Dockerfile which demonstrates how this works:

Basically the echo statement has $'SOME-STRING'$VARIABLE$'SOME-STRING'

If we build and run this Docker image, we can see that we have properly substituted the BASEURL variable into our file as seen in the screenshot below.

docker build -t sample .
docker run --rm -it sample cat /etc/yum.repos.d/powershell.repo


I personally prefer to keep such logic within a separate script which the Dockerfile can reference, but I was also sympathetic to that fact that my colleague wanted to keep things simple and just have everything within the Dockerfile. I figure I would share this in case other comes across this problem as well as benefiting myself as I will probably forget in a months time 🙂

ESXi Learnswitch – Enhancement to the ESXi MAC Learn DvFilter

The ESXi MAC Learn dvFilter Fling was released a little over two years ago and it has become a must have when it comes to running our ESXi Hypervisor within a VM, also referred to as Nested ESXi. The reason this Fling has become such a popular hit amongst our customers and partners is that it greatly improves the performance when “Promiscuous Mode” is enabled on a Virtual or Distributed Virtual Portgroup, which is a requirement for using Nested ESXi. Although this Fling works great, there are a couple of limitations with this solution today. The first of which is called out in the original Fling release notes, that once a MAC Address has been learned, it never ages out which is not ideal for long running Nested ESXi environments that generates a large amount of new MAC Addresses. The second, is the lack of vMotion support where the learned MAC Address table is not transfered to the destination ESXi host and must be re-learned.

To help address both of these limitations, the folks over in the Network and Security Business Unit (NSBU) have been working hard to improve upon the existing solution and have developed a new native MAC Learning VMkernel module called the Learnswitch. This new Learnswitch not only helps improves Nested ESXi workloads but it can also potentially benefit other workloads such as Nested Containers or other 3rd Party network inspection software. One immediate difference from the previous MAC Learn dvFilter solution is that rather than operating on the Network IO Chain, the filtering is now performed within the outer virtual switch layer itself which will provide some additional performance gains. The other added benefit from an internal VMware standpoint is that the Learnswitch is now vmkapi compatible, which means we will have a better backwards compatible story for supporting old releases of ESXi. One downside to this new solution compared to the previous one is that because the dvFilter operated below the virtual switch layer, it could support both a Virtual Standard Switch as well as the Distributed Virtual Switch. With the new Learnswitch, a Distributed Virtual Switch will be required. If you currently do not meet the requirements of the new Learnswitch, you can continue using the dvFilter, but it is recommended that you do not mix both on a single system but you can definitely make use of both solutions across different ESXi hosts depending on the constraints of your environment.

Here are some of the new capabilities provided by the new Learnswitch module:

  • Overlay Network based that learning and filtering are done in Etherswitch forwarding check
  • MAC Address learning is based on VLAN ID or VXLAN ID on uplink and leaf port
  • Packet is filtered on uplink and leaf port if the MAC is learned on a different port
  • MAC Address table size is 32k per system
  • MAC Address aging support with default aging time of 5 minutes and configurable
  • Unknown unicast packet is flooded by default and configurable to drop
  • vMotion support that the MAC table learned on the port is transferred to destination host and RARP packet is sent
  • Standalone VMkernel module available as a VIB
  • net-learnswitch CLI to display MAC Address table, configuration and stats

Continue reading

Managing & silencing vSAN Health Checks using PowerCLI

One neat capability that was introduced with vSAN 6.6 is the ability to "silence" and disable specific vSAN Health Checks. A recent use for this came up on Duncan's blog where the vMotion health check would fail if you are using a vMotion network stack. As you can imagine, this feature can also come in handy for vSphere Home Labs where your hardware may not be on the official VMware HCL and wish to disable those specific vSAN Health Checks.


Continue reading