Automation

Tanzu Kubernetes Grid (TKG) Demo Appliance 1.1.3

08/10/2020 by William Lam 1 Comment

It has been awhile since I have updated my Tanzu Kubernetes Grid (TKG) Demo Appliance Fling, which is a virtual appliance that enables anyone to go from zero to Kubernetes in less than 30 minutes with just an SSH client and a web browser. For VMware Cloud on AWS customers interested in running TKG, this is a great way to quickly get started on a proof of concept, demo or for development and testing purposes. One great benefit is that everything required for TKG is self contained within the appliance including an embedded Harbor registry and the respective TKG container images, great for air-gapped or non-internet accessible environments.

Here is a summary of what is new:

Support for latest TKG 1.1.3

There have been several of smaller releases to TKG since their 1.0.0 release but due to their short lifecycle, I decided to hold off. Behind the scenes, I have actually been working closely with TKG team on the latest TKG 1.1.3 release which was just release last week. One really cool feature that was introduced in TKG 1.1.2 is the ability to upgrade an existing TKG Workload Cluster to a newer version of Kubernetes.

With TKG 1.1.3, support for Kubernetes v1.18.6 and v1.17.9 is now possible and the latest version of the demo appliance will also support this workflow. In fact, I have also updated my TKG Workshop Guide to include all new updates including the upgrade workflow. To reduce the maintenance burden on myself, the TKG Demo Appliance 1.0.0 will be removed in the near future, for now it has been deprecated but all existing content is still available. I highly recommend checking out the latest version as you will get all the latest features of TKG.

Integrating VMware Cloud Notification Gateway with VMware Event Broker Appliance (VEBA)

07/29/2020 by William Lam Leave a Comment

I previously wrote about the VMware Cloud Notification Gateway (NGW) which provides curated notifications delivered to VMware Cloud on AWS users. By default, NGW supports several types of notification channels such as email, VMware Cloud Console UI, VMware Cloud Activity Log, vRealize Log Intelligence Cloud (vRLIC) and the vSphere UI when using the vCenter Cloud Gateway. A lesser known feature of NGW is the ability to extend into even more channels by leveraging its webhook functionality which is available when using NGW API.

For a basic "pass through" of the NGW notification to another cloud service such as Slack or Microsoft Teams as example, you can simple setup an incoming webhook on Slack or Microsoft Teams, which I had covered in the previous blog post. From there, you can configure an NGW subscription and forward the NGW notification to the specified incoming webhook.

For more interesting scenarios where customers may want to perform some additional data processing when the NGW notification arrives or run some code/automation and integrate that with other systems which can include your on-premises infrastructure, the basic webhook workflow is not sufficient. Having said that, at the end of the previous blog post I did hint at a solution that would enable customers to support such scenarios which is by leveraging the VMware Event Broker Appliance (VEBA) solution.

The way this works is that we are still taking advantage of the NGW webhook capability but instead of forwarding the NGW notification to a cloud service that supports an incoming webhook, we are sending it to VEBA for processing. Once the notification has been received by VEBA, customers can apply additional logic by using any language of their choice which runs as an automated function and is then responsible for sending the final payload to its destination. This is really the power of VEBA which enables customers to perform any additional processing or business logic to an event before sending it out to its intended target.

Retrieving network statistics on VMware Cloud on AWS using NSX-T Policy API

07/16/2020 by William Lam Leave a Comment

One question that has come up lately from VMware Cloud on AWS customers is to understand their network traffic usage, especially as it pertains to traffic that exit or egress their SDDC. There are a number of graphical tools that can be used today to get insights into this information, one is the popular vRealize Network Insight Cloud solution which many of our VMware Cloud on AWS customers are taking advantage of to not only understand traffic usage and flow data history but is also instrumental in aiding customers when planning workload migrations from their on-premises datacenter to VMware Cloud on AWS.

While researching this topic, I also came to learn that this information can be retrieved using the NSX-T Policy API which is available to all customers to use. We are going to be leveraging the Tier-0 statistics interface API from NSX-T which will give us both transmit and receive stats on all supported interfaces. From the diagram below, we can see the interfaces that are applicable to VMware Cloud on AWS is the Internet interface which includes VPN traffic, VPC interface which includes traffic going to Linked VPC and Direct Connect interface which includes traffic when using AWS Direct Connect.

As you might expect, these exact same three interface types is then represented as logical interfaces within the NSX-T Policy API which uses the following IDs:

cross-vpc
public
direct-connect

Note: Statistics on the Direct Connect interface will also include traffic if you are using the new VMware Transit Connect with AWS Transit Gateway feature.

These interface can be discovered by performing a GET on /policy/api/v1/infra/tier-0s/vmc/locale-services/default/interfaces and then you would then identify the two NSX-T Edge (Active/Passive) and construct the T0 URL to retrieve the statistics. I will not bore you with the details and have implemented this as a new PowerShell function called Get-NSXTT0Stats and for those interested in the implementation, please see the code here.

Note: For those wanting to see the full NSX-T Policy REST URLs, simply append -Troubleshoot flag and that will output additional information on how I am retrieving the various pieces of information required to call into the T0 Stats API.

Custom notification and automation based on host failure in VMware Cloud on AWS

07/09/2020 by William Lam Leave a Comment

Physical hardware failure is inevitable, this is true whether it is running in your on-premises datacenter or in the Cloud like VMware Cloud on AWS. Although vSphere HA will automatically restart all affected VMs after detecting a host failure, there is usually additional activities that must be performed by a customer such as notifying all impacted application owners and even creating an incident ticket for hardware replacement.

With VMware Cloud on AWS, the hardware replacement is done automatically for you but the downstream activity of notifying application owners to verify the application is functional is still managed by the customer. There are many ways in how customers can manage such incidents and one solution that I am a huge advocate of is taking advantage of the powerful vCenter Server Events, which has over 1700+ events, not to mention any of the 2nd/3rd party events.

When an ESXi host fails, the com.vmware.vc.HA.DasHostFailedEvent event will be generated which contains all the relavent information related to the host failure including the specific hostname/IP, when the incident occurred and details about the vSphere Cluster and Datacenter is also provided. This information is visible using the vSphere UI but it can also be programmatically retrieved using the vSphere API, which is how the vSphere UI renders this information.

Note: Everything described in this blog post including the VEBA example is applicable to any environment that contains vCenter Server and is not limited to just VMware Cloud on AWS.

Configuring dnsmasq as PXE Server for ESXi

07/09/2020 by William Lam Leave a Comment

One really cool thing that I came to learn while setting up the infrastructure to network boot the latest Raspberry Pi 4 was the use of dnsmasq, which I have used in the past but I did not realize it could do so much more. In addition to providing DNS services, it can also be configured to run TFTP and provide DHCP capabilities which can then be used to support PXE installations.

Another neat feature of dnsmasq is ability to proxy to an existing DHCP server which is extremely useful for anyone with an existing DHCP infrastructure. Given the simplicity of dnsmasq and having already set this up for the rPI, I figure it would also be useful to take folks through in setting up dnsmasq to also support ESXi installations over PXE, since this still comes up from new folks just getting started with ESXi kickstart automation.

For more details about PXE installation of ESXi, I highly recommend this whitepaper and although it states 6.0, the concepts and configurations are still applicable to the latest ESXi 7.0 release.

Two methods to network boot Raspberry Pi 4

07/07/2020 by William Lam 3 Comments

My Raspberry Pi 4 (8GB) model just came last week and after completing my RADIO (VMware's R&D Innovation Offsite) session recording, I wanted to setup my new rPI so I can start playing with it when I had some spare time. I also have the 4GB model but it was running quite hot as I was using the default case (do not recommend) and decided to put that aside for now. I ended up purchasing the 8GB model from Canakit which includes additional heatsinks and nice built-in fan with their custom case.

Look who just arrived to join the rest of 🥧 family! This will be my reward for tomorrow after I finish my #RADIO session recording pic.twitter.com/h9kxWRpM8S

— William Lam (@lamw) July 1, 2020

After spending some time setting up the rPI OS and applying the various updates, I was not too keen on using the SD Card, especially as some folks on forums mentioned that it can easily be worn out depending on the type of workload. While researching online and some references pointed out by colleagues, I came to learn that in addition to booting from USB which is now possible with rPI, you can also network boot the rPI without an SD Card or any storage for that matter! This immediately resonated with me, especially as I am a huge fan of scripted installations such as Kickstart/Jumpstart.

Here are all the resources that I had used that provided all the heavy lifting. I have optimized some of the commands to make it easy for anyone to simply copy and past who is new to setting up an rPI. The authors below did a fantastic job in outlining all the details, so I highly recommend a read if you would like to get more information. I also will be going over an alternative method at the end for booting the rPI over the network which is not really talked about.

One really cool thing that I came to learn while setting up the infrastructure to network boot an rPI was the use of dnsmasq, which I have used in the past but I did not realize it could do so much more. I may have spent more time playing with dnsmasq than with the rPI itself and I will probably cover this in another blog post on how you can easily setup a PXE/gPXE/iPXE system to enable automated OS installation (e.g. Kickstart) that can be used with ESXi or any other OS for that support network installations via BIOS/UEFI.