VMware Cloud on AWS

Tanzu Kubernetes Grid (TKG) Demo Appliance 1.1.3

08/10/2020 by William Lam 1 Comment

It has been awhile since I have updated my Tanzu Kubernetes Grid (TKG) Demo Appliance Fling, which is a virtual appliance that enables anyone to go from zero to Kubernetes in less than 30 minutes with just an SSH client and a web browser. For VMware Cloud on AWS customers interested in running TKG, this is a great way to quickly get started on a proof of concept, demo or for development and testing purposes. One great benefit is that everything required for TKG is self contained within the appliance including an embedded Harbor registry and the respective TKG container images, great for air-gapped or non-internet accessible environments.

Here is a summary of what is new:

Support for latest TKG 1.1.3

There have been several of smaller releases to TKG since their 1.0.0 release but due to their short lifecycle, I decided to hold off. Behind the scenes, I have actually been working closely with TKG team on the latest TKG 1.1.3 release which was just release last week. One really cool feature that was introduced in TKG 1.1.2 is the ability to upgrade an existing TKG Workload Cluster to a newer version of Kubernetes.

With TKG 1.1.3, support for Kubernetes v1.18.6 and v1.17.9 is now possible and the latest version of the demo appliance will also support this workflow. In fact, I have also updated my TKG Workshop Guide to include all new updates including the upgrade workflow. To reduce the maintenance burden on myself, the TKG Demo Appliance 1.0.0 will be removed in the near future, for now it has been deprecated but all existing content is still available. I highly recommend checking out the latest version as you will get all the latest features of TKG.

Integrating VMware Cloud Notification Gateway with VMware Event Broker Appliance (VEBA)

07/29/2020 by William Lam Leave a Comment

I previously wrote about the VMware Cloud Notification Gateway (NGW) which provides curated notifications delivered to VMware Cloud on AWS users. By default, NGW supports several types of notification channels such as email, VMware Cloud Console UI, VMware Cloud Activity Log, vRealize Log Intelligence Cloud (vRLIC) and the vSphere UI when using the vCenter Cloud Gateway. A lesser known feature of NGW is the ability to extend into even more channels by leveraging its webhook functionality which is available when using NGW API.

For a basic "pass through" of the NGW notification to another cloud service such as Slack or Microsoft Teams as example, you can simple setup an incoming webhook on Slack or Microsoft Teams, which I had covered in the previous blog post. From there, you can configure an NGW subscription and forward the NGW notification to the specified incoming webhook.

For more interesting scenarios where customers may want to perform some additional data processing when the NGW notification arrives or run some code/automation and integrate that with other systems which can include your on-premises infrastructure, the basic webhook workflow is not sufficient. Having said that, at the end of the previous blog post I did hint at a solution that would enable customers to support such scenarios which is by leveraging the VMware Event Broker Appliance (VEBA) solution.

The way this works is that we are still taking advantage of the NGW webhook capability but instead of forwarding the NGW notification to a cloud service that supports an incoming webhook, we are sending it to VEBA for processing. Once the notification has been received by VEBA, customers can apply additional logic by using any language of their choice which runs as an automated function and is then responsible for sending the final payload to its destination. This is really the power of VEBA which enables customers to perform any additional processing or business logic to an event before sending it out to its intended target.

Using the new installation method for deploying OpenShift 4.5 on VMware Cloud on AWS

07/18/2020 by William Lam 1 Comment

I recently saw a tweet from Jason Shiplett who works over on the VMware Validated Design (VVD) team (also my former team before joining VMware Cloud) who shared a new validated design for running Redhat OpenShift 4.3 on VMware Cloud Foundation. Funny enough, a couple of days ago I was just researching into deploying OpenShift running on VMware Cloud on AWS from a customer inquiry.

Timing could not have been better as RedHat just announced their OpenShift 4.5 release a few days ago as and one of the major updates is support for vSphere using their full stack automation also known as te Installer Provisioned Infrastructure (IPI) option. Previous to this, customers who wanted to deploy OpenShift on vSphere had to use the User Provisioned Infrastructure (UPI) method, which the VVD design also uses, which is much lengthier and complex when compared to the native IPI method.

For someone who has never worked with OpenShift before, this was great news and I get to try out this new deployment method on an VMware Cloud on AWS infrastructure 🙂

Pre-Requisites:

Step 1 - You will need a Linux system to perform the installation and it should have access to the vCenter Server running in VMware Cloud on AWS (VMC). In my example, I am using an Ubuntu Server 20.04 VM which is also running in the SDDC and has outbound internet connectivity.

Step 2 - Login to VMware Cloud on AWS console and create a new NSX-T network segment that is DHCP enabled. In my example, I named it openshift-network with a 192.168.3.0/24 configuration.

Step 3 - Navigate to Inventory->Groups and create the following groups and replace the CIDR networks with that of your SDDC:

Group	Name	IP Address Members
Compute	OpenShift Network	192.168.3.0/24
Compute	SDDC Management Network	10.2.0.0/16
Management	OpenShift Network	192.168.3.0/24

Retrieving network statistics on VMware Cloud on AWS using NSX-T Policy API

07/16/2020 by William Lam Leave a Comment

One question that has come up lately from VMware Cloud on AWS customers is to understand their network traffic usage, especially as it pertains to traffic that exit or egress their SDDC. There are a number of graphical tools that can be used today to get insights into this information, one is the popular vRealize Network Insight Cloud solution which many of our VMware Cloud on AWS customers are taking advantage of to not only understand traffic usage and flow data history but is also instrumental in aiding customers when planning workload migrations from their on-premises datacenter to VMware Cloud on AWS.

While researching this topic, I also came to learn that this information can be retrieved using the NSX-T Policy API which is available to all customers to use. We are going to be leveraging the Tier-0 statistics interface API from NSX-T which will give us both transmit and receive stats on all supported interfaces. From the diagram below, we can see the interfaces that are applicable to VMware Cloud on AWS is the Internet interface which includes VPN traffic, VPC interface which includes traffic going to Linked VPC and Direct Connect interface which includes traffic when using AWS Direct Connect.

As you might expect, these exact same three interface types is then represented as logical interfaces within the NSX-T Policy API which uses the following IDs:

cross-vpc
public
direct-connect

Note: Statistics on the Direct Connect interface will also include traffic if you are using the new VMware Transit Connect with AWS Transit Gateway feature.

These interface can be discovered by performing a GET on /policy/api/v1/infra/tier-0s/vmc/locale-services/default/interfaces and then you would then identify the two NSX-T Edge (Active/Passive) and construct the T0 URL to retrieve the statistics. I will not bore you with the details and have implemented this as a new PowerShell function called Get-NSXTT0Stats and for those interested in the implementation, please see the code here.

Note: For those wanting to see the full NSX-T Policy REST URLs, simply append -Troubleshoot flag and that will output additional information on how I am retrieving the various pieces of information required to call into the T0 Stats API.

Custom notification and automation based on host failure in VMware Cloud on AWS

07/09/2020 by William Lam Leave a Comment

Physical hardware failure is inevitable, this is true whether it is running in your on-premises datacenter or in the Cloud like VMware Cloud on AWS. Although vSphere HA will automatically restart all affected VMs after detecting a host failure, there is usually additional activities that must be performed by a customer such as notifying all impacted application owners and even creating an incident ticket for hardware replacement.

With VMware Cloud on AWS, the hardware replacement is done automatically for you but the downstream activity of notifying application owners to verify the application is functional is still managed by the customer. There are many ways in how customers can manage such incidents and one solution that I am a huge advocate of is taking advantage of the powerful vCenter Server Events, which has over 1700+ events, not to mention any of the 2nd/3rd party events.

When an ESXi host fails, the com.vmware.vc.HA.DasHostFailedEvent event will be generated which contains all the relavent information related to the host failure including the specific hostname/IP, when the incident occurred and details about the vSphere Cluster and Datacenter is also provided. This information is visible using the vSphere UI but it can also be programmatically retrieved using the vSphere API, which is how the vSphere UI renders this information.

Note: Everything described in this blog post including the VEBA example is applicable to any environment that contains vCenter Server and is not limited to just VMware Cloud on AWS.

Extending VMware Cloud on AWS Notifications using the Notification Gateway API

06/10/2020 by William Lam 2 Comments

The VMware Cloud Notification Gateway (NGW) Service was launched back in May 2019 and is used to communicate important customer-facing notifications which can be delivered across a number of different communication channels as shown in the diagram below.

Of all the different communication channels, I think one of the most interesting one is the ability to send an outgoing webhook based on a specific VMware Cloud Event. In fact, this was the very first thing that caught my attention when I had first learned about the NGW Service from Nancy Cheng, the Product Manager for this service.

You can probably guess why I was so excited for this feature as it mimics a similiar capability to our VMware Event Broker Appliance (VEBA) solution. This not only enables our customers to consume other public cloud services that support webhooks but it also opens up the door for more advanced integrations, more on this at the end of this blog post 😀

As of publishing this blog post, there are over 75+ VMware Cloud Events which customers can subscribe to such including when a new SDDC is created or deleted, a new ESXi host has been added either manually or automatically via our Elastic DRS (eDRS) Service, SDDC maintenance notices to subscription reminders to just name a few. Although the default email and UI channels are great, many customers would also like to receive these notifications using other popular communications channels such as Slack or Microsoft Teams.

To help demonstrate the webhook functionality of the NGW Service API, I have created a PowerShell Module for VMware Cloud Notifications called VMware.VMC.Notification which is also published i then Microsoft Powershell Gallery. The module contains the following functions:

Connect-VmcNotification
Get-VmcNotificationEvent
Get-VmcNotificationWebhook
Test-VmcNotificationWebhook
Remove-VmcNotificationWebhook