William Lam

Extending VMware Cloud on AWS Notifications using the Notification Gateway API

06/10/2020 by William Lam 2 Comments

The VMware Cloud Notification Gateway (NGW) Service was launched back in May 2019 and is used to communicate important customer-facing notifications which can be delivered across a number of different communication channels as shown in the diagram below.

Of all the different communication channels, I think one of the most interesting one is the ability to send an outgoing webhook based on a specific VMware Cloud Event. In fact, this was the very first thing that caught my attention when I had first learned about the NGW Service from Nancy Cheng, the Product Manager for this service.

You can probably guess why I was so excited for this feature as it mimics a similiar capability to our VMware Event Broker Appliance (VEBA) solution. This not only enables our customers to consume other public cloud services that support webhooks but it also opens up the door for more advanced integrations, more on this at the end of this blog post 😀

As of publishing this blog post, there are over 75+ VMware Cloud Events which customers can subscribe to such including when a new SDDC is created or deleted, a new ESXi host has been added either manually or automatically via our Elastic DRS (eDRS) Service, SDDC maintenance notices to subscription reminders to just name a few. Although the default email and UI channels are great, many customers would also like to receive these notifications using other popular communications channels such as Slack or Microsoft Teams.

To help demonstrate the webhook functionality of the NGW Service API, I have created a PowerShell Module for VMware Cloud Notifications called VMware.VMC.Notification which is also published i then Microsoft Powershell Gallery. The module contains the following functions:

Connect-VmcNotification
Get-VmcNotificationEvent
Get-VmcNotificationWebhook
Test-VmcNotificationWebhook
Remove-VmcNotificationWebhook

Admin account for embedded Harbor registry in vSphere with Kubernetes

06/09/2020 by William Lam Leave a Comment

After setting up a vSphere with Kubernetes Cluster, customers have the option of enabling a built-in private container registry that can be used with the Supervisor Cluster. This private container registry uses the popular Opensource Harbor solution which is also a Cloud Native Computing Foundation (CNCF) project.

Although this is a convenient capability, one thing to be aware of is that the embedded Harbor registry is limited in functionality compared to a standalone Harbor deployment and this is by design. When logging into Harbor with your vCenter SSO user, you will be able to do perform basic operations such as pushing and pulling images from this registry. For customers that require additional functionality from Harbor, it is recommended that you setup an external Harbor instance which can also be used as a common registry for both the Supervisor Cluster as well any Tanzu Kubernetes Grid (TKG) Clusters that you may provision.

With that said, I have heard from a few folks who were interested in accessing the Harbor UI using the "admin" account, mostly from an exploration standpoint. The admin credentials for Harbor are dynamically generated each time the service is enabled and it is stored as a K8s secret within the Supervisor Cluster. This means the admin password is unique for each environment and the instructions below will show you how to obtain the credentials.

Disclaimer: This is not officially supported by VMware and the behaviors described below could change in the future without notice.

Interesting Kubernetes application demos

06/08/2020 by William Lam Leave a Comment

I am always on the lookout for cool and interesting demos to deploy, especially with some of the work I have been doing lately with vSphere with Kubernetes (K8s) and Tanzu Kubernetes Grid (TKG). I am sure many of you have probably seen the basic wordpress demos which seems to be the typical "Hello World" app for K8s and having something more compelling not only makes the demo more interesting but it can also help folks better understand how a modern applications can be built, deployed and run.

Below is a list of of the K8s demo applications that I have come across as part of my exploration and by no means is this an exhaustive list. I have been able to successfully deploy these applications running on the latest version of K8s (1.17 and 1.18) as I did come across other demos which did not work or I had issues setting up. If there are other K8s demos that folks have used, feel free to leave a comment and I will update the blog post after doing some basic testing.

For those of you who may not have a K8s environment and is running either vSphere 6.7 Update 3 or have access to a VMware Cloud on AWS SDDC, you can easily setup a TKG Cluster in under 30 minutes leveraging my TKG Demo Appliance Fling.

Setup custom login banner when logging into a vSphere with Kubernetes Cluster

05/20/2020 by William Lam Leave a Comment

While working on my PowerCLI module for enabling workload management for a vSphere with Kubernetes (K8s) Cluster, I came to discover a pretty cool feature that is only available when using the vSphere with K8s API to enable Workload Management on a vSphere Cluster.

As part of the enablement spec, there is a new property called login_banner. Taking a closer look, this property allows you to specify a custom message that would be displayed as part of the initial login to your vSphere with K8s Cluster using the vSphere kubectl plugin. This is similar to an SSH login banner which can be used to provide internal disclaimers and/or additional instructions for your end users.

Here is an example of what the login banner can look like. Yup, vSphere with K8s supports emojis or rather the terminal you are using to login can potentially render emojis 😀

The good news is that I have already added this feature into the new New-WorkloadManagement function and you can specify a message by adding the -LoginBanner parameter.

For those interested in rendering emojis within their banner, you can take a look at the following example and you can find the complete list of emoji unicodes here.

$LoginBanner = "

" + [char]::ConvertFromUtf32(0x1F973) + "vSphere with Kubernetes Cluster enabled by virtuallyGhetto " + [char]::ConvertFromUtf32(0x1F973) + "

"

$LoginBanner = "

" + [char]::ConvertFromUtf32(0x1F973) + "vSphere with Kubernetes Cluster enabled by virtuallyGhetto " + [char]::ConvertFromUtf32(0x1F973) + "

Workload Management PowerCLI Module for automating vSphere with Kubernetes

05/19/2020 by William Lam Leave a Comment

One of the last things on my to-do list after creating my Automated vSphere 7 and vSphere with Kubernetes Lab Deployment Script which is still the quickest and most reliable way to have a fully deployed and configured environment to try out vSphere with Kubernetes using Nested ESXi, was to also automate the enablement of Workload Management for a given vSphere Cluster.

There are two new vCenter Server REST APIs to be aware of as it pertains to vSphere with Kubernetes:

namespaces = Manages the lifecycle and access control to a vSphere Namespace
namespace-management = Despite the name, this refers to lifecycle and management of a Workload Management Cluster

I also have to mention that Vikas Shitole, who works on vCenter Server, has fantastic blog series covering various parts of the new vSphere with Kubernetes API along with Python examples if you want to dive further. Since Vikas has done a great job covering Python, I figure I will demonstrate how to consume these new vSphere with Kubernetes API using PowerCLI, which many of our customers use to automate.

I have created a new WorkloadManagement.psm1 PowerCLI module which includes following functions:

Get-WorkloadManagement
New-WorkloadManagement
Remove-WorkloadManagement

Below are the two steps required to get started with the Workload Management PowerCLI Module.

Step 1 - Download and import the WorkloadManagement PowerCLI Module by running the following command:

Import-Module ./VMware.WorkloadManagement.psm1

Step 2 - A connection to the vCenter REST API endpoint using the Connect-CisServer cmdlet is required for enabling and disabling Workload Management Cluster

Connect-CisServer -Server pacific-vcsa-2.cpbu.corp -User *protected email* -Password VMware1!

A connection to vCenter Server using Connect-VIServer cmdlet is only required if you wish to retrieve information about an existing Workload Management Cluster

Connect-VIServer -Server pacific-vcsa-2.cpbu.corp -User *protected email* -Password VMware1!

How to configure network proxy with Tanzu Kubernetes Grid (TKG)?

05/18/2020 by William Lam Leave a Comment

Network Proxies are commonly used by customers to provide connectivity from internal servers/services to access external networks like the Internet in a controlled and secured manner. While working on a recent network proxy enhancement for our VMware Event Broker Appliance (VEBA) Fling, I had setup a Squid server which is a popular network proxy solution.

I had noticed a couple of folks were asking about network proxy configuration for Standalone Tanzu Kubernetes Grid (TKG) and figure this might be interesting to explore, especially for my recently released TKG Demo Appliance Fling which enables folks to quickly go from zero to Kubernetes in just 30 minutes! I figured this would be another good opportunity to learn a bit more about TKG as well as Kubernetes (K8s) and I jokingly said to myself, how hard could this be!? 😉 Apparently it was not trivial and took a bit of trial/error to figure out the correct combination and below is the procedure that can be followed for both standard deployment of TKG as well as the TKG Demo Appliance Fling.

Proxy Setting configurations for TKG CLI

The TKG CLI uses KinD (Kubernetes in Docker) under the hood to setup the initial K8s bootstrap cluster to deploy the TKG Management Cluster. If you have not already downloaded KinD node image (registry.tkg.vmware.run/kind/node:v1.17.3_vmware.2) or if you need to go through a network proxy to do so, then the following instructions can be followed to make your Docker Client aware of a network proxy.

Here is an example of the error if Docker Client can not download the image:

# docker pull registry.tkg.vmware.run/kind/node:v1.17.3_vmware.2
Error response from daemon: Get https://registry.tkg.vmware.run/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

If you are not using a private container registry with TKG, then you also need to also ensure that the KinD Cluster can connect to your network proxy when it pulls down the required containers from the internet. Luckily, KinD can simply detect the network proxy settings of your operating system. You can either set the proxy using traditional environmental variables (http_proxy, https_proxy and no_proxy) during your use of TKG CLI or you can simply set it globally so you do not forget.

In my setup, TKG CLI is running in a Photon OS VM and global proxy settings are configured in /etc/sysconfig/proxy Proxy settings will vary across operating systems and you should check with the vendor documentation for specific instructions. The following command will set both HTTP and HTTPS proxy variables to use my proxy server and you will also want to make sure you whitelist all networks and addresses which you want to by-pass the proxy.

cat > /etc/sysconfig/proxy << EOF
PROXY_ENABLED="yes"
HTTP_PROXY="http://192.168.1.3:3128"
HTTPS_PROXY="http://192.168.1.3:3128"
NO_PROXY="localhost,192.168.1.0/24,192.168.2.0/24,registry.rainpole.io,10.2.224.4,.svc,100.64.0.0/13,100.96.0.0/11"
EOF

Note: If you are using the TKG Demo Appliance, you only need to configure the Photon OS global proxy settings. In my example, I have white listed my local 192.168.* addresses, registry.rainpole.io which is the embedded Harbor registry, 10.2.224.4 which is the internal IP Address of VMC vCenter Server, *.svc addresses which all the internal K8s services and 100.64.0.0/13 which is the CIDR range used by TKG for the Service networks and 100.96.0.0/11 which is the CIDR range used by TKG Cluster networks.