Kubernetes

Setup custom login banner when logging into a vSphere with Kubernetes Cluster

05/20/2020 by William Lam Leave a Comment

While working on my PowerCLI module for enabling workload management for a vSphere with Kubernetes (K8s) Cluster, I came to discover a pretty cool feature that is only available when using the vSphere with K8s API to enable Workload Management on a vSphere Cluster.

As part of the enablement spec, there is a new property called login_banner. Taking a closer look, this property allows you to specify a custom message that would be displayed as part of the initial login to your vSphere with K8s Cluster using the vSphere kubectl plugin. This is similar to an SSH login banner which can be used to provide internal disclaimers and/or additional instructions for your end users.

Here is an example of what the login banner can look like. Yup, vSphere with K8s supports emojis or rather the terminal you are using to login can potentially render emojis 😀

The good news is that I have already added this feature into the new New-WorkloadManagement function and you can specify a message by adding the -LoginBanner parameter.

For those interested in rendering emojis within their banner, you can take a look at the following example and you can find the complete list of emoji unicodes here.

$LoginBanner = "

" + [char]::ConvertFromUtf32(0x1F973) + "vSphere with Kubernetes Cluster enabled by virtuallyGhetto " + [char]::ConvertFromUtf32(0x1F973) + "

"

$LoginBanner = "

" + [char]::ConvertFromUtf32(0x1F973) + "vSphere with Kubernetes Cluster enabled by virtuallyGhetto " + [char]::ConvertFromUtf32(0x1F973) + "

How to configure network proxy with Tanzu Kubernetes Grid (TKG)?

05/18/2020 by William Lam Leave a Comment

Network Proxies are commonly used by customers to provide connectivity from internal servers/services to access external networks like the Internet in a controlled and secured manner. While working on a recent network proxy enhancement for our VMware Event Broker Appliance (VEBA) Fling, I had setup a Squid server which is a popular network proxy solution.

I had noticed a couple of folks were asking about network proxy configuration for Standalone Tanzu Kubernetes Grid (TKG) and figure this might be interesting to explore, especially for my recently released TKG Demo Appliance Fling which enables folks to quickly go from zero to Kubernetes in just 30 minutes! I figured this would be another good opportunity to learn a bit more about TKG as well as Kubernetes (K8s) and I jokingly said to myself, how hard could this be!? 😉 Apparently it was not trivial and took a bit of trial/error to figure out the correct combination and below is the procedure that can be followed for both standard deployment of TKG as well as the TKG Demo Appliance Fling.

Proxy Setting configurations for TKG CLI

The TKG CLI uses KinD (Kubernetes in Docker) under the hood to setup the initial K8s bootstrap cluster to deploy the TKG Management Cluster. If you have not already downloaded KinD node image (registry.tkg.vmware.run/kind/node:v1.17.3_vmware.2) or if you need to go through a network proxy to do so, then the following instructions can be followed to make your Docker Client aware of a network proxy.

Here is an example of the error if Docker Client can not download the image:

# docker pull registry.tkg.vmware.run/kind/node:v1.17.3_vmware.2
Error response from daemon: Get https://registry.tkg.vmware.run/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

If you are not using a private container registry with TKG, then you also need to also ensure that the KinD Cluster can connect to your network proxy when it pulls down the required containers from the internet. Luckily, KinD can simply detect the network proxy settings of your operating system. You can either set the proxy using traditional environmental variables (http_proxy, https_proxy and no_proxy) during your use of TKG CLI or you can simply set it globally so you do not forget.

In my setup, TKG CLI is running in a Photon OS VM and global proxy settings are configured in /etc/sysconfig/proxy Proxy settings will vary across operating systems and you should check with the vendor documentation for specific instructions. The following command will set both HTTP and HTTPS proxy variables to use my proxy server and you will also want to make sure you whitelist all networks and addresses which you want to by-pass the proxy.

cat > /etc/sysconfig/proxy << EOF
PROXY_ENABLED="yes"
HTTP_PROXY="http://192.168.1.3:3128"
HTTPS_PROXY="http://192.168.1.3:3128"
NO_PROXY="localhost,192.168.1.0/24,192.168.2.0/24,registry.rainpole.io,10.2.224.4,.svc,100.64.0.0/13,100.96.0.0/11"
EOF

Note: If you are using the TKG Demo Appliance, you only need to configure the Photon OS global proxy settings. In my example, I have white listed my local 192.168.* addresses, registry.rainpole.io which is the embedded Harbor registry, 10.2.224.4 which is the internal IP Address of VMC vCenter Server, *.svc addresses which all the internal K8s services and 100.64.0.0/13 which is the CIDR range used by TKG for the Service networks and 100.96.0.0/11 which is the CIDR range used by TKG Cluster networks.

Tanzu Kubernetes Grid (TKG) Demo Appliance for VMC and vSphere

05/11/2020 by William Lam 4 Comments

As some of you can probably tell from my recent Twitter updates and blog posts (here and here) that I have been spending some time lately with both vSphere with Kubernetes and Tanzu Kubernetes Grid (TKG). Like many of you in the community, I am still pretty new to Kubernetes (K8s) and I am still learning about what it has to offer both from an infrastructure standpoint but more importantly how it can be used to deliver new and modern applications. I am also very lucky to be part of the the VMware Event Broker Appliance Open Source Fling project which builds and runs on top K8s and this project has allowed me to really get hands on which is how I learn best.

A couple of months back I was asked to put together a workshop to demonstrate how to deploy TKG Clusters running on VMware Cloud on AWS (VMC) and while developing the workshop, I thought it would be really cool if I could make it even easier for anyone that is brand new to K8s to quickly get started with TKG. I wanted to have a solution that can literally be dropped into any supported vSphere-based environment with basic networking to go from Zero to Kubernetes in less than 30 minutes!

Enter the Demo Appliance for Tanzu Kubernetes Grid (TKG) Fling

A Virtual Appliance that pre-bundles all required dependencies to help customers in learning and deploying standalone Tanzu Kubernetes Grid (TKG) clusters running on either VMware Cloud on AWS and/or vSphere 6.7 Update 3 environment for Proof of Concept, Demo and Dev/Test purposes. This appliance will enable you to quickly go from zero to Kubernetes in less than 30 minutes with just an SSH client and a web browser!

In addition to the appliance, I have also put together a step by step workshop-style guide which not only walks you through in deploying your first TKG Cluster but also provide some example demos and references which you can explore further. Below are some of the highlights of the Demo Appliance for TKG:

Configure non-secure Harbor registry with Tanzu Kubernetes Grid (TKG)

05/09/2020 by William Lam Leave a Comment

In an earlier blog post, I shared the steps to to configure Harbor with a proper signed SSL certificate that would serve as private container registry for Tanzu Kubernetes Grid (TKG) CLI running in an air-gapped environment.

Although Harbor can easily be configured to support custom CA signed certificate, self-sign certificate and even just using HTTP, there are several additional steps and dependencies that is required if you wish to use a non-secure container registry with TKG CLI. This definitely was a bunch of trial/error and hopefully this can be made easier in the future to easily enable non-secure registry support with TKG CLI out of the box for development and testing purpose.

I also want to give a huge thanks to Jun Wang from our Modern Application Business Unit (MAPU), he was instrumental in helping me out and ultimately his tip on updating the containerd configuration was the last piece to the puzzle so that the K8s images deployed would use our insecure Harbor registry for pulling container images.

Troubleshooting tips for configuring vSphere with Kubernetes

05/05/2020 by William Lam 3 Comments

With more and more folks trying out the new vSphere with Kubernetes capability, I have seen an uptick in questions both internally and externally around the initial setup of the infrastructure required for vSphere with Kubernetes but also during the configuration of a vSphere Cluster for Workload Management.

One of the most common question is why are there no vSphere Clusters listed or why a specific vSphere Cluster is showing up as Incompatible? There are a number of reasons that this can occur including vCenter Server not being able to communicate with NSX-T Manager to retrieve the list of NSX pre-checks which would cause the list to either be empty or listed as incompatible. Not having proper time sync between vCenter Server and NSX-T which can also manifest in a similar behavior among other infrastructure issues.

Having ran into some of these issues myself when developing my automation script, I figure it might be useful to share some of the troubleshooting tips I have used when trying to figure out what is going on whether that is during the initial setup or actually deploying workloads using vSphere with Kubernetes.

Deploying a minimal vSphere with Kubernetes environment

04/29/2020 by William Lam 6 Comments

A very useful property of automation is the ability to experiment. After creating my vSphere 7 with Kubernetes Automation Lab Deployment Script, I wanted to see what was the minimal footprint in terms of the physical resources but also the underlying components that would be required to allow me to still a fully functional vSphere with Kubernetes environment.

Before diving in, let me give you the usual disclaimer 😉

Disclaimer: This is not officially supported by VMware and you can potentially run into issues if you deviate from the official requirements which the default deployment script adheres to out of the box.

In terms of the physical resources, you will need a system that can provision up to 8 vCPU (this can be further reduced, see Additional Resource Reduction section below), 92GB memory and 1TB of storage (thin provisioned).

which translates to following configuration within the script:

1 x Nested ESXi VM with 4 vCPU and 36GB memory
1 x VCSA with 2 vCPU and 12GB memory
1 x NSX-T Unified Appliance with 4 vCPU and 12GB memory
1 x NSX-T Edge with 8 vCPU and 12GB memory

Note: You can probably reduce memory footprint of the ESXi VM further depending on your usage and the VCSA is using the default values for "Tiny", so you can probably trim the memory down a bit more.

Another benefit to this solution is by reducing the number of ESXi VMs required, it also speeds up the deployment and in just 35 minutes, you can have the complete infrastructure fully stood up and configured to try out vSphere with Kubernetes!

The other trick that I leveraged to reduce the amount of resources is by changing the default number of Supervisor Control Plane VMs required for enabling vSphere with Kubernetes. By default, three of these VMs are deployed as part of setting up the Supervisor Cluster, however I found a way to tell the Workload Control Plane (WCP) to only deploy two 🙂

This minimal deployment of vSphere with Kubernetes has already been incorporated into my vSphere with Kubernetes deployment script, but it does require altering several specific settings. You can find the instructions below.