NSX

Getting started with VMware Pivotal Container Service (PKS) Part 3: NSX-T

03/28/2018 by William Lam 6 Comments

In this article, we are now going to start configuring NSX-T so that it will be ready for us to install PKS and consume the networking and security services provided by NSX-T. The result is that PKS can deliver on demand provisioning of all NSX-T components: Container Network Interface (CNI), NSX-T Container Plugin (NCP) POD, NSX Node Agent POD, etc) automatically when a new Kubernetes (K8S) Cluster is requested, all done with a single CLI or API call. In addition, PKS also provides a unique capability through its integration with NSX-T to enable network micro-segmentation at the K8S namespace level which allows Cloud/Platform Operators to manage access between application and/or tenant users at at a much finer grain level than was possible before which is really powerful!

As mentioned in the previous blog post, I will not be walking through a step-by-step NSX-T installation and I will assume that you already have a basic NSX-T environment deployed which includes a few ESXi host prepped as Transport Nodes and at least 1 NSX-T Controller and 1 NSX-T Edge. If you would like a detailed step by step walk through, you can refer to the NSX-T documentation here or you can even leverage my Automated NSX-T Lab Deployment script to setup the base environment and modify based on the steps in this article. In fact, this is the same script I have used to deploy my own NSX-T environment for PKS with some minor modification which I will be sharing at a later date.

If you missed any of the previous articles, you can find the complete list here:

ESXi host with network redundancy using NSX-T and only 2 pNICs?

03/27/2018 by William Lam 7 Comments

In todays data centers, it is not uncommon to find servers with only 2 x 10GbE network interfaces, this is especially true with the rise of Hyper-Converged Infrastructure over the last several years. For customers looking to deploy NSX-T with ESXi, there is an important physical network constraint to be aware of which is quickly mentioned in the NSX-T documentation here.

For example, your hypervisor host has two physical links that are up: vmnic0 and vmnic1. Suppose vmnic0 is used for management and storage networks, while vmnic1 is unused. This would mean that vmnic1 can be used as an NSX-T uplink, but vmnic0 cannot. To do link teaming, you must have two unused physical links available, such as vmnic1 and vmnic2.

As shown in the diagram below, an ESXi host with only two physical NICs can not provide complete network redundancy as each pNIC can only be associated with a single switch (VSS/VDS or the new N-VDS) as pNICs can not be shared across switches.

For customers, this means that you need to allocate a minimum of 4 pNICs to provide redundancy for both overlay traffic and non-overlay VMkernel traffic such as Management, vMotion, VSAN, etc. This is much easier said than done as not all hardware platforms can easily be expanded and even if they can, there still is a huge cost in expanding the physical network footprint (switch port, cabling, etc).

UPDATE (06/12/18) - As of NSX-T 2.2, which was recently released, there is now a UI in NSX-T Manager for managing the migration of VMkernel interfaces to the N-VDS. For automation purposes, you may still find this article useful but now you have option of using the UI.

Getting started with VMware Pivotal Container Service (PKS) Part 2: PKS Client

03/26/2018 by William Lam 5 Comments

In this article, we will configure the various command-line tools that will be used to interact with the PKS Platform which will then be consumed by either the Operators (managing the PKS infrastructure) and/or the the Developers (consumers of the Kubernetes Clusters).

Below is a quick summary, description and the consumers of the CLIs that we will be installing:

CLI	Description	Consumer
pks	Used to create/delete and manage K8S Clusters	Operator
kubectl	Used to interact with K8S Cluster and deploy applications including scaling up/down	Developer
uaac	Used to manage user accounts and authorization for the PKS platform	Operator
bosh	Used to manage PKS deployments and provides information about the VMs using its Cloud Provider Interface (CPI) which is vSphere in this case	Operator
om	Used to Used to manager and interact with Ops Manager	Operator
nsx-cli.sh	Used to clean NSX-T objects after a K8S have been deleted (will be Automated by PKS in future release)	Operator

Both PKS and Kubectl CLIs are supported on either Windows, MacOS and Linux, you can refer to Part 1 for a link to the binary downloads. The remainder of the tools are primarily used by Operators and to make them accessible for multiple users, you can deploy a centralized management VM. In my lab, I am referring to this VM as the "PKS Client" which is where we will be installing all the CLIs. You can use a variety of supported Operating Systems, but I found Ubuntu to work the best, especially for some of the package dependencies. I did try to use our own PhotonOS, but I was having some trouble figuring out the required packages. If I figure it out, then I will update the article as that may be preferred over Ubuntu if you have never worked with it before.

If you missed any of the previous articles, you can find the complete list here:

Getting started with VMware Pivotal Container Service (PKS) Part 1: Overview

03/23/2018 by William Lam 14 Comments

This past week and half, I have been spending quite a bit of time familiarizing myself with the recently released VMware Pivotal Container Service solution, also referred to as VMware PKS for short (yes, that is a K not a C which is a nod to Google's container scheduler Kubernetes). VMware PKS is part of a project that I am currently working on and I figure I would share the process and steps I took to deploy VMware PKS in my own personal lab, in case other folks are interested in trying out this neat and powerful solution for deploying Cloud Native Apps using Kubernetes which was co-developed between VMware, Pivotal and Google.

If you would like to learn more about this first release of VMware PKS and the benefits it provides to both developers (consumers) and operators (admins/SRE) for Kubernetes infrastructure, check out this blog post here. Merlin Glynn, one of the Product Managers for PKS also did an awesome light board video overview of VMware PKS if you want the sparks notes version. If you simply want to give PKS a try without deploying anything, the CNA folks have also published a PKS HOL which can you find here. Another useful resource is the Getting Started with Kubernetes-as-a-Service post from Michael West who works in CNA team and built the PKS HOL.

This will be the first, in a series of articles outlining my VMware PKS deployment and configuration which hopefully can help benefit others as it took me several attempts while learning about the solution. Although the first few articles will include manual guidance, rest assure, there will be some cool automation towards the end but I figure that folks may want to go through this once by hand to get a good understanding on all the different components and how they interact with each other. Plus, some of the PKS-specific automation is still being worked on by the product team and hopefully I will be able to share some of that real soon.

If you missed any of the previous articles, you can find the complete list here:

How to troubleshoot NSX-T API errors when using PowerCLI?

03/21/2018 by William Lam Leave a Comment

In the last few days, I have been ramping back up on NSX-T usage, especially as I take a closer look at VMware Pivotal Container Service (PKS) which hopefully I will be able to share more with an upcoming blog series (at least, thats the plan when I find some spare time). While working on some Automation using the NSX-T REST APIs and the new NSX-T PowerCLI cmdlets, I found it to be pretty challenging and frustrating to troubleshoot NSX-T error messages thrown from the PowerCLI cmdlets.

For example, below is a PowerCLI snippet which I wrote to create a new T1 Logical Router in NSX-T.

$edgeClusterService = Get-NsxtService -Name "com.vmware.nsx.edge_clusters"
$edgeCluster = $edgeClusterService.list().results | where { $_.display_name -eq "Edge-Cluster-01" }
$logicalRouterService = Get-NsxtService -Name "com.vmware.nsx.logical_routers"
$lrSpec = $logicalRouterService.Help.create.logical_router.Create()
$lrSpec.display_name = "Foo"
$lrSpec.router_type = "TIER1"
$lrSpec.high_availability_mode = "ACTIVE_STANDBY"
$lrSpec.failover_mode = "FAILOVER_MODE_PREEMPTIVE"
$lrSpec.edge_cluster_id = "ea655f88-e184-47fe-9352-8ab9d8c955a7"
$lrAdd = $logicalRouterService.create($lrSpec)

$edgeClusterService = Get-NsxtService -Name "com.vmware.nsx.edge_clusters"

$edgeCluster = $edgeClusterService.list().results | where { $_.display_name -eq "Edge-Cluster-01" }

$logicalRouterService = Get-NsxtService -Name "com.vmware.nsx.logical_routers"

$lrSpec = $logicalRouterService.Help.create.logical_router.Create()

$lrSpec.display_name = "Foo"

$lrSpec.router_type = "TIER1"

$lrSpec.high_availability_mode = "ACTIVE_STANDBY"

$lrSpec.failover_mode = "FAILOVER_MODE_PREEMPTIVE"

$lrSpec.edge_cluster_id = "ea655f88-e184-47fe-9352-8ab9d8c955a7"

$lrAdd = $logicalRouterService.create($lrSpec)

However, when I ran the code, PowerCLI threw back a pretty generic and useless error which I am sure some of you have probably seen before "A server error occurred: 'com.vmware.vapi.std.errors.invalid.frequest': . Check $Error[0].Exception.ServerError". Even checking the error variable as stated in the error message did not yield any further details or specific to the issue I was running into.

Quick Tip – NSX-T 2.1 upgrade gotcha, use NSX Manager IP Address vs Hostname

01/02/2018 by William Lam Leave a Comment

NSX-T 2.1 was released right before the holiday break and one of the biggest enhancement is support for VMware's upcoming Pivotal Container Service (PKS) which you can read more about here. Right before I took some much needed time off with the family, I had attempted an upgrade of my existing NSX-T 2.0 environment and ran into an issue which turned out to be a known issue but apparently it is no where listed on the NSX-T 2.1 release notes.

I figure I would share the simple workaround as it drove me a bit insane trying to figure out what was going on. After successfully uploading the NSX-T 2.1 upgrade bundle and then starting the upgrade, which starts off by updating the Upgrade Coordinator component and shortly after, I am shown the following error message:

This page is only available on the NSX Manager where Upgrade Coordinator is running. To enable the service, run the command "set service install-upgrade enabled" on the NSX Manager. If the install-upgrade service is already enabled, try disabling it using "clear service install-upgrade enabled" and then enable it again.

Although the error seems to suggest the Upgrade Coordinator service may not be running (it was), even after performing the recommended operations, the error message continue to persists as seen in the screenshot below.

After reaching out to Engineering as I was out of ideas on how to further troubleshoot the issue, it turns out this was actually a known upgrade issue going from 2.0 to 2.1. Instead of connecting to the NSX-T Manager using its Hostname (FQDN), I needed to use the IP Address instead for the upgrade. Once I logged back in using the IP Address, I was no longer seeing the error message and could proceed with my NSX-T upgrade as shown in the screenshot below.