virtuallyGhetto

Configure NSX-T Edge to run on AMD Ryzen CPU

05/06/2020 by William Lam 1 Comment

The vast majority of VMware Homelabs is still Intel-based today but I have been seeing a slow rise of AMD-based kits being adopted, especially with AMD's desktop line of CPUs known as Ryzen. One of the considerations on whether you could use an AMD processor was whether you were planning to deploy NSX-T and in earlier releases, only Intel was supported as the NSX-T Edge required support for Data Plane Development Kit (DPDK) and this was only supported with Intel-based processors.

With the latest NSX-T 3.0 release, AMD-based processors are now supported and per the release notes, the following CPUs can be used:

AMD EPYC 7xx1 Series (Naples)
AMD EPYC 3000 Embedded Family and newer
AMD EPYC 7xx2 Series (Rome)

You will notice that only AMD's server line of CPUs known as EPYC are currently supported, which makes sense for running Production workloads. If you attempt to deploy an NSX-T Edge Node running on a non-EPYC platform, you will get an error message stating the CPU is not supported and I figured this was probably due to the lack of DPDK support in the consumer CPUs.

Yesterday, in our internal "Homelab" Slack channel, I came across an interesting tidbit from Andrea Spagnolo, a Sr. Field Engineer in our Cloud Native Business Unit who shared a pretty neat trick on how to get latest NSX-T 3.0 release to work with a Ryzen-based CPU.

Disclaimer: This is not officially supported by VMware. The behaviors described here can change in the future

First off, I want to thank Andrea for sharing but also credit to Beniamino Guarnaschelli and his blog post here which actually gave Andrea the idea to take a closer look as he was trying to get this setup in his own personal homelab.

Troubleshooting tips for configuring vSphere with Kubernetes

05/05/2020 by William Lam 4 Comments

With more and more folks trying out the new vSphere with Kubernetes capability, I have seen an uptick in questions both internally and externally around the initial setup of the infrastructure required for vSphere with Kubernetes but also during the configuration of a vSphere Cluster for Workload Management.

One of the most common question is why are there no vSphere Clusters listed or why a specific vSphere Cluster is showing up as Incompatible? There are a number of reasons that this can occur including vCenter Server not being able to communicate with NSX-T Manager to retrieve the list of NSX pre-checks which would cause the list to either be empty or listed as incompatible. Not having proper time sync between vCenter Server and NSX-T which can also manifest in a similar behavior among other infrastructure issues.

Having ran into some of these issues myself when developing my automation script, I figure it might be useful to share some of the troubleshooting tips I have used when trying to figure out what is going on whether that is during the initial setup or actually deploying workloads using vSphere with Kubernetes.

Quick Tip – Suppress new core dump warning in ESXi 7.0

05/04/2020 by William Lam 1 Comment

You may have noticed new warning message after installing ESXi 7.0, this can happen if your ESXi installation is on a smaller local disk (Nested ESXi Appliance would also be affected) which does not have enough capacity for setting up a core dump target or if ESXi is booting from USB which also does not support core dumps by default.

No coredump target has been configured. Host core dumps cannot be saved.

It is certainly recommended that you have a core dump target configured, especially for Production systems. However, if you wish to suppress the warning, there is an ESXi Advance Setting which you can toggle called UserVars.SuppressCoredumpWarning (can be configure using Embedded ESXi Host Client or vSphere UI in vCenter Server) as well as ESXCLI. Below is the command to suppress the warning:

esxcli system settings advanced set -o /UserVars/SuppressCoredumpWarning -i 1

VMware Cloud Services Status APIs

05/04/2020 by William Lam Leave a Comment

For customers that use any of the VMware Cloud Services, you can check the status of any or all services by visiting https://status.vmware-services.io. You even have the option of subscribe via a number of communication channels to receive updates.

Awhile back, I had someone ask whether our status page which uses the status.io service provides any APIs that could be used to programmatically retrieve status for specific service. I figured there probably is a way but only recently did I have a chance to follow-up on that request. The answer is yes and below is a quick example on how you use the REST APIs.

Changing the default size of the ESX-OSData volume in ESXi 7.0

05/02/2020 by William Lam 8 Comments

In ESXi 7.0, a new partition scheme was introduced which also brings along a new set of storage requirements. These changes are explained in the official documentation here and the following VMware KB 77009 also contains some additional info which can be helpful. Storage changes are not easy but this was necessary to not only better support some of the current capabilities but more importantly, it setups the foundation for future ESXi capabilities.

The biggest change to the partition layout is the consolidation of VMware Tools Locker, Core Dump and Scratch partitions into a new ESX-OSData volume (based on VMFS-L). This new volume can vary in size (up to 138GB) depending on a number of factors including the current ESXi boot media (USB SD-Card, Local Disk) but also the size of the device itself, which is explained in the official documentation.

From some of the comments on Twitter, Reddit and the direct inquiries that I have received, this new behavior seems to be most impactful to smaller homelabs where a fresh install of ESXi 7.0 has been performed. Folks have shared that their ESX-OSData volume has taken up 120GB which can be quite significant if you have a smaller disk which can be quite common. I normally install ESXi on a USB device and I also use vSAN, which has a different behavior and I have also not upgraded my physical ESXi host (E200-8D) to 7.0 yet.

I performed a fresh installation of ESXi 7.0 (running as Nested ESXi VM) that was configured with 1TB of storage and here is what the filesystem layout now looks:

We can see that the ESX-OSData volume takes up ~119.75GB, which is not too bad for 1TB volume but I can understand this may not be ideal if you have something smaller such as 250GB to 512GB disk. Due to the size of the local device, the boot options mentioned in the KB would not be helpful and I was curious myself if this ESX-OSData volume size could be configurable. In doing some research it looks like the size of the ESX-OSData can be specified using the following ESXi boot option (SHIFT+O during the initial boot) called autoPartitionOSDataSize

Disclaimer: This may not be officially supported by VMware as it deviates from the system defaults and can have other unintended behaviors. Use at your own risk.

Deploying a minimal vSphere with Kubernetes environment

04/29/2020 by William Lam 6 Comments

A very useful property of automation is the ability to experiment. After creating my vSphere 7 with Kubernetes Automation Lab Deployment Script, I wanted to see what was the minimal footprint in terms of the physical resources but also the underlying components that would be required to allow me to still a fully functional vSphere with Kubernetes environment.

Before diving in, let me give you the usual disclaimer 😉

Disclaimer: This is not officially supported by VMware and you can potentially run into issues if you deviate from the official requirements which the default deployment script adheres to out of the box.

In terms of the physical resources, you will need a system that can provision up to 8 vCPU (this can be further reduced, see Additional Resource Reduction section below), 92GB memory and 1TB of storage (thin provisioned).

which translates to following configuration within the script:

1 x Nested ESXi VM with 4 vCPU and 36GB memory
1 x VCSA with 2 vCPU and 12GB memory
1 x NSX-T Unified Appliance with 4 vCPU and 12GB memory
1 x NSX-T Edge with 8 vCPU and 12GB memory

Note: You can probably reduce memory footprint of the ESXi VM further depending on your usage and the VCSA is using the default values for "Tiny", so you can probably trim the memory down a bit more.

Another benefit to this solution is by reducing the number of ESXi VMs required, it also speeds up the deployment and in just 35 minutes, you can have the complete infrastructure fully stood up and configured to try out vSphere with Kubernetes!

The other trick that I leveraged to reduce the amount of resources is by changing the default number of Supervisor Control Plane VMs required for enabling vSphere with Kubernetes. By default, three of these VMs are deployed as part of setting up the Supervisor Cluster, however I found a way to tell the Workload Control Plane (WCP) to only deploy two 🙂

This minimal deployment of vSphere with Kubernetes has already been incorporated into my vSphere with Kubernetes deployment script, but it does require altering several specific settings. You can find the instructions below.