virtuallyGhetto

VMware Cloud Services Status APIs

05/04/2020 by William Lam Leave a Comment

For customers that use any of the VMware Cloud Services, you can check the status of any or all services by visiting https://status.vmware-services.io. You even have the option of subscribe via a number of communication channels to receive updates.

Awhile back, I had someone ask whether our status page which uses the status.io service provides any APIs that could be used to programmatically retrieve status for specific service. I figured there probably is a way but only recently did I have a chance to follow-up on that request. The answer is yes and below is a quick example on how you use the REST APIs.

Changing the default size of the ESX-OSData volume in ESXi 7.0

05/02/2020 by William Lam 7 Comments

In ESXi 7.0, a new partition scheme was introduced which also brings along a new set of storage requirements. These changes are explained in the official documentation here and the following VMware KB 77009 also contains some additional info which can be helpful. Storage changes are not easy but this was necessary to not only better support some of the current capabilities but more importantly, it setups the foundation for future ESXi capabilities.

The biggest change to the partition layout is the consolidation of VMware Tools Locker, Core Dump and Scratch partitions into a new ESX-OSData volume (based on VMFS-L). This new volume can vary in size (up to 138GB) depending on a number of factors including the current ESXi boot media (USB SD-Card, Local Disk) but also the size of the device itself, which is explained in the official documentation.

From some of the comments on Twitter, Reddit and the direct inquiries that I have received, this new behavior seems to be most impactful to smaller homelabs where a fresh install of ESXi 7.0 has been performed. Folks have shared that their ESX-OSData volume has taken up 120GB which can be quite significant if you have a smaller disk which can be quite common. I normally install ESXi on a USB device and I also use vSAN, which has a different behavior and I have also not upgraded my physical ESXi host (E200-8D) to 7.0 yet.

I performed a fresh installation of ESXi 7.0 (running as Nested ESXi VM) that was configured with 1TB of storage and here is what the filesystem layout now looks:

We can see that the ESX-OSData volume takes up ~119.75GB, which is not too bad for 1TB volume but I can understand this may not be ideal if you have something smaller such as 250GB to 512GB disk. Due to the size of the local device, the boot options mentioned in the KB would not be helpful and I was curious myself if this ESX-OSData volume size could be configurable. In doing some research it looks like the size of the ESX-OSData can be specified using the following ESXi boot option (SHIFT+O during the initial boot) called autoPartitionOSDataSize

Disclaimer: This may not be officially supported by VMware as it deviates from the system defaults and can have other unintended behaviors. Use at your own risk.

Deploying a minimal vSphere with Kubernetes environment

04/29/2020 by William Lam 6 Comments

A very useful property of automation is the ability to experiment. After creating my vSphere 7 with Kubernetes Automation Lab Deployment Script, I wanted to see what was the minimal footprint in terms of the physical resources but also the underlying components that would be required to allow me to still a fully functional vSphere with Kubernetes environment.

Before diving in, let me give you the usual disclaimer 😉

Disclaimer: This is not officially supported by VMware and you can potentially run into issues if you deviate from the official requirements which the default deployment script adheres to out of the box.

In terms of the physical resources, you will need a system that can provision up to 8 vCPU (this can be further reduced, see Additional Resource Reduction section below), 92GB memory and 1TB of storage (thin provisioned).

which translates to following configuration within the script:

1 x Nested ESXi VM with 4 vCPU and 36GB memory
1 x VCSA with 2 vCPU and 12GB memory
1 x NSX-T Unified Appliance with 4 vCPU and 12GB memory
1 x NSX-T Edge with 8 vCPU and 12GB memory

Note: You can probably reduce memory footprint of the ESXi VM further depending on your usage and the VCSA is using the default values for "Tiny", so you can probably trim the memory down a bit more.

Another benefit to this solution is by reducing the number of ESXi VMs required, it also speeds up the deployment and in just 35 minutes, you can have the complete infrastructure fully stood up and configured to try out vSphere with Kubernetes!

The other trick that I leveraged to reduce the amount of resources is by changing the default number of Supervisor Control Plane VMs required for enabling vSphere with Kubernetes. By default, three of these VMs are deployed as part of setting up the Supervisor Cluster, however I found a way to tell the Workload Control Plane (WCP) to only deploy two 🙂

This minimal deployment of vSphere with Kubernetes has already been incorporated into my vSphere with Kubernetes deployment script, but it does require altering several specific settings. You can find the instructions below.

Update on ESXi on Apple Mac Mini 2018 & Mac Pro 2019

04/28/2020 by William Lam 14 Comments

Although there has not been any news in some time regarding the support for ESXi on the latest Apple Mac Mini 2018 and the recently released Apple Mac Pro 2019, there has definitely been work happening behind the scenes at VMware. Today, I would like to share a pretty significant update as a result of some of these efforts.

MacOS Guest

One of the biggest issue which I had observed when using a T2-based Apple system with ESXi is that it would fail to boot a MacOS Guest and just keep rebooting the VM. I am very happy to announce that this issue has been resolved and ESXi can now properly recognize the Apple System Management Controller (SMC) device which is used as part of the MacOS Guest start up process. This now means a MacOS Guest will be able to properly boot on a T2-based Apple system.

Thunderbolt 3

Another impact of a T2-based Apple system with ESXi is that storage and networking devices connected to the Thunderbolt 3 ports are not visible. I am also happy to announce that this issue has been resolved and ESXi can now see PCIe devices that are attached to the Thunderbolt 3 ports.

An ESXi Advanced Setting change is required for Thunderbolt 3 to work correctly and the following command will need to be executed after installing ESXi:

esxcli system settings kernel set -s pciExperimentalFlags -v 16

Once the setting has been applied, a system reboot will be required and your PCIe devices will show up properly. In future, this additional configuration may not be required and can be detected based on the underlying hardware.

Both of the fixes mentioned above are included in the latest ESXi 6.7 Patch 02 (ESXi670-202004002) release which is available today! Hopefully this was the news that many of you have been waiting for 😀

Now, before you rush out to start deploying MacOS Guests on either the Mac Mini or Mac Pro, I do have to mention that neither the Mac Mini 2018 or the Mac Pro 2019 will be officially supported by VMware. Due to the current situation that we are all in with COVID-19, personnel access to VMware facilities like many other organizations has been severely restricted and/or prohibited. In fact, much of the early validation was done by yours truly using a Mac Mini 2018 which I had access to (Thanks Michael Roy) as Engineering did not have access to hardware during the shelter in place orders. This also means that certifications of these platforms is still on-going and until these systems are officially listed on VMware's HCL, they will not be officially supported by VMware.

Disclaimer: VMware currently does not officially support the Apple 2018 Mac Mini8,1 or the Apple 2019 Mac Pro7,1

Configure log forwarding from Tanzu Kubernetes Grid (TKG) to vRealize Log Insight Cloud

04/27/2020 by William Lam Leave a Comment

As much as I enjoy kubectl'ing logs in real time for troubleshooting and debugging purposes, this usually does not scale beyond a couple of Kubernetes (K8s) Clusters if you are lucky. Even then, you will not retain any of the historical logs which may be required for deeper analysis or for auditing purposes. This is usually solved by having a centralized log management platform and while working with Tanzu Kubernetes Grid (TKG) running on VMware Cloud on AWS, a solution like vRealize Log Insight Cloud (vRLIC) makes a lot of sense.

While browsing through the vRLIC console, I noticed that it supports a number of log sources including K8s which was exactly what I was looking for. However, after going through the instructions in configuring fluentd on my TKG Cluster, I found that that nothing was being sent. After a bit of debugging, I realized a few steps were actually missing that was required to setup this up on TKG Cluster.

I eventually figured it out and will be sharing this feedback with the vRLIC folks but in the meantime, you can follow the instructions below on how to forward both system and application logs from your TKG Cluster or any K8s deployment for that matter which has outbound connectivity to connect to vRLIC.

Deploy Harbor in an Air-Gapped environment for Tanzu Kubernetes Grid (TKG)

04/24/2020 by William Lam Leave a Comment

When using Tanzu Kubernetes Grid (TKG) and the new TKG CLI, outbound internet connectivity is required as part of the initial setup on the machine running TKG CLI but also on the TKG Management Cluster which is automatically stood up as part of the deployment. For demo and testing purposes, this is usually not a problem but for anyone looking to run this in a Production or datacenter environment, direct internet access is generally not available.

TKG does support air-gapped environments today by requiring a private container registry that has been configured with all the required containers. Once your registry has been setup, you will also need to update the TKG YAML manifest files to specify your private registry as by default, it will point to registry.tkg.vmware.run. You can use any container registry that is supported with Kubernetes including the popular Harbor solution. One thing to note is that your private registry must have a proper signed SSL certificate, custom CA certificates or self-signed certificates are not officially supported today with TKG.

Since I recently had to set this up for a project I am working on, which I hope to talk about in a future blog post, I thought it would be useful to share the instructions on how to setup and configure Harbor to be used in-conjunction with TKG as well as any other solution that requires a container registry running in your own environment. In my deployment, I will be using Let's Encrypt for generating the required SSL certificate, but you can use any existing service for performing this operation. I will also be installing Harbor on Photon OS, but you can use any operating system of your choice that Harbor is supported on.

Pre-Requisites

Access to a public DNS domain which you have ownership of (e.g. adding new records)
Access to your internal DNS server to add a custom DNS zone lookup entry (e.g. registry.<yourdomain>.com)