Terraform

Using Terraform to deploy a Tanzu Kubernetes Grid (TKG) Cluster in vSphere with Tanzu

11/10/2020 by William Lam 1 Comment

A few months back I saw that HashiCorp had released a new Kubernetes (K8s) Provider for Terraform, currently in Alpha state, which enable users to deploy K8s resources using the popular Infrastructure-as-Code (IaC) tool. I thought this would be pretty cool if it works with our vSphere with Tanzu solution, since the Tanzu Kubernetes Grid (TKG) Service uses ClusterAPI via a custom VM Operator to deploy TKG Guest Clusters which is just a fancy way of saying it uses K8s API to deploy more K8s 🙂

The setting up the new K8s provider was pretty straight forward and after spending a few minutes in figuring out how to convert my existing TKG YAML to the required HCL format for Terraform to understand, I was able to to run a terraform "plan" but quickly ran into the following error:

failed: admission webhook "default.mutating.tanzukubernetescluster.run.tanzu.vmware.com" does not support dry run

It looks like our tanzukubernetescluster admission webhooks does not currently support dry run operations which can be quite useful but also common when using Terraform. I figured this was the end of that idea and I ended up just filing a feature enhancement internally for adding this support in the future as I can see this being quite useful for our customers.

After finishing up recent pet project of getting a fully functional vSphere with Tanzu on a homelab budget and just using 32GB of memory, I decided to take another look at this and discovered the required tweak to get this working was super trivial, literally a single line change.

Disclaimer: This is not officially supported by VMware, use at your own risk.

Using ESXi-Arm Fling as a lightweight vSphere Automation environment for PowerCLI and Terraform

10/09/2020 by William Lam 1 Comment

A set of use cases that I was really excited for when I first heard about ESXi-Arm a few years ago was around the topic of vSphere Automation and Development. I speak with many customers who are just starting out on their Automation journey whether that is using PowerCLI, one of our many vSphere Automation SDK or even directly to the new vCenter REST API which all new features are being exposed through these days.

One of the biggest challenge for new comers is simply getting access to hardware that they can start playing around with and although there are is plethora of vSphere Homelab choices, it does require some amount of investment, which is definitely worth it in the long run. However, if you are just getting started and maybe you want something that is a bit more lighter weight, there are not too many options outside of an Intel NUC. I know many consultants actually carry around an Intel NUC that contains several VM images that they use to with their clients, including demos.

With the small form factor, low cost and reduced power consumption of the Raspberry Pi, I think this really opens up the door for some interesting creative solutions:

Basic vSphere footprint that can be used for work or learning purposes
Easy way to learn and explore the vSphere API with an actual host and enabling real VM deployments
Trying out Infrastructure-as-Code (IaC) tools such as Terraform and Ansible
Quick way to run through basic demos in front of customers
On-demand and self-contained lab environment for small Hackathon at your local VMUG or even at VMworld

Something I was really interested in early on was to be able to use ESXi-Arm with the Raspberry Pi to not only have a basic ESXi environment but also have PowerCLI environment up and running in an Arm VM. My first thought was to get this setup using Photon OS, which not only has Arm distribution but also has support for Powershell and PowerCLI. I was hoping with some tinkering, I could easily get Powershell for Arm to run on PhotonOS (which it did) but I then ran into issues installing PowerCLI itself.

I decided to give up for now and take a look at Ubuntu which also supports Powershell for Arm, but the Microsoft documentation only listed instructions for 32-bit and ESXi-Arm requires a 64-bit. Taking a look at the Powershell release files, I noticed there was 64-bit package and with a few minor adjustments to the commands, I got PowerCLI installed and connected back to my rPI which was attached to my x86 vCenter Server!

Quick Tip – Certificates in Apple Keychain causes Terraform init to fail with Registry service unreachable

06/22/2020 by William Lam 1 Comment

I have been struggling with an interesting Terraform issue on my MacOS system where running the "init" operation would throw the following error:

Initializing the backend...

Initializing provider plugins...
- Checking for available provider plugins...

Registry service unreachable.

This may indicate a network issue, or an issue with the requested Terraform Registry.

Error: registry service is unreachable, check https://status.hashicorp.com/ for status updates

This was extremely frustrating to debug which I had filed a Github issue here. From what I have gathered, this actually had nothing to do with connectivity to the HashiCorp endpoint which works perfectly but probably was related to some other issue. What was even more strange was that using "sudo" which another user reported in an older issue allowed the operation to go through. I was also not having this problem on my other MacOS system, so I knew this was probably environmental but was running out of ideas to try.

I took another look this past weekend while doing some testing and I stumbled onto this thread here which the user found the real root cause. It looks like certain certificates within Apple Keychain Access, possibly related to Microsoft Remote Desktop that have expired was actually causing the problem. When I took at look at the Keychain Access login->certificates, I saw a number of certificates which had expired but were still marked trusted. After removing these entries (although this can be automated using the security utility, it was not trivial given the lack of arguments to quickly list out expired certificates), that I simply used the UI to delete the entries.

Once all the expired certificates were removed, I was able to successfully perform the Terraform init operation! I have already shared this update in my Github issue and hopefully this error message can be improved in the future as it was very miss-leading on the actual issue.

Full OVA/OVF property support coming to Terraform provider for vSphere

06/11/2020 by William Lam 10 Comments

Terraform is one of the most popular Infrastructure as Code (IaC) tool out there today and it should come as no surprise there is Terraform provider for vSphere which many of our customers have been using. In fact, VMware just recently released a couple more new providers (here and here) supporting VMware Cloud on AWS and NSX-T solutions respectively.

Although I have used Terraform and the vSphere provider in the past, it has not been my tool of choice for automation as it still lacks a number of basic vSphere capabilities which I require on a regular basis. The most common one being the ability to deploy a Virtual Appliance (OVA/OVF) which has been my biggest barrier and I know this has been a highly requested feature from the community as well.

In early May of this year, I noticed that v1.18 of the vSphere provider finally added support for OVA/OVF deployment and I was pretty excited to give this a try and may even have been the first to kick the tires on this feature? Although OVA/OVF support was added, it looks like support for customizing OVF properties which is commonly included as part of an OVA/OVF would only possible if you are cloning from an existing imported OVA/OVF image. One of the most common use case is to import an OVF/OVA from either your local computer or from a URL and it looks like this use case was not possible.

I filed two Github issues, one for supporting OVF properties for initial OVA/OVF deployment and another regarding a bug I ran into when importing OVA/OVF from a remote URL. Just yesterday, I got the good news that my feature request has been completed and I was given an early drop of the vSphere provider to try out this feature. I may have also hinted to the Engineering team to use my popular Nested ESXi Appliance OVA as a reference test implementation as I knew this was something many customers will want to deploy 🙂

UPDATE (06/23/20) - Support for OVA/OVF properties is now available as part of 1.20 of the Terraform Provider for vSphere