Automation

How to deploy the vCenter Server Appliance (VCSA) with a custom MAC Address?

02/20/2020 by William Lam Leave a Comment

I recently had a question that came in from our field where a customer needed to deploy the vCenter Server Appliance (VCSA) with a specific MAC Address which was a requirement to ensure property connectivity within their network. This type of network requirement is not really new or unique, it is a common practice used to ensure only valid VMs with a static DHCP reservation can actually connect to a specific network but it certainly was the first time I had heard of this request for the VCSA.

With the default VCSA installer workflow, there is currently not a way to modify the network MAC Address which is automatically generated after the deployment of the OVA. Having said that, I have spent quite a bit of time exploring the various non-standard methods of deploying the VCSA in the past (see here, here and here) and with that information, you definitely can affect the MAC Address while still maintaining a valid VCSA deployment. With a bit of trial/error, there are two options depending if you are deploying the VCSA directly to an ESXi host (for initial setup) or to an existing vCenter Server. To demonstrate how this works, I have created a basic shell script called VCSAStaticMACAddress.sh which you can easily adapt to for a Windows-based environment.

The trick is that when you deploy to a vCenter Server endpoint, the required OVF properties are persisted which would allow you to only deploy the VCSA but not actually power it on and there you can easily augment a number of settings including the MAC Address. In the case of deploying directly to an ESXi host, OVF properties are not persisted and hence a challenge if you wish to make changes prior to powering on the VM. In earlier versions, it was possible to set these OVF properties by way of using the extraConfig property of the VM but it looks like this trick no longer works and requires a slight variation of the workflow which is described in the instructions below.

How to automate the creation multiple routable VLANs on single L2 network using VyOS

02/12/2020 by William Lam 4 Comments

My personal homelab has a very simple network topology, everything is connected to a single flat network. This has served me well over the years, but sometimes it can prevent me from deploying more complex scenarios. Most recently while working with NSX-T and Project Pacific, I had a need for additional VLANs which my home router does not support. There are a number of software solutions that can be used including the popular pfSense, which I have used before.

Over the Winter break, a colleague introduced me to VyOS, which is another popular software firewall and router solution. I had not heard of VyOS before but later realized it was derived from Vyatta, which I had heard of, but development of that solution had stopped and VyOS is now the open source version of that software. Having never played with VyoS before, I thought this might be a good learning opopournity and started to dabble with VyOS over the holiday. At a high level, I have VyOS connected to two networks: Outside network as VyOS refers which is your local LAN and Inside network as VyOS refers which is an is an isolated vSphere Portgroup (VSS/VDS) that is not connected to anything and configured to pass all traffic (4095). From here, you can create multiple VLANs in VyOS which can then be untagged using Virtual Guest Tagging (VGT) by placing a Nested ESXi VM on the same isolated portgroup and then creating the respective portgroups within the Nested ESXi VM mapping to the VyOS VLANs you have created.

One of the nice benefits of this solution is that you can create multiple "Isolated" yet routable networks that can still reach your primary LAN network and still have to access core infrastructure services running like Active Directory, DNS, etc. which was one of my requirements. After figuring out how VyOS works and applying that to my specific use case, I thought why not build some basic automation to setup this solution as I probably will forget how I setup everything. Initially I was using the VyOS OVA but later found out it was an extremely out of date there was no public version of the latest VyOS release in OVA form. I decided to use their latest rolling release and apply some vSphere API Automation to not only install VyOS but also fully configure based on template containing VyOS commands. I know the latest version of VyOS now includes a REST API but its a bit of a chicken/egg to enable and not very friendly to use compared to the solution I have built.

How to exclude VCSA UI/CLI Installer from MacOS Catalina Security Gatekeeper?

02/08/2020 by William Lam 1 Comment

A couple of weeks ago I had upgraded my personal home computer to the latest MacOS Catalina (10.15) and one of the first issues I ran into was being able to access my vCenter Server. It turned out this was due to changes to MacOS security (which is a good thing) but certainly caught me and others off guard. In fact, I spent quite some time searching online and eventually found this workaround here.

After sharing this tidbit online (which several others also ran into) I came to learn that both Duncan Epping blogged about this issue back in Nov 2019 here and Christian Mohn blogged about this in Dec 2019 here. Sadly I did not come across either of their blogs using "NET::ERR_CERT_REVOKED macos catalina" in Google. I had assumed this was a Chrome issue and simply landed on the first few links and looking back, I now see Duncan's blog was #6 in the search results (doh!)

Today, I ran into another issue when attempting to use the VCSA CLI Installer, the following error was thrown:

“vcsa-deploy.bin” cannot be opened because the developer cannot be verified

This is again due to a security change in MacOS Catalina which now prevents terminal-based applications which are not notarized from running. For a single application/binary, you can go into System Preferences->Security & Privacy and allow anyway. For more complex applications like the VCSA CLI Installer which has a number of libraries and scripts, this will take awhile and end up frustrating end users. The updated security enhancement is actually a good thing and I did not want to disable the Gatekeeper service but I was interested in disabling it for the VCSA CLI Installer. While searching online, I came across this Hashicorp Terraform thread where folks were having the exact same issue and I found out there was a way to disable the MacOS Security Gatekeeper for a specific application.

To do so, we just need to recursively remove the metadata attribute "com.apple.quarantine" for the extracted VCSA ISO by running the following command:

sudo xattr -r -d com.apple.quarantine VMware-VCSA-all-6.7.0-Update-15132721

After the quarantine attribute has been removed, you can now run the VCSA CLI Installer (including UI Installer) without being prompted with an error. Hopefully VMware will consider notarizing future releases of the VCSA Installer and I will be sharing this feedback internally if it has not already.

Using PowerCLI to automate the retrieval of VCSA Password Policies

02/06/2020 by William Lam Leave a Comment

I hope that every vSphere administrator or operator by now is familiar with the extremely powerful vSphere Guest Operations API functionality (details here and here), which can easily be consumed using PowerCLI's Invoke-VMScript cmdlet. If not, highly recommend you check out the links referenced. I know the GuestOps API is certainly my top favorite with sending VM keystrokes capability a very close second!

Not only does the GuestOps API unlock functionality that simply may not be possible (e.g. there's no API or automation interface) but it also enables automation within a VM without requiring any type of remote management services enabled (e.g. SSH or WinRM) or even networking to the VM for that matter!

The reason I am bringing all this up is that although there is not an API for managing and retrieving vCenter Single Sign-On (SSO) configurations which includes password policies, there is a way in which customers can still automate and retrieve this and other information by leveraging the GuestOps API. In fact, back in 2015 I demonstrated on how you can retrieve VCSA SSO password policy and configurations and we can simply apply the GuestOps API to help us automate this task. In addition, most customers do not enable SSH by default and we can still apply the GuestOps API technique and perform automation tasks to VSCSA without requiring SSH as described in this blog post back in 2016.

Workspace One Access (vIDM) Powershell Module to automate creating 3rd Party Identity Provider

02/05/2020 by William Lam Leave a Comment

One of the projects I am currently working on involves Workspace One Access (formally VMware Identity Manager) and configuring a 3rd Party Identity Provider for Identity Federation. As with anything, using the UI for the first time to validate the workflow is perfectly fine for me but after that, I normally prefer to automate, especially as I was rebuilding this particular setup a few times. I saw that Workspace One Access (WSO Access) had a REST API but I was surprised that there were no APIs for actually managing the configurations.

I figured before giving up, I should see at least see how the UI was performing these operations as "some API" should exists and started up one of my favorite browser tools Chrome Developer Console to inspect the HTTP requests. I came to learn there were an additional set of "Jersey" APIs (no background on the Jersey name, but its part of the API URI) that might do exactly what I was looking for. After a bit of trial/error, I was able to fully automate the creation of both a WSO Access Directory as well as 3rd Party Identity Provider.

Cross vCenter Workload Migration Fling v3.1

01/22/2020 by William Lam 1 Comment

Here is a small update to the Cross vCenter Workload Migration Fling which includes a couple of commonly requested features along with some bug fixes.

What's New in v3.1

Support for disk format conversion between Thick (Lazy Zeroed), Thick (Eager Zeroed) and Thin provisioning
Support for VM rename pattern for Clone operation
Fixed duplicated network selection when performing bulk migration
Fixed startup failure when a new home vCenter is specified as a command line argument

vSphere HTML5 and Standalone UI Client Support

In our 3.0 release, we added support for a native vSphere HTML5 (H5) Client experience which leverages new remote plugin framework that was introduced in vSphere 6.7 Update 1 and enables customers who prefer to use the H5 Client for their day to day use to also take advantage of the Cross vCenter Workload Migration Tool directly from the same UI. However, the addition of this new consumption UI created some confusion as some folks assumed this was the only mechanism. As stated in the release notes, we support both the new H5 UI as well as the standalone UI which many customers have been using for quite some time.

I suspect the confusion was due to the new CLI syntax which now requires specifying a vCenter Server endpoint to register. It is true that if you wish to use the new H5 Client integration, you will need to have a vSphere 6.7 Update 1 environment and provide credentials to that "home" vCenter Server. However, if you do not wish to use the H5 Client and you wish to use the old standalone client, you simply omit the vCenter Server registration details and the standalone client will work. In fact, even if you decide to use the H5 Client UI, you can always use the standalone client as that is the actual backend of the system.

Option 1: vSphere H5 Client Plugin

The following command will register the Cross vCenter Workload Migration Fling plugin to the specified vCenter Server:

java -jar xvm-3.1.jar --vcenter.fqdn=VCENTER-IP-OR-FQDN --vcenter.user=ADMIN-USER --vcenter.pass=ADMIN-PASSWORD

You will need to logout and then log back in to see the plugin which is located under "Menu" as shown in the screenshot below.

Option 2: Standalone UI Client

The following command will start the Cross vCenter Workload Migration Fling in standalone mode:

java -jar xvm-3.1.jar

You can then access the standalone client by opening a browser to localhost and port specified (default is 8443). You can always access the plugin locally whether you are using Option 1 or 2.