vSphere 6.7

Using ESXi Kickstart %firstboot with Secure Boot

06/26/2018 by William Lam 3 Comments

If you install ESXi via a Kickstart script and make use of the %firstboot option to execute commands on the first boot of the ESXi host after installation, you should be aware of its incompatibility with the Secure Boot feature. If you install ESXi where Secure Boot is enabled, the Kickstart will install ESXi normally only execute up to the %post section. However, it will not execute the %firstboot scripts and if you look at the /var/log/kickstart.log after the host boots, you should see the following message:

INFO UEFI Secure Boot Enabled, skipping execution of /var/lib/vmware/firstboot/001.firstboot_001

If you have Secure Boot enabled, %firstboot is not supported. The reason for this is Secure Boot mandates only known tardisks which can hold executable scripts, and a kickstart script is an unknown source so it can not run when Secure Boot is enabled. If you wish to continue using %firstboot scripts, the only option is to disable Secure Boot and then re-enable it after the installation. A preferred alternative is to convert your %firstboot logic into an external script which can then be applied using the vSphere API (recommended method) and this way you can still customize your ESXi host after the initial installations. I have already filed an internal documentation bug to add a note regarding Secure Boot and %firstboot, hopefully that will roll out with the net documentation refresh.

How to simulate Persistent Memory (PMem) in vSphere 6.7 for educational purposes?

05/24/2018 by William Lam 2 Comments

A really cool new capability that was introduced in vSphere 6.7 is the support for the extremely fast memory technology known as non-volatile memory (NVM), also known as persistent memory (PMem). Customers can now benefit from the high data transfer rate of volatile memory with the persistence and resiliency of traditional storage. As of this blog post, both Dell and HP have Persistent Memory support and you can see the list of supported devices and systems here and here.

PMem can be consumed in one of two methods:

Virtual PMem (vPMem) - In this mode, the GuestOS is actually PMem-aware and can consume the physical PMem device on the ESXi host as standard, byte-addressable memory. In addition to using an OS that supports PMem, you will also need to ensure that the VM is running the latest Virtual Hardware 14
Virtual PMem Disks (vPMemDisk) - In this mode, the GuestOS is NOT PMem-aware and does not have access to the physical PMem device. Instead, a new virtual PMem hard disk can be created and attached to a VM. To ensure the PMem hard disk is placed on the PMem Datastore as part of this workflow, a new PMem VM Storage Policy will be applied automatically. There are no additional GuestOS or VM Virtual Hardware requirement for this scenario, this is great for legacy OS that are not PMem-aware

Customers who may want to familiarize themselves with these new PMem workflows, especially for Automation or educational purposes, could definitely benefit from the ability to simulate PMem in their vSphere environment prior to obtaining a physical PMem device. Fortunately, this is something you can actually do if you have some additional spare memory from your physical ESXi host.

Disclaimer: This is not officially supported by VMware. Unlike a real physical PMem device where your data will be persisted upon a reboot, the simulated method will NOT persist your data. Please use this at your own risk and do not place important or critical VMs using this method.

Quick Tip – What hashing algorithm is supported for ESXi Kickstart password?

05/21/2018 by William Lam Leave a Comment

I had a question the other day asking whether the encrypted password which can be specified within an ESXi Kickstart file (denoted by the --isencrypted flag) can use a different hashing algorithm other than MD5? The answer is absolutely yes. In fact, MD5 as a default hashing algorithm has NOT been used for a number of releases, probably dating back to classic ESX (you know, the version that had the Service Console).

For all recent releases of ESXi including 5.5 to 6.7, the default hashing algorithm has been SHA512 for quite some time now. Below are two ways in which you can check which default hashing algorithm is currently being used:

Option 1 - SSH to ESXi host and take a look at /etc/pam.d/passwd

Option 2 - SSH to ESXi host and take a look at /etc/shadow and look at the field prior to the salt.

As a reference:

$1$ - MD5
$5$ - SHA256
$6$ - SHA512

Leveraging Instant Clone in vSphere 6.7 for extremely fast Nested ESXi provisioning

05/17/2018 by William Lam 2 Comments

The idea of "Instant Cloning" a Nested ESXi VM (running ESXi in a VM) is not a new concept. In fact, I had shared a solution back in 2015 using the private VMFork APIs. However, what has changed is the ease of consumption, primarily due to the re-architecture of Instant Clone in vSphere 6.7 (more details here and here) which resulted in a public and simplified API. Some of you might ask, why not simply clone a Nested ESXi VM or create a Link Clone? What benefit would I get by using Instant Clone?

The answer is not only speed, but the fact that the instantiated VM is fully operational and ready to start executing where as a traditional full clone or linked clone requires a full OS boot up that can take up to several minutes to deploy and configure. This may not sound like much for a small number of Nested ESXi VMs, but as you increase the number of instances, Instant Clone really shines while still maintaining speed and the instant availability of the VM. As you can imagine, this definitely opens up for some interesting use cases whether it be for personal home lab or educational purposes like VMware HOL. In addition, we also have customers who deploy Nested ESXi not only at high scale but also with a high churn rate for development purposes, think CI/CD type of a workload who can also benefit from Instant Clone.

So how fast are we talking about? Lets say you wanted to test out the latest version of VSAN in vSphere 6.7, you would normally deploy 3 Nested ESXi VMs, power them up and wait for them to be ready on the network. With Instant Clone, you can deploy three fully functional Nested ESXi VMs in just 30seconds! As the VMs are instantly available for consumption, you can start the VSAN enablement workflow immediately and even parts of that can be baked into the Instant Clone workflow. With the ease of provisioning Nested ESXi VMs, you can simply maintain a catalog of ESXi templates which are in "frozen" states and then leverage Instant Clone to deploy just-in-time Nested ESXi environments and discard them once you are done. Pretty slick if you ask me! and something I plan on using going forward.

Disclaimer: Nested ESXi is still not officially supported by VMware. Please use at your own risk.

Changes to vSphere Client Login UI customizations in vSphere 6.7

05/09/2018 by William Lam 3 Comments

For those that have customized their vSphere Client Login UI using the instructions here and here, it looks like the process can not be applied to the vSphere 6.7 release. From what I can tell, it looks like we have now consolidated the various WAR files into a single file /usr/lib/vmware-sso/vmware-sts/webapps/ROOT.war. The original contents of the websso directory, which pertains to the UI customization, is now located here. This was a fairly minor change, but something to be aware of and for details on how to persist your configuration changes, please see the instructions below.

Disclaimer: This is not officially supported by VMware. If you decide to enable this, please use at your own risk and ensure you backup all original files in case you need revert back to the original configuration.

As part of looking into this, I also had some fun incorporating a cool little animated login page directly into the vSphere UI which I had shared on Twitter yesterday. Stay tuned for more details on #vYetti 🙂

New Instant Clone Architecture in vSphere 6.7 – Part 2

04/30/2018 by William Lam 1 Comment

In the previous article, I provided an overview of the new "Parentless" Instant Clone feature which was introduced in vSphere 6.7 and some of the architectural differences between prior versions of Instant Clone. In this post, I will show you how to use the new Instant Clone feature, which is currently only available with the vSphere API.

There are two important parts when using Instant Clone:

A user defined script which runs within the GuestOS and is responsible for customizing the network identity of the Instant Clone. This script is not just limited to network configurations but can also be used to customize other OS and/or application settings.
The deployment script which runs outside of the GuestOS and instantiates new Instant Clones using the vSphere API. This script is responsible for passing in data (network configuration, OS and/or application settings) to the GuestOS which can then be accessed directly by the user defined script running within each Instant Clone for actual customization.

To demonstrate the use of the Instant Clone API, I will be using PowerCLI and have also created an Instant Clone PowerCLI module called InstantClone.psm1. However, the API is not limited to just PowerCLI and can be consumed using any of the new vSphere 6.7 SDKs. For our example, we will be exercising the creating Instant Clones from a "Frozen" Source VM workflow since that is the most efficient method, especially as you scale to larger number of Instant Clones. For those that wish to experiment with the other workflow of creating Instant Clones from a "Running" Source VM, feel free to take a look at the previous post for the required steps.

For my setup, I will be using an Ubuntu Server 16.04 system as my Source VM and I will create 30 Instant Clones from this system. Each Instant Clone will immediately be available for use and will be fully customized and reachable from a network perspective (e.g. unique IP and MAC Address). Here is a screenshot of one of my deployments to give you an idea of how blazing fast Instant Clone is 🙂

Here is a short video that I just recorded that demos the Instant Clone APIs using PowerCLI (this demo is using an Ubuntu VM configured with 2GB memory where as the original screenshot was 1GB)

Demo of Instant Clone in vSphere 6.7 from lamw on Vimeo. [Read more...] about New Instant Clone Architecture in vSphere 6.7 – Part 2