vSphere 6.5

Quick Tip – Crucial NVMe SSD not recognized by ESXi 6.7 & 7.0

05/19/2019 by William Lam 54 Comments

If you own or have recently purchased Crucial NVMe SSD such as CT1000P1SSD8 (1TB M.2 NVMe SSD) or CT500P1SSD8 (500GB M.2 NVMe SSD), please be aware that these devices may no be recognized by ESXi after upgrading to the latest release. Thanks to Pete Lindley, (OCTO for End-User Computing), who reached out last week regarding the observation as well as a workaround for the problem. This was also quite timely as I recently purchased a Crucial M.2 NVMe SSD and would have also ran into this problem.

It turns out these Crucial devices were working fine while running on ESXi 6.5 Update 2 but was no longer recognized in latest release of ESXi 6.7 Update 2. It is unclear whether support for these SSDs were removed intentionally or unintentionally, but in either case, these devices are not officially on VMware's Hardware Compatibility List (HCL).

UPDATE (07/29/20) - Over the past few months, I have had a number of folks share feedback that using the trick mentioned below for ESXi 7.0, they have had success of ESXi detecting their NVMe SSD. I wanted to share some of the model and/or vendors that folks have reported success with. I will keep this list updated, so feel free to leave a comment below.

OWC Aura Pro X2 2TB NVMe
ADATA XPG
Sabrent

UPDATE (06/13/20) - Thanks to reader Dave, it looks like this trick also works with ESXi 7.0 but the filename has changed. Simply copy nvme.v00 VIB from the ESXi 6.5 Update 2 and replace it on ESXi 7.0 system (either live under /bootbank or part of the installer) but rename the file to nvme_pci.v00 which is the new filename for NVMe driver.

UPDATE (05/23/19) - After speaking with a few folks who took a closer look, the issue is due to the fact that we added support for NVMe 1.3 spec in latest ESXi 6.7 Update 2 release, but because these are "consumer" devices, they did not conform to the latest specification and hence the driver is unable to claim the device. This is another good reminder when using components not on VMware HCL, this is always a risk from a home lab perspective. In general, I know Samsung and Intel NVMe SSD usually works quite well without issues but always good to do some research. I think Engineering is looking to see if there are other workarounds for the future, but for now, you can use the workaround below.

The easy workaround that Pete found was to simply replace the NVMe driver from ESXi 6.7 Update 2 (1.2.2.27-1vmw.670.2.48.13006603) with one found in ESXi 6.5 Update 2 (1.2.1.34-1vmw.650.2.50.8294253). To so do, simply copy nvme.v00 to /bootbank from either an existing ESXi 6.5 Update 2 system or directly from the ISO. Please note, any future updates or patches to the ESXi host will most likely override the updated driver.

Enhancements to Hybrid Linked Mode (HLM) in VMC using the new vCenter Cloud Gateway

10/04/2018 by William Lam Leave a Comment

It has been almost a year since VMware introduced the Hybrid Linked Mode (HLM) capability, which provides customers with a consistent operating experience for managing and consuming resources from both their on-premises and VMware Cloud on AWS (VMC) environments. Feedback from customers on HLM has been fantastic, especially when new or prospective VMC customers learn about HLM for the very first time. Customers were pleasantly surprised at how seamless the experience was when consuming VMC resources, using a familiar interface, the vSphere UI.

Here is a quick recap of what HLM provides today:

HLM allows customers to link a single VMC instance to a single on-prem SSO Domain which can contain one or more vCenter Servers (Enhanced Linked Mode) while maintaining separate administrative domains (e.g. on-prem user is Administrator while VMC user is CloudAdmin only)
SSO Domains will be different between on-prem and VMC, however it is a 1:1 relationship
A trust is established where the on-prem vCenter Server trusts the incoming connections from VMC as they share the same Active Directory identity source. Data is sync'ed uni-directionally from on-prem to VMC
Can be configured at any point in the on-prem vCenter Server lifecycle, no restrictions to initial install and can easily be un-linked unlike ELM
Both Embedded & External vCenter Server deployments are supported
HLM supports different versions of vCenter Server between on-prem (6.5d+) and VMC, especially as VMC will almost always run a newer version of vSphere
Users MUST login to VMC vCenter Server for single-pane of glass management (H5 Client supported only), logging into on-prem vCenter Server will NOT show VMC vCenter Server
Roles are NOT replicated due to the restrictive access model in VMC

Automatically retrieve CVE CVSS score for all ESXi security bulletins

07/20/2018 by William Lam 9 Comments

I always enjoying learning new things, especially when it is outside of my immediate domain expertise and if I can thrown in some Automation to help solve a solution, it is a win for everyone. I bring this up because, yesterday I had noticed an interesting question from one of our field folks where their customer is looking to implement a process for applying ESXi security patches to help determine compliance timeline (e.g. when a specific security update will be applied to infrastructure).

To do this, the customer would like to use the Common Vulnerability Scoring System (CVSS) score which ranges from 0-10, 0 being low and 10 being high. The CVSS score is part of the Common Vulnerabilities and Exposures (CVE) which is also referenced for every ESXi security patch (bulletin) that is published by VMware. The question that came up was how easily it would be to determine the CVSS score for a given ESXi security patch. First, I will outline the "manual" process and once that is understood, I will demonstrate an automated solution which customers can take advantage of to easily retrieve this information for all ESXi security patches.

Using ESXi Kickstart %firstboot with Secure Boot

06/26/2018 by William Lam 5 Comments

If you install ESXi via a Kickstart script and make use of the %firstboot option to execute commands on the first boot of the ESXi host after installation, you should be aware of its incompatibility with the Secure Boot feature. If you install ESXi where Secure Boot is enabled, the Kickstart will install ESXi normally only execute up to the %post section. However, it will not execute the %firstboot scripts and if you look at the /var/log/kickstart.log after the host boots, you should see the following message:

INFO UEFI Secure Boot Enabled, skipping execution of /var/lib/vmware/firstboot/001.firstboot_001

If you have Secure Boot enabled, %firstboot is not supported. The reason for this is Secure Boot mandates only known tardisks which can hold executable scripts, and a kickstart script is an unknown source so it can not run when Secure Boot is enabled. If you wish to continue using %firstboot scripts, the only option is to disable Secure Boot and then re-enable it after the installation. A preferred alternative is to convert your %firstboot logic into an external script which can then be applied using the vSphere API (recommended method) and this way you can still customize your ESXi host after the initial installations. I have already filed an internal documentation bug to add a note regarding Secure Boot and %firstboot, hopefully that will roll out with the net documentation refresh.

Quick Tip – What hashing algorithm is supported for ESXi Kickstart password?

05/21/2018 by William Lam 2 Comments

I had a question the other day asking whether the encrypted password which can be specified within an ESXi Kickstart file (denoted by the --isencrypted flag) can use a different hashing algorithm other than MD5? The answer is absolutely yes. In fact, MD5 as a default hashing algorithm has NOT been used for a number of releases, probably dating back to classic ESX (you know, the version that had the Service Console).

For all recent releases of ESXi including 5.5 to 6.7, the default hashing algorithm has been SHA512 for quite some time now. Below are two ways in which you can check which default hashing algorithm is currently being used:

Option 1 - SSH to ESXi host and take a look at /etc/pam.d/passwd

Option 2 - SSH to ESXi host and take a look at /etc/shadow and look at the field prior to the salt.

As a reference:

$1$ - MD5
$5$ - SHA256
$6$ - SHA512

Bulk VM Migration using new Cross vCenter vMotion Utility Fling

12/20/2017 by William Lam 51 Comments

Over the last few years, I have spoken to a number of customers who have greatly benefited from the ability to live migrate Virtual Machines across different vCenter Servers that are NOT part of the same vCenter Single Sign-On (SSO) Domain, which I had first shared back in 2015 here and here. This extended capability of the Cross vCenter vMotion feature enabled customers to solve new use cases that were challenging, especially for scenarios such as Datacenter migration, consolidation or even migrating existing workloads from their current environment into new SDDC deployments such as VMware Cloud Foundation (VCF) as an example.

Although customers could initiate Cross vCenter vMotions using the vSphere API which included PowerCLI (Move-VM cmdlet was enhanced in 6.5, more details here), the overall experience was still not as friendly. This was especially true for customers who may only have a small number of VMs to migrate and prefer a UI-based interface rather than an API/CLI only option. In addition, for large number of VM migrations, there was not an easy way to perform "batch" VM migrations that was easily consumable for folks who may not have a strong background in Automation or the vSphere APIs.

Today, I am pleased to share a new VMware Fling called the Cross vCenter Migration Utility that will help simplify the consumption of initiating VM migration(s) across different vCenter Servers, especially between dispart SSO Domains where a graphical interface was not available. This solution was developed out of our VMware Cloud Foundation (VCF) Engineering group which is part of the Integrated Systems Business Unit at VMware. I had spoken to a number of folks within the group about the extended Cross vCenter vMotion capability and I was super excited when I heard they were planning to release this tool as a Fling and make it available to all customers. I was fortunately to have been involved in the project alongside the Engineering lead Vishal Gupta and we are excited that we can finally talk about this project and see how customers will be using this new tool.

UPDATE (05/07/18) - The Fling has just been updated to 2.0 with the following new features:

Added support to select individual host as the placement target
Added support for migrating VMs with shared datastore
Added clone functionality in addition to relocate
Added resource summary details for placement targets
Added a prompt to verify site thumbprint during SSL verification
Added a link to refresh vm list in the inventory view
Updated REST APIs to add operation type parameter

Cross vCenter Migration Utility Fling

Cross vCenter vMotion Requirements: KB 2106952

Download Fling here

Features

Completely UI-driven workflow for VM migration
Provides REST API for managing migration operations
Works with vCenter not part of the same SSO domain
Supports both live/cold migration of VMs
Batch migration of multiple VMs in parallel
Flexible network mappings b/w source and destination sites