William Lam

Thunderbolt 3 enclosures with (Single, Dual & Quad) M.2 NVMe SSDs for ESXi

06/03/2019 by William Lam 9 Comments

Thunderbolt 3 (TB3) and eventually USB 4 is a really fascinating technology and I believe it still has so much untapped potential, especially when looking at Remote/Branch Office (ROBO), Edge and IoT types of deployments. TB3 was initially limited to Apple-based platforms, but in the last couple of years, adoption has been picking up across a number of PC desktop/laptops including the latest generations of Intel NUCs which are quite popular for vSphere/vSAN/NSX Home Labs. My hope with USB 4 is that in the near future, we will start to see servers with this interface show up in the datacenter 🙂

In the mean time, I have been doing some work with TB3 from a home lab standpoint. Some of you may have noticed my recent work on enabling Thunderbolt 3 to 10GbE for ESXi and it should be no surprise that the next logical step was TB3 storage. Using a Thunderbolt interface to connect to external storage, usually Fibre Channel is something many of our customers have been doing for quite some time. In fact, I have a blog post from a few years back which goes over some of the solutions customers have implemented, the majority use case being Virtualizing MacOS on ESXi for iOS/MacOS development. These solutions were usually not cheap and involved a sizable amount of infrastructure (e.g. storage arrays, network switches, etc) but worked very well for large vSphere/MacOS based environments.

Connecting to NSX-T Policy API using NSX-T Private IP in VMC

05/30/2019 by William Lam 1 Comment

As explained in my Getting started with NSX-T Policy API in VMware Cloud on AWS (VMC) article, there are two ways in which you can interact with the NSX-T Policy API in VMC. The initial method is with the NSX-T Reverse Proxy which designed for initial setup including Edge Firewall and connectivity configuration (VPN/Direct Connect). Once you have enabled remote access from your network to the SDDC, you can continue using the reverse proxy method or you can connect directly to the NSX-T Manager via its private IP Address.

So how do you actually connect to the NSX-T Manager using its private IP? To be honest, this was not something I had to do before as I really like the simplicity of the reverse proxy but since this came up today in one of our VMC Slack channels, I figured I take a closer look.

Automating VMware Site Recovery (DRaaS) with VMC

05/28/2019 by William Lam Leave a Comment

VMware Site Recovery provides customers with an easy and on-demand Disaster Recovery as-as-Service (DRaaS) for both their on-premises and VMware Cloud on AWS (VMC) workloads. DRaaS is one of the most popular VMware Cloud Service as it literally enables customers to setup a fully functional target DR site with just a click of a button, no joke!

As simple as this is, customers are also interested in automating the setup of DRaaS, especially for testing purposes. A couple of weeks back, I had a customer reach out asking if it was possible to automate the setup of DRaaS?

Funny enough, I just had this conversation a few weeks prior with the PM/Engineer and was in the process of looking into this. It turns out we do indeed have DRaaS Service API, which is what the VMC Console uses, however we had not publicized the API nor showed how to actually get more information about it. Do not worry, I have already feature request with the PM to not only get this published publicly but also show up as part of the VMC Developer Center API Explorer 🙂

NSX-T Policy API Explorer, Docs and Sample Updates for VMC

05/20/2019 by William Lam Leave a Comment

The VMware Cloud on AWS (VMC) in-product API Explorer is something that I use on a regular basis, especially for executing a quick tasks without needing to write a single line of code or for documentation purposes to understand the required parameter to a specific API. Today, the VMC API Explorer includes both the VMC Service and Cloud Services Platform API, but one key API that has been missing is the inclusion of the NSX-T Policy API.

With the release of VMC 1.7 which just came out a few days ago, customers will now be able to also access the NSX-T Policy API directly from within the VMC API Explorer! In addition to adding to the VMC API Explorer, the NSX team has also enhanced their Swagger JSON and documentation to provide a better developer experience when working with NSX-T in VMC.

NSX-T Policy API Explorer

The new NSX-T Policy API Explorer is only available for SDDCs running 1.7, which can be newly created SDDC or an SDDC that has been upgraded to 1.7. In addition, since there still SDDC running NSX-V, to view the new API Explorer, you will need to select an NSX-T 1.7 SDDC and then two new NSX-T API references will be available: NSX VMC Policy API and NSX VMC AWS Integrations API

Quick Tip – Crucial NVMe SSD not recognized by ESXi 6.7

05/19/2019 by William Lam 18 Comments

If you own or have recently purchased Crucial NVMe SSD such as CT1000P1SSD8 (1TB M.2 NVMe SSD) or CT500P1SSD8 (500GB M.2 NVMe SSD), please be aware that these devices may no be recognized by ESXi after upgrading to the latest release. Thanks to Pete Lindley, (OCTO for End-User Computing), who reached out last week regarding the observation as well as a workaround for the problem. This was also quite timely as I recently purchased a Crucial M.2 NVMe SSD and would have also ran into this problem.

It turns out these Crucial devices were working fine while running on ESXi 6.5 Update 2 but was no longer recognized in latest release of ESXi 6.7 Update 2. It is unclear whether support for these SSDs were removed intentionally or unintentionally, but in either case, these devices are not officially on VMware's Hardware Compatibility List (HCL).

UPDATE (05/23/19) - After speaking with a few folks who took a closer look, the issue is due to the fact that we added support for NVMe 1.3 spec in latest ESXi 6.7 Update 2 release, but because these are "consumer" devices, they did not conform to the latest specification and hence the driver is unable to claim the device. This is another good reminder when using components not on VMware HCL, this is always a risk from a home lab perspective. In general, I know Samsung and Intel NVMe SSD usually works quite well without issues but always good to do some research. I think Engineering is looking to see if there are other workarounds for the future, but for now, you can use the workaround below.

The easy workaround that Pete found was to simply replace the NVMe driver from ESXi 6.7 Update 2 (1.2.2.27-1vmw.670.2.48.13006603) with one found in ESXi 6.5 Update 2 (1.2.1.34-1vmw.650.2.50.8294253). To so do, simply copy nvme.v00 to /bootbank from either an existing ESXi 6.5 Update 2 system or directly from the ISO. Please note, any future updates or patches to the ESXi host will most likely override the updated driver.

Deploying a vCenter Server Appliance (VCSA) in VMC?

05/07/2019 by William Lam Leave a Comment

During the VMware Cloud on AWS (VMC) Customer Summit last week, I received an interesting question from one of our field folks on whether it was possible to deploy a vCenter Server Appliance (VCSA) to VMC for testing purposes? This was not a use case I had heard of before but it would enable the team to quickly prototype a solution to demonstrate to their customer.

I figured this should work and you should be able to just point the VCSA Installer to an existing VMC environment for deployment. It was mentioned that they had attempted to run the installer but ran into a permission issue where it required a full administrator role, which in VMC, customers do not have.

In taking a look for myself in one of my VMC environment using the VCSA UI Installer, I did indeed run into the same permission issue as shown in the screenshot below.

User has no administrative privileges

This surprised me as the VCSA Installer does not actually require administrative privileges to deploy a VCSA, just the privileges for deploying a regular VM. I captured the logs and screenshots and have shared this with the VCSA PM for further investigation.