virtuallyGhetto

esxi 5.5

Automatically retrieve CVE CVSS score for all ESXi security bulletins 

by 9 Comments

I always enjoying learning new things, especially when it is outside of my immediate domain expertise and if I can thrown in some Automation to help solve a solution, it is a win for everyone. I bring this up because, yesterday I had noticed an interesting question from one of our field folks where their customer is looking to implement a process for applying ESXi security patches to help determine compliance timeline (e.g. when a specific security update will be applied to infrastructure).

To do this, the customer would like to use the Common Vulnerability Scoring System (CVSS) score which ranges from 0-10, 0 being low and 10 being high. The CVSS score is part of the Common Vulnerabilities and Exposures (CVE) which is also referenced for every ESXi security patch (bulletin) that is published by VMware. The question that came up was how easily it would be to determine the CVSS score for a given ESXi security patch. First, I will outline the "manual" process and once that is understood, I will demonstrate an automated solution which customers can take advantage of to easily retrieve this information for all ESXi security patches.

[Read more...] about Automatically retrieve CVE CVSS score for all ESXi security bulletins 

Functional USB-C Ethernet Adapter for ESXi 5.5, 6.0 & 6.5

by 22 Comments

While attending an offsite this week, there were some discussions amongst my colleagues about their new Apple Mac Pro and its USB-C only ports. The discussion was completely unrelated to work, however that did get me thinking about the USB-C peripheral market and specifically their ethernet adapters. While searching online, I came across several new USB-C to gigabit ethernet adapters that were now available and one in particular that was very interesting, was the Plugable USB-C to 10/100/1000 Gigabit Ethernet LAN Network Adapter. What caught my eye about this specific network adapter was that it uses the exact same ASIX AX88179 driver as my USB 3.0 to Ethernet Adapter ESXi VIBs were built off of! There was a good chance this might just work.


As you can probably guess, I was pretty excited and quickly ordered one of the Plugable USB-C Ethernet Adapters. The next challenge was getting access to a system that has a USB-C port. After asking around, I finally got my hands on a Dell XPS 13 which has a USB-C port that I could use for a few days. Funny enough, the Dell laptop only has USB 3.0 and USB-C ports, so the first challenge was to disable Secure Boot since I had built a custom ESXi 6.5 image that included my USB 3.0 Ethernet Adapter VIB. Below are the ESXi VIBs or offline bundles that will be required for this solution.

UPDATE (02/12/19) - A new VMware Native Driver for USB-based NICs has just been released, please use this driver going forward.

Please see this blog post for more detailed instructions on installing the VIB as well as accessing the vusbX pNIC.

Disclaimer: This is not officially supported by VMware. Use at your own risk.

Once I got ESXi up and running, I was disappointed to see that the USB-C device was not being detected. I had tried a few more things but nothing worked and I decided to sleep on it. The next morning, I realize maybe there was some additional settings that needed to be tweaked in the BIOS. With a bit of trial/error, I found out that you needed to enable the "Thunderbolt Boot Support" which apparently is disabled by default, at least on this Dell system. Below is a screenshot of the BIOS USB/Thunderbolt Settings and this was the only change required from the system defaults.


Once I rebooted, I immediately saw the link up on the USB-C device while ESXi was starting up 😀

[Read more...] about Functional USB-C Ethernet Adapter for ESXi 5.5, 6.0 & 6.5

How to Netboot install ESXi onto Apple Mac Hardware?

by 11 Comments

The ability to perform an ESXi Scripted Installation over the network has been a basic capability for non-Apple hardware customers since the initial release of classic ESX. However, for customers who run ESXi on Apple Mac Hardware (first introduced in vSphere 5.0), being able to remotely boot and install ESXi over the network has not been possible and customers could only dream of this capability which many of us have probably taken for granted.

Unlike traditional scripted network installations which commonly uses Preboot eXecution Environment (PXE), Apple Mac Hardware actually uses its own developed Boot Service Discover Protocol (BSDP) which ESXi and other OSses do not support. In addition, there are very few DHCP servers that even support BSDP (at least this may have been true 4 years ago when I had initially inquired about this topic). It was expected that if you were going to Netboot (equivalent of PXE/Kickstart in the Apple world) a server that you would be running a Mac OS X system. Even if you had set this up, a Netboot installation was wildly different from a traditional PXE installation and it would be pretty difficult to near impossible to get it working with an ESXi image. With no real viable solution over the years, it was believed that a Netboot installation of ESXi onto Mac Hardware just may not be possible.

tl;dr - If you are interested in the background to the eventual solution, continue reading. If not and you just want the goods, jump down a bit further. Though, I do think it is pretty interesting and worth getting the full context 🙂

[Read more...] about How to Netboot install ESXi onto Apple Mac Hardware?

Functional USB 3.0 Ethernet Adapter (NIC) driver for ESXi 5.5 & 6.0

by 81 Comments

Earlier this month I wrote an article demonstrating a functional USB ethernet adapter for ESXi 5.1. This was made possible by using a custom built driver for ESXi that was created over three years ago by a user named Trickstarter. After having re-discovered the thread several years later, I had tried reaching out to the user but concluded that he/she has probably moved on given the lack of forum activity in the recent years. Over the last few weeks I have been investigating to see if it was possible to compile a new version of the driver that would function with newer versions of ESXi such as our 5.5 and 6.0 release.

UPDATE (02/12/19) - A new VMware Native Driver for USB-based NICs has just been released for ESXi 6.5/6.7, please use this driver going forward. If you are still on ESXi 5.5/6.0, you can continue using the existing driver but please note there will be no additional development in the existing vmklinux-based driver.

UPDATE (01/22/17) - For details on using a USB-C / Thunderbolt 3 Ethernet Adapter, please see this post here.

UPDATE (11/17/16) - New driver has been updated for ESXi 6.5, please find the details here.

After reaching out to a few folks internally, I was introduced to Songtao Zheng, a VMware Engineer who works on some of our USB code base. Songtao was kind enough to provide some of assistance in his spare time to help with this non-sanction effort that I was embarking on. Today, I am please to announce that we now have a functional USB ethernet adapter driver based on the ASIX AX88179 that works for both ESXi 5.5 and 6.0. This effort could not have been possible without Songtao and I just want to say thank you very much for all of your help and contributions. I think it is safe to say that the overall VMware community also thanks you for your efforts. This new capability will definitely enable new use cases for vSphere home labs that were never possible before when using platforms such as the Intel NUC or Apple Mac Mini for example. Thank you Songtao! I would also like to extend an additional thank you to Jose Gomes, one of my readers, who has also been extremely helpful with his feedback as well as assistance on testing the new drivers.

Now, Before jumping into the goods, I do want to mention there are a few caveats to be aware of and that I think it is important to understand them before making any purchasing decisions.

  • First and foremost, this is NOT officially supported by VMware, use at your own risk.
  • Secondly, we have observed there is a substantial difference in transfer speeds between Transmit (Egress) and Receive (Ingress) traffic which may or may not be acceptable depending on your workload. On Receive, the USB network adapter is performing close to a native gigabit interface. However, on Transmit, the bandwidth mysteriously drops by ~50% which includes very inconsistent transfer speeds. We are not exactly sure why this is the case, but given ESXi does not officially support USB based ethernet adapters, it is possible that the underlying infrastructure was never optimized for such devices. YMMV
  • Lastly, for the USB ethernet adapter to properly function, you will need a system that supports USB 3.0 which kind of makes sense for this type of a solution to be beneficial in the home lab. If you have a system with USB 2.0, the device will probably not work at least from testing that we have done.

Note: For those interested in the required source code changes to build the AX88179 driver, I have published all of the details on my Github repo here.

Disclaimer: In case you some how missed it, this is not officially supported by VMware. Use at your own risk.

Without further ado, here are the USB 3.0 gigabit ethernet adapters that are supported with the two drivers:

Note: There may be other USB ethernet adapters that uses the same chipset which could also leverage this driver but these are the only two that have been verified.

usbnic
Here are the ESXi driver VIB downloads:

Note: Although the drivers were compiled against a specific version of ESXi, they should also work on the same major version of ESXi, but I have not done that level of testing and YMMV.

Verify USB 3.0 Support

As mentioned earlier, you will need a system that is USB 3.0 capable to be able to use the USB ethernet adapter. If you are unsure, you can plug in a USB 3.0 device and run the following command to check:

lsusb

usb3nic-0
What you will be looking for is an entry stating "Linux Foundation 3.0 root hub" which shows that ESXi was able to detect a USB 3.0 port on your system. Secondly, look for the USB device you just plugged in and ensure the "Bus" ID matches that of the USB 3.0 bus. This will tell you if your device is being claimed as a USB 3.0 device. If not, you may need to update your BIOS as some systems may have USB 2.0 enabled by default like earlier versions of Intel NUC as desribed here. You may also be running pre-ESXi 5.5 which did not support USB 3.0 as mentioned here, so you may need to upgrade your ESXi host to at least 5.5 or greater.

Install Driver

You can either install the VIB directly onto your ESXi host or by creating a custom ESXi ISO that includes the driver using a popular tool like ESXi Customizer by Andreas Peetz.

To install the VIB, upload the VIB to your ESXi host and then run the following ESXCLI command specifying the full path to the VIB:

esxcli software vib install -v /vghetto-ax88179-esxi60u2.vib -f

usb3nic-1
Lastly, you will need to disable the USB native driver to be able to use this driver. To do so, run the following command:

esxcli system module set -m=vmkusb -e=FALSE

You will need to reboot for the change to go into effect.

To verify that the USB network adapter has been successfully claimed, run either of the following commands to list your physical NICs:

esxcli network nic list
esxcfg-nics -l

usb3nic-2
To add the USB uplink, you will need to either use the vSphere Web Client or ESXCLI to add the uplink to either a Virtual or Distributed Virtual Switch.

usb3nic-3
To do so using ESXCLI, run the following command and specify the name of your vSwitch:

esxcli network vswitch standard uplink add -u vusb0 -v vSwitch0

Uninstall Driver

To uninstall the VIB, first make sure to completely unplug the USB network adapter from the ESXi first. Next, run the following ESXCLI command which will automatically unload the driver and remove the VIB from your ESXi host:

esxcli software vib remove -n vghetto-ax88179-esxi60u2

Note: If you try to remove the VIB while the USB network adapter is still plugged in, you may hang the system or cause a PSOD. Simply reboot the system if you accidentally get into this situation.

Troubleshooting

If you are not receiving link on the USB ethernet adapter, it is most likely that your system does not support USB 3.0. If you find the a similar message like the one below in /var/log/vmkernel.log then you are probably running USB 1.0 or 2.0.

2016-03-21T23:30:49.195Z cpu6:33307)WARNING: LinDMA: Linux_DMACheckConstraints:138: Cannot map machine address = 0x10f5b6b44, length = 2 for device 0000:00:1d.7; reason = address exceeds dma_mask (0xffffffff))

Persisting USB NIC Configurations after reboot

ESXi does not natively support USB NIC and upon a reboot, the USB NICs are not picked up until much later in the boot process which prevents them from being associated with VSS/VDS and their respective portgroups. To ensure things are connected properly after a reboot, you will need to add something like the following in /etc/rc.local.d/local.sh which re-links the USB NIC along with the individual portgroups as shown in the example below.

You will also need to run /sbin/auto-backup.sh to ensure the configuration changes are saved and then you can issue a reboot to verify that everything is working as expected.

Summary

For platforms that have limited built-in networking capabilities such as the Intel NUC and Apple Mac Mini, customers now have the ability to add additional network interfaces to these systems. This will now open up a whole new class of use cases for vSphere based home labs that were never possible before, especially with solutions such as VSAN and NSX. I look forward to seeing what our customers can now do with these new networking capabilities.

Additional Info

Here are some additional screenshots testing the dual USB 3.0 ethernet adapter as well as a basic iPerf benchmark for the single USB ethernet adapter. I was not really impressed with the speeds for the dual ethernet adapter which I had shared some more info here. Unless you are limited on number of USB 3.0 ports, I would probably recommend just sticking with the single port ethernet adapter.

usb3nic-5
usb3nic-6

iPerf benchmark for Ingress traffic (single port USB ethernet adapter):
usb3nic-7
iPerf benchmark for Egress traffic (single port USB ethernet adapter):
usb3nic-8

Heads up: ESXi 5.x & 6.0 unable to detect newer Apple Mac Pro 6,1 local SSD Device

by 12 Comments

Over the last couple of weeks there have been several reports coming in from customers that the local SSD device found in newer Apple Mac Pro 6,1 were no longer being detected by ESXi. Starting with ESXi 5.5 Patch03 and ESXi 6.0, the Apple Mac Pro 6,1 was officially supported but it looks like the latest versions of the Mac Pro 6,1 that are being shipped contain  a slightly different local SSD device which is not recognized by ESXi.

This was not the first time that this has happened, when the 2014 Mac Mini were first released, they too had a similar issue in which a custom VIB was required to get the internal device to get recognized by ESXi. There is an internal bug (PR 1487494) that is currently tracking the issue and if you are also experiencing this problem, please file an SR and have the GSS Engineer attach your case to this bug.

In the meantime, there is an unofficial workaround which was discovered by one of my readers (Mr. Spock) that by installing the community SATA-XACHI VIB over on Andreas Peetz VIB Depot site, both ESXi 5.5 and ESXi 6.0 will then recognize the local SSD Device. You will need to either use VMware Image Builder or Andreas ESXi Customizer tool to create a custom image if you decide to install ESXi directly on the local SSD device. I personally would recommend installing ESXi on USB device, this would allow you to install the VIB as a post-installation and not requiring a custom ESXi image.

How to create custom ESXi boot menu to support multiple Kickstart files?

by 23 Comments

I recently received a question from one of my readers who was looking to migrate from ESXi 4.1 to newer version and one of the challenges they faced was around their ESXi scripted installs, better known as ESXi Kickstart. Previously, they had relied on using a custom syslinux boot menu to be able to select a specific Kickstart configuration file that resided locally on a bootable ESXi Image (USB, ISO or CDROM) as a PXE/DHCP environment was not allowed in their environment. There was a small change to how ESXi boot files were reference between ESXi 4.x and ESXi 5.x/6.x and a new boot.cfg configuration is now used which I had written about here with respect to scripted installs when ESXi 5.0 was first released.

Luckily, even with these changes one can still use a custom menu with ESXi 5.x/6.x and be able to select a specific Kickstart configurations based on user input. Here is a screenshot example of a custom ESXi Image that I built providing three different install options that could be selected which would map to three different Kickstart configurations which can be either local to the boot media or can also be retrieved remotely.

bootable-esxi-image-with-multiple-kickstart-option
The first thing you should be aware of if you plan to boot the custom ESXi Image from local media such as USB, CDROM or ISO is that the path to the Kickstart file must be in all UPPER CASE which is mentioned in this VMware KB 1026373. The next caveat that I found in my testing is that if you plan to store the local Kickstart files inside of a directory within the ESXi Image, the name of the directory can not be too long. I would recommend using "ks" as "kickstart" apparently was too long.

After you have extracted the contents of an ESXi ISO which you have downloaded, you will want to create a root directory called "ks" which will contain the different Kickstart configuration files. Here is an example of what structure look like:

ks
├── ks1.cfg
├── ks2.cfg
└── ks3.cfg

Next, you will need to edit the isolinux.cfg file which comes by default within the ESXi ISO. This is where you will add the different Kickstart options that a user will be able to select from. In this first example, we will look at referencing the Kickstart files locally on the media which can be either USB or CDROM and you will need to ensure you specify the right boot option as shown here in the VMware documentation. The path to the Kickstart file needs to be appended to the line that contains boot.cfg reference and you must ensure you include "+++" at the end of that line.

Here is an example of referencing a Kickstart file that lives on a USB device under this path /ks/ks.cfg:

APPEND -c boot.cfg ks=usb:/KS/KS.CFG +++

Here is an example of my isolinux.cfg for the boot menu that I have shown above which provides three different options mapping to three different Kickstart configuration files:

As I mentioned earlier, the Kickstart configuration file can either be retrieved locally or it can also be retireved remotely using one of the following supported protocols: http, https, ftp & nfs as shown here in the VMware documentation.

Here is an example of isolinux.cfg for a boot menu which references both a local kickstart as well as one that remotely lives on a web server:

For additional ESXi Kickstart resources and example, be sure to check out my pages here.