vib

Neat way of installing or updating any VIB using just the ESXi Embedded Host Client

11/10/2015 by William Lam 4 Comments

A couple of months back I had tossed out an idea on Twitter asking if others would like to see an automatic update mechanism built into the ESXi Embedded Host Client which would allow users to easily update to newer releases of the Fling versus the current method which requires copying the VIB and then running command in the ESXi Shell.

Wonder if its just me,but would others like to see an automatic update mechanism in the ESXi Embedded Host Client UI? pic.twitter.com/R9KFMOE4zu

— William Lam (@lamw) August 26, 2015

To no surprise, the feedback was an astounding yes! Literally within a couple of hours, Etienne Le Sueur, one of the two VMware Engineers working on the Fling shared a screenshot that demonstrated that this would possible. The first release of this feature would simply ask for the URL to the updated ESXi Embedded Host Client VIB and this was included in the v3 release of the Fling.

One additional tidbit that Etienne had shared was that the way this feature was implemented, it was not only limited to Embedded Host Client VIB but you could do this for any ESXi VIB. This is done by using the vSphere API and calling into the InstallHostPatchV2_Task() method which allows you to install or update an ESXi VIB from a URL source. Most recently, there a twitter conversation between myself, Etienne and Christian Mohn on how this capability could be further extended to include updating ESXi itself which can either be from an Image Profile or offline bundle. For those with a detailed eye, you may have noticed that the same API method can also support an offline bundle URL which would make this possible. As of right now, the feature is actually included in an internal build of the Embedded Host Client, but perhaps we will see this in a future update of the Embedded Host Client? 😉

Going back to the original topic of this blog post, to use the VIB install/update mechanism, you would need to first upload the ESXi VIB to an HTTP Server and then specify the URL. This is fine if you have an existing HTTP Server but if you do not, it is sort of a pain and though there are other methods like uploading directly to the ESXi's python based HTTP Server as mentioned by Christian, it would still require using something like SCP which is an additional step. My initial goal and hope was to be able to install or update an ESXi VIB or ESXi itself using purely the Embedded Host Client. This would keep things simple and not require things like SSH to be enabled on the ESXi host.

After a bit of brainstorming with Etienne, he actually found a super clever way of accomplishing this after our conversation. The idea I had was to make use of the ESXi Datastore to store the VIB which can be uploaded through the Embedded Host Client. By default, there is also an HTTP based interface to the datastore, however it requires authentication which would be a problem. The neat idea that was suggested was why not try to specify the local VMFS path to the ESXi VIB (e.g./vmfs/volumes/datastore1/my.vib)? It turns out that this actually works as well!

With just two easy steps, you can now upload an ESXi VIB and then install/update all using just the Embedded Host Client with no additional dependencies

Step 1 - Navigate to the Datastore section in the Embedded Host Client and then upload the ESXi VIB that you wish to install or update.

Step 2 - To install/update the VIB, click on Help in the upper right hand corner of the Embedded Host Client and select the "Update" option. Specify the local VMFS path to ESXi VIB and then click on Update to apply.

Note: A reboot may be required after applying a new VIB. It will be your responsibility to shutdown the VMs and reboot the ESXi host for changes to go into effect if required.

At this point, you should also see a task kicked off applying the VIB. If there are any errors thrown, they will be displayed else you should see a successful task completion. For educational purposes, here is a quick screenshot of /var/log/esxupdate.log showing the VIB being applied, this can be used for further troubleshooting if required.

Hope you enjoyed this neat little trick and with just two easy steps you can install or update any ESXi VIB using the Embedded Host Client without additional dependencies or enabling SSH on the ESXi host.

How to create a custom ESXi VIB to execute a script once?

07/16/2015 by William Lam 14 Comments

Earlier this week I received a question from a customer who was interested in creating a custom ESXi VIB that could execute a specific script within the payload as part of the installation and only running it once. This was a fairly common request that I have seen in the past and as far as I knew, this type of behavior was not possible. What was unique about this particular custom inquiry was that they mentioned they found some references on this being possible. Being the curious person I am, I decided to take another look and reach out to a few folks in Engineering.

After speaking with one of the Engineers familiar with our VIB specification, to my surprise I learned that this type of behavior was actually indeed possible but was not very well documented externally. The typical use case for this is usually to apply certain configurations which are not exposed through the traditional ESXi interfaces like the vSphere API/CLI which includes Host Profiles. One example that comes to mind is being able to deploy a VIB across several hundred ESXi hosts that could configure a specific ESXi Advanced Setting which might be marked hidden. Another example would be updating a configuration file like /etc/vmware/config or running a series of ESXi Shell commands which can not available in the vSphere API and require the use of SSH and the ESXi Shell. There are many other examples, but this should give you an idea of some of the common use cases that I have heard from customers.

Here is what is required to execute a specific script as part of the VIB installation:

Created an "init" script which needs to be stored under /etc/init.d as part of your VIB payload (e.g. /etc/init.d/lamw-special-vib)
You must ensure that the "live-install-allowed" param is set to true in your VIB's descriptor.xml file (this assumes the changes can be applied without requiring reboot to take affect)
The init script will need to parse special keywords passed as command-line arguments

Here is a table showing the pair of special keywords that are passed to the init script as command-line arguments during a VIB install, upgrade or remove:

Arg1	Arg2	VIB Operation
start	install	VIB install
start	upgrade	VIB upgrade
stop	remove	VIB removal

It is up to creator of the init script to handle the different VIB operations by parsing the command-line arguments which would then determine the operations that would get executed within the script. This not only allows you to control the commands that are executed during an installation of a VIB but it also allows you specify the commands to run upon the removal of a VIB which is quite handy for properly cleaning up an uninstall. In addition, since these special keywords are not passed as part of the ESXi boot up process, the commands within the script will not execute and ensures it only runs once during the install.

Here is very simple shell script (you could also do this in Python as well) which demonstrates how to handle the three different types of VIB operations and then uses the "logger" utility to write some output to syslog:

#!/bin/sh

ARG1=$1
ARG2=$2

if [[ "${ARG1}" == "start" ]] && [[ "${ARG2}" == "install" ]]; then
 # commands to run go here #
 /bin/logger "William's custom script ran on start-install"
elif [[ "${ARG1}" == "start" ]] && [[ "${ARG2}" == "upgrade" ]]; then
 /bin/logger "William's custom script ran on start-upgrade"
 # commands to run go here #
elif [[ "${ARG1}" == "stop" ]] && [[ "${ARG2}" == "remove" ]]; then
 /bin/logger "William's custom script ran on stop-remove"
 # commands to run go here #
fi

#!/bin/sh

ARG1=$1

ARG2=$2

if [[ "${ARG1}" == "start" ]] && [[ "${ARG2}" == "install" ]]; then

# commands to run go here #

/bin/logger "William's custom script ran on start-install"

elif [[ "${ARG1}" == "start" ]] && [[ "${ARG2}" == "upgrade" ]]; then

/bin/logger "William's custom script ran on start-upgrade"

# commands to run go here #

elif [[ "${ARG1}" == "stop" ]] && [[ "${ARG2}" == "remove" ]]; then

/bin/logger "William's custom script ran on stop-remove"

# commands to run go here #

As you can see, this allows you to perform a variety of tasks through the use of a custom ESXi VIB which is a great way to be able to roll out a set of changes that may not be possible using either the vSphere API or Host Profiles for example. Best of all, this solution does not require the use of SSH which is great since many customers already disable this by default.

For more information on creating a custom VIB, be sure to check out my blog post here and you can even use this Docker image I created for building custom ESXi VIBs.

ghettoVCB VIB & offline bundle for ESXi

05/28/2015 by William Lam 55 Comments

It is still amazing to see that the number of contributions and suggestions from the community continues to grow for my free and simple VM backup solution called ghettoVCB. I created ghettoVCB almost 8 years ago which now has over 1.2 million views, pretty insane if you ask me! Although I am quite busy these days which includes a new born, I still try to find time to update the script as time permits. A couple of weeks back I received an email from one of my readers who came across ghettoVCB and was quite happy with the free solution. He also had some feedback asking why I did not provide an installable VIB for ghettoVCB?

A totally valid question and the answer was quite simple. When I had first created ghettoVCB back in the classic ESX 3.x days, the concept of a VIB had not existed yet. With the release of ESXi 5.0, the idea of the VIB was introduced but it was only recently in 2012 did VMware publish a method for customers to create custom VIBs for ESXi using the VIB Author Fling. I do have to admit at one point I did think about providing a VIB for ghettoVCB, but I guess I never went through with it for whatever reason. Looking back now, this was a no-brainer to provide a simplified user experience and not to mention the benefit of having ghettoVCB installed as a VIB is that it will automatically persist on ESXi after reboots which was a challenge for new users to ESXI.

So without further ado, here is ghettoVCB provided in either a VIB or offline bundle form:

To install the ghettoVCB VIB, you just need to download the VIB and run the following ESXCLI command and specifying the full path to the VIB:

esxcli software vib install -v /vghetto-ghettoVCB.vib -f

Once installed, you will find all ghettoVCB configuration files located in:

/etc/ghettovcb/ghettoVCB.conf
/etc/ghettovcb/ghettoVCB-restore_vm_restore_configuration_template
/etc/ghettovcb/ghettoVCB-vm_backup_configuration_template

Both ghettoVCB and ghettoVCB-restore scripts are located in:

/opt/ghettovcb/bin/ghettoVCB.sh
/opt/ghettovcb/bin/ghettoVCB-restore.sh

One additional thing I would like to point out is that you can also quickly tell which version of ghettoVCB is running by inspecting the installed VIB by using the following ESXCLI command:

esxcli software vib get -n ghettoVCB

If you look at the screenshot above, I have highlighted two important pieces of information in green. The first is the "Description" property which includes the Github commit hash of the particular revision of ghettoVCB and the "Creation Date" property which contains the date of that commit. This can be handy if you want to compare it to the latest ghettoVCB repository found on Github here. Thanks again Markus for the suggestion!

For those of you who are interested in the details for creating your own ghettoVCB VIB, the next section is specifically for you. Earlier this week I blogged about a Docker Container that I have created to help build custom ESXi VIBs and as you can see now, that was the basis for us to be able to quickly create ghettoVCB VIB based on the latest revision of the script.

Step 1 - Create a new Docker Machine following the steps outlined here.

Step 2 - Login to the Docker Machine and create a new Dockerfile which contains the following:

FROM lamw/vibauthor

# Due to https://stackoverflow.com/a/49026601
RUN rpm --rebuilddb
RUN yum clean all
RUN yum update -y nss curl libcurl;yum clean all

# Download ghettoVCB VIB build script
RUN curl -O https://raw.githubusercontent.com/lamw/vghetto-scripts/master/shell/create_ghettoVCB_vib.sh && chmod +x create_ghettoVCB_vib.sh

# Run ghettoVCB VIB build script
RUN /root/create_ghettoVCB_vib.sh

CMD ["/bin/bash"]

FROM lamw/vibauthor

# Due to https://stackoverflow.com/a/49026601

RUN rpm --rebuilddb

RUN yum clean all

RUN yum update -y nss curl libcurl;yum clean all

# Download ghettoVCB VIB build script

RUN curl -O https://raw.githubusercontent.com/lamw/vghetto-scripts/master/shell/create_ghettoVCB_vib.sh && chmod +x create_ghettoVCB_vib.sh

# Run ghettoVCB VIB build script

RUN /root/create_ghettoVCB_vib.sh

CMD ["/bin/bash"]

Step 3 - Next we need to build our new Docker Container which will use the VIB Author Container by running the following command:

docker build -t lamw/ghettovcb .

The output will be quite verbose, but what you will be looking for is text highlighted in green as shown in the screenshot above. You should see the successful build of both the VIB and offline bundle as well as Docker Container showing a successful build.

Step 4 - After a successful build of our Docker Container, we can now launch the container by running the following command:

docker run --rm -it lamw/ghettovcb

Once logged into the Docker Container, you will see the generated VIB and the offline bundle for ghettoVCB as shown in the screenshot above.

If you wish to copy the VIB and offline bundle out of the Docker Container into the Docker Host, you can use Docker Volumes. I found this useful thread over on Stack overflow which I have modified to include the copying of the ghettoVCB VIB and offline bundle out to Docker Host by running the following command:

docker run -i -v ${PWD}/artifacts:/artifacts lamw/ghettovcb sh << COMMANDS
cp vghetto-ghettoVCB* /artifacts
COMMANDS

Finally, to copy the ghettoVCB VIB from the Docker Host to your desktop, we first need to identify the IP Address given to our Docker Machine by running the following command:

docker-machine ip osxdock

Currently, Docker Machine does not include a simple "scp" command so we will need to use regular scp command and specify the private SSH keys which you can find by running "docker-machine inspect [NAME-OF-DOCKER-HOST]" and connecting to our Docker Host to copy the ghettoVCB VIB by running the following command:

scp -i /Users/lamw/.docker/machine/machines/osxdock/id_rsa [email protected]:artifacts/vghetto-ghettoVCB.vib .

A Docker Container for building custom ESXi VIBs

05/26/2015 by William Lam 8 Comments

I recently had a need to create a custom ESXi VIB using the VIB Author Fling for a project that I was working on. As part of the project's deliverables, I wanted to also provide an ESXi VIB which would need to be built against any new updates for the project. Given this would be an infrequent operation, I thought why not use a Docker Container for this operation? I could just spin up a Docker Container on-demand and not have to worry about managing a local VM for just running this particular task.

With that I have created a VIB Author Docker Container which can be used to author custom ESXi VIBs. I have also made this container available on the Docker Registry for others to use which you can find more details here: https://registry.hub.docker.com/u/lamw/vibauthor/

If you already have a Docker host running, you can pull down the VIB Author Docker Container by jumping to Step 5 in the instructions below. If you do not and you are running Mac OS X like I am, you can follow the instructions below using Docker Machine and VMware Fusion to try out my VIB Author Docker Container.

Step 1 - Install the Docker client by running the following command:

brew install docker

Step 2 - Download and install Docker Machine by running the following commands:

curl -L https://github.com/docker/machine/releases/download/v0.2.0/docker-machine_darwin-amd64 > /usr/local/bin/docker-machine
chmod +x /usr/local/bin/docker-machine

Step 3 - Create Docker Machine using the VMware Fusion driver by running the following command:

docker-machine create --driver vmwarefusion osxdock --vmwarefusion-memory-size 1024
eval "$(docker-machine env osxdock)"

Note: Thanks to Omer Kushmaro for his blog post here on how to quickly get started with Docker Machine with VMware Fusion

Step 4 - Once the Docker Machine is booted up, we can now connect to it using SSH by running the following command:

docker-machine ssh osxdock

At this point, we are now logged into our Docker Machine which has both the Docker client/server running and we are now ready to pull down the VIB Author container from the Docker registry.

Step 5 - To pull down the VIB Author Docker Container that I have built, run the following command within the Docker Machine:

docker pull lamw/vibauthor

Step 6 - Once the Docker Container has been successfully downloaded, you can now run the VIB Author Container by running the following command:

docker run --rm -it lamw/vibauthor

Once logged into the VIB Author Container, you confirm that the VIB Author Fling has been installed by running the "vibauthor" command as shown in the screenshot above. In the next blog post, I will go through an example of building a custom ESXi VIB using the VIB Author Container as well as transferring the outputted files from the Docker host back onto your desktop. Stay tuned!

New VMware Fling to improve Network/CPU performance when using Promiscuous Mode for Nested ESXi

08/28/2014 by William Lam 44 Comments

I wrote an article awhile back Why is Promiscuous Mode & Forged Transmits required for Nested ESXi? and the primary motivation behind the article was in regards to an observation a customer made while using Nested ESXi. The customer was performing some networking benchmarks on their physical ESXi hosts which happened to be hosting a couple of Nested ESXi VMs as well as regular VMs. The customer concluded in his blog that running Nested ESXi VMs on their physical ESXi hosts actually reduced overall network throughput.

UPDATE (04/24/17) - Please have a look at the new ESXi Learnswitch which is an enhancement to the existing ESXi dvFilter MAC Learn module.

UPDATE (11/30/16) - A new version of the ESXi MAC Learning dvFilter has just been released to support ESXi 6.5, please download v2 for that ESXi release. If you have ESXi 5.x or 6.0, you will need to use the v1 version of the Fling as it is not backwards compat. You can all the details on the Fling page here.

This initially did not click until I started to think about this a bit more and the implications when enabling Promiscuous Mode which I think is something that not many of us are not aware of. At a very high level, Promiscuous Mode allows for proper networking connectivity for our Nested VMs running on top of a Nested ESXi VMs (For the full details, please refer to the blog article above). So why is this a problem and how does this lead to reduced network performance as well as increased CPU load?

The diagram below will hopefully help explain why. Here, I have a single physical ESXi host that is connected to either a VSS (Virtual Standard Switch) or VDS (vSphere Distributed Switch) and I have a portgroup which has Promiscuous Mode enabled and it contains both Nested ESXi VMs as well as regular VMs. Lets say we have 1000 Network Packets destined for our regular VM (highlighted in blue), one would expect that the red boxes (representing the packets) will be forwarded to our regular VM right?

What actually happens is shown in the next diagram below where every Nested ESXi VM as well as other regular VMs within the portgroup that has Promiscuous Mode enabled will receive a copy of those 1000 Network Packets on each of their vNICs even though they were not originally intended for them. This process of performing the shadow copies of the network packets and forwarding them down to the VMs is a very expensive operation. This is why the customer was seeing reduced network performance as well as increased CPU utilization to process all these additional packets that would eventually be discarded by the Nested ESXi VMs.

This really solidified in my head when I logged into my own home lab system which I run anywhere from 15-20 Nested ESXi VMs at any given time in addition to several dozen regular VMs just like any home/development/test lab would. I launched esxtop and set the refresh cycle to 2seconds and switched to the networking view. At the time I was transferring a couple of ESXi ISO’s for my kicskstart server and realized that ALL my Nested ESXi VMs got a copy of those packets.

As you can see from the screenshot above, every single one of my Nested ESXi VMs was receiving ALL traffic from the virtual switch, this definitely adds up to a lot of resources being wasted on my physical ESXi host which could be used for running other workloads.

I decided at this point to reach out to engineering to see if there was anything we could do to help reduce this impact. I initially thought about using NIOC but then realized it was primarily designed for managing outbound traffic where as the Promiscuous Mode traffic is all inbound and it would not actually get rid of the traffic. After speaking to a couple of Engineers, it turns out this issue had been seen in our R&D Cloud (Nimbus) which provides IaaS capabilities to the R&D Organization for quickly spinning up both Virtual/Physical instances for development and testing.

Christian Dickmann was my go to guy for Nimbus and it turns out this particular issue has been seen before. Not only has he seen this behavior, he also had a nice solution to fix the problem in the form of an ESXi dvFilter that implemented MAC Learning! As many of you know our VSS/VDS does not implement MAC Learning as we already know which MAC Addresses are assigned to a particular VM.

I got in touch with Christian and was able to validate his solution in my home lab using the latest ESXi 5.5 release. At this point, I knew I had to get this out to the larger VMware Community and started to work with Christian and our VMware Flings team to see how we can get this released as a Fling.

Today, I am excited to announce the ESXi Mac Learning dvFilter Fling which is distributed as an installable VIB for your physical ESXi host and it provides support for ESXi 5.x & ESXi 6.x

Note: You will need to enable Promiscuous Mode either on the VSS/VDS or specific portgroup/distributed portgroup for this solution to work.

You can download the MAC Learning dvFilter VIB here or you can install directly from the URL shown below:

To install the VIB, run the following ESXCLI command if you have VIB uploaded to your ESXi datastore:

esxcli software vib install -v /vmfs/volumes/<DATASTORE>/vmware-esx-dvfilter-maclearn-0.1-ESX-5.0.vib -f

To install the VIB from the URL directly, run the following ESXCLI command:

esxcli software vib install -v http://download3.vmware.com/software/vmw-tools/esxi-mac-learning-dvfilter/vmware-esx-dvfilter-maclearn-1.0.vib -f

A system reboot is not necessary and you can confirm the dvFilter was successfully installed by running the following command:

/sbin/summarize-dvfilter

You should be able see the new MAC Learning dvFilter listed at the very top of the output.

For the new dvFilter to work, you will need to add two Advanced Virtual Machine Settings to each of your Nested ESXi VMs and this is on a per vNIC basis, which means you will need to add N-entries if you have N-vNICs on your Nested ESXi VM.

ethernet#.filter4.name = dvfilter-maclearn
ethernet#.filter4.onFailure = failOpen

This can be done online without rebooting the Nested ESXi VMs if you leverage the vSphere API. Another way to add this is to shutdown your Nested ESXi VM and use either the “legacy” vSphere C# Client or vSphere Web Client or for those that know how to append and reload the .VMX file as that’s where the configuration file is persisted
on disk.

I normally provision my Nested ESXi VMs with 4 vNICs, so I have four corresponding entries. To confirm the settings are loaded, we can re-run the summarize-dvfilter command and we should now see our Virtual Machine listed in the output along with each vNIC instance.

Once I started to apply this changed across all my Nested ESXi VMs using a script I had written for setting Advanced VM Settings, I immediately saw the decrease of network traffic on ALL my Nested ESXi VMs. For those of you who wish to automate this configuration change, you can take a look at this blog article which includes both a PowerCLI & vSphere SDK for Perl script that can help.

I highly recommend anyone that uses Nested ESXi to ensure you have this VIB installed on all your ESXi hosts! As a best practice you should also ensure that you isolate your other workloads from your Nested ESXi VMs and this will allow you to limit which portgroups must be enabled with Promiscuous Mode.

Quick Tip – Listing Image Profiles From an ESXi Patch Using ESXCLI

06/21/2013 by William Lam 5 Comments

I was cleaning out a few of my to-do items (list just keeps getting longer everyday) this morning and there was a question that I received a few weeks back asking how to retrieve the list of Image Profiles for a given ESXi patch. This is actually quite easy and you will want to use ESXCLI.

Note: The examples shown below is using ESXCLI on the ESXi Shell, but these commands can be execute remotely as well using ESXCLI or through PowerCLI with Get-EsxCli cmdlet.

To list the available Image Profiles for an ESXi patch, run the following command (ensure you substitute the full path to your ESXi patch):

esxcli software sources profile list -d /vmfs/volumes/datastore1/ESXi510-201212001.zip

To get more details on a particular Image Profile, run the following command and specify the -p for the specific Image Profile:

esxcli software sources profile get -d /vmfs/volumes/datastore1/ESXi510-201212001.zip -p ESXi-5.1.0-20121204001-no-tools

To install/update a specific Image Profile, run the following command with the Image Profile name:

esxcli software profile update -d /vmfs/volumes/datastore1/ESXi510-201212001.zip -p ESXi-5.1.0-20121204001-no-tools

If you just want to install the ESXi patch, run the following command which will install the esx-base Image Profile by default which will include everything:

esxcli software vib update -d /vmfs/volumes/datastore1/ESXi510-201212001.zip

To check for the Image Profile you have installed on your ESXi host, run the following command:

esxcli software profile get

Here are some additional resources for ESXi patch management that may also be useful: