Patching VMware Virtual Appliances Using vamicli

Last week I wrote an article about creating offline update repository for VMware virtual appliances and demonstrating the use of the VAMI web interface for updating and upgrading a VMware virtual appliance. However, from an automation perspective, the web interface is probably not the right tool for the job and this where the vamcli can help.

Usage: vamicli [options]
may be:
  network          Network Configuration
  update           Update Management
  version          Version Information
  service          Service Management

Use vamicli to see a list of options.

The vamicli is a command-line tool that is available on all virtual appliances built using VMware Studio and it provides a subset of the functionality of the VAMI web interface. Using the "update" operation, you can check for available updates as well as performing the installation of an update.

To check for the latest update just like you would using the VAMI web interface, you would run the following command:

vamicli update --check

To install the latest update, you would run the following command:

vamicli update --install latest --accepteula

Here is a screenshot example of going through both of these commands on a VIN 1.2 virtual appliance and then upgrading to VIN 2.0:

As you can see the process is pretty straight forward and this allows you to easily automate the updates of your virtual appliances without having to resort to the VAMI web interface.

For those of you who read my previous article and wish to configure a custom update repository without using the VAMI web interface, you can add the following configuration to /opt/vmware/var/lib/vami/update/provider/provider-runtime.xml where value specifies the HTTP address to your update repository as you would configure using the VAMI web interface.

If you would like to configure additional authentication properties such as username and password, then the /opt/vmware/var/lib/vami/update/provider/provider-runtime.xml should look like the following: The password value is encoded using base64, so to generate the encoding you can use the following python snippet (where password is the password you wish to encode:

python -c "import base64; print base64.b64encode('password')"

Note: The configuration changes above go into effect immediately and you can then use vamicli to perform both check and install operations.

How To Create Offline Update Repository For VMware Virtual Appliances

Virtual appliances built from VMware Studio provides a very easy mechanism of updating or upgrading the software on the appliance by using the VAMI (Virtual Appliance Management Interface) web interface. The VAMI web interface provides three methods of updating or upgrading an appliance: online update repository hosted by the author of the appliance, CD-ROM or alternate update repository can be specified.

For VMware virtual appliances, a VMware hosted update repository is configured by default and internet connectivity to the configured URL will be required (proxy configurations are supported). However, there are environments where network connectivity to VMware's online repository is just not possible or the update repository must be hosted internally due to security requirements and this is where the third option can be used.

The process to setup your own update repository is not really documented and I have been noticing more requests from customers looking for a way to update or upgrade their VMware virtual appliances without requiring access to VMware's online repository. There are also other virtual appliances such as VCSA (vCenter Server Appliance) which ships both the update contents for an ISO and zip file which can then be used with the two other update/upgrade methods.

Though these files can be generated from VMware Studio as part of the appliance build process, the majority of the VMware virtual appliances do not provide these files for download.

I decided to research this topic a bit and look into building my own offline update repository based on the online update repository from VMware. I figured it should be fairly easy to replicate what is being hosted to a local web server that runs within your own datacenter. After some investigation, I found the  process to be pretty straight forward and only requirement is a web server that can be used to host the contents for update repository. In this example, I will show you how to build an update repository to upgrade VIN (vSphere Infrastructure) 1.2 to 2.0.

Step 1 - Login to the VAMI interface of VIN (https://[VIN-IP]:5480) and under the Update tab and make a note of the the default repository URL.

Step 2 - Download buildVARepo.sh shell script and upload that to a Linux based web server which will automatically build out our update repository.

Step 3 - The script accepts two arguments: default repository URL for a particular virtual appliance (from the previous step) and the name of the directory in which the repository will be created. In this example, I will be using the VIN repository URL and I will name the repository vin:

./buildVARepo.sh http://vapp-updates.vmware.com/vai-catalog/valm/vmw/302ce45f-64cc-4b34-b470-e9408dbbc60d/1.2.0.290.latest vin

The system that the script runs will need to have access to the URL above as it needs to download the required manifest files. Using the manifest files, parses out the package name and downloads the RPM packages to the web server using wget.

Step 4 - The result is a directory structure that will look like the following:

vin/manifest = List of XML manifest and signature files that describes the update and the path to the appliance packages
vin/package-pool = The RPM packages for the appliances for a given update

Depending on the location of where the script was executed, you may need to move it to the proper path in which your web server is configured to serve up content. You should be able to open a browser and point that to the /vin directory and view the contents.

Step 5 - We now log back into the VAMI interface and specify our update repository URL which will be http://[IP-OR-HOSTNMAE]/[REPO-NAME] and save the settings.

Step 6 - Now we head over to the Status sub-tab under Update and click on the "Check Updates" and we should see a new update for our virtual appliance. To update the appliance, we then select "Install Updates" and shortly after we should see our VIN appliance upgrade to 2.0

Note: Not all virtual appliances provide upgrades to the latest versions of the appliance, be sure to check the documentation of each individual appliances to see what is supported. 

Executing Commands During Boot Up In ESXi 5.1

There maybe certain use cases where you need to execute a command or series of commands during the boot up of an ESX(i) host and historically administrators have added commands to the /etc/rc.local which is one of the last scripts to be executed in the boot up process. However, with ESXi 5.1, the location of the file has changed from /etc/rc.local to /etc/rc.local.d/local.sh.

VMware also provides a KB2043564 that describes the locations for the various version of ESX(i):

• In ESX 3.x/4.x - You would add the command(s) to /etc/rc.d/rc.local
• In ESXi 4.x/5.0 - You would add the command(s) to /etc/rc.local
• In ESXi 5.1 - It has changed to /etc/rc.local.d/local.sh

Disclaimer: As mentioned in the VMware KB, though this is supported, it is still not recommended that you edit these files unless directed by VMware support or under special scenarios.

A question that came up recently was about automatically loading the "multiextent" VMkernel module (which is required if you wish to use the 2gbsparse VMDK format, you can find more details in this blog article here) during the boot up of an ESXi 5.1 host as this had to be done manually, even if you had enabled the module. To do so, you just need to edit the /etc/rc.local.d/local.sh and add the following command above the "exit 0".

localcli system module load -m multiextent

Note: The reason I used localcli versus ESXCLI is that hostd may not be completely ready and hence the command may fail during the boot up process. One can also loop and wait for ESXCLI to be ready, but this is another way of performing the operation.

How To Add A Tag (Log prefix) To Syslog Entries

Last year I wrote an article on how to forward vCenter Server logs to a remote syslog server using the built in syslog-ng client in the VCSA. A few weeks back, I received an interesting email from Michael White sharing details about adding a "tag" or more specifically, adding a string prefix to each syslog entry being forwarded. This was interesting as it enables a user to easily search for a specific log entry based on a "tag" and comes really in handy when you have multiple log sources being forwarded from the same host. An example of this would be the various logs from a vCenter Server such as vpxd, vws, inventoryservice, etc. which all have their own individual logs coming from the same host.

Within the Syslog-ng client configuration, you can specify the log_prefix() option and the string you wish to prefix a given log source. The tag has a specific syntax that must contain a : (colon) and a whitespace after the string (e.g. "VC_APP: ").

Using the vCenter Server as example, we could add the following tags:
After restarting the syslog-ng client for the changes to going into effect, you can head over to your syslog server to view the updated syslog entries. In the screenshot below, we can see we have log sources from both our VC_APP (vpxd.log) and VC_IS (ds.log) entries as specified in our syslog-ng client configurations.

Note: For newer versions of syslog-ng, program_override() is used instead of log_prefix(). The syntax for that would be program_override("VC_APP").

I want to thank Michael for sharing this cool tidbit!