How to Create Bootable ESXi 5 ISO & Specifying Kernel Boot Options

This week I helped to answer a few questions about creating your own ESXi 5 bootable ISO along with automatically using a static IP Address when the custom ISO first boots up. Although all this information is available via the vSphere documentation, it may not always be easy to put all the pieces together and thought I share the steps for others to also benefit.

You will need access to a UNIX/Linux system and a copy of the base ESXi 5 ISO image. In this example I will be using VMware vMA and VMware-VMvisor-Installer-5.0.0.update01-623860.x86_64.iso and walk you through two different configurations. We will also be referencing the vSphere documentation Create an Installer ISO Image with a Custom Installation or Upgrade Script and Kernel Boot Options.

Create ESXi 5 Bootable ISO w/Remote ks.cfg:

In this configuration, we will create a custom ESXi ISO that will boot with a static IP Address and use a remote ks.cfg (kickstart) configuration file.

Step 1 - Mount base ESXi ISO using the "mount" utility:

$ mkdir esxi_cdrom_mount
$ sudo mount -o loop VMware-VMvisor-Installer-5.0.0.update01-623860.x86_64.iso esxi_cdrom_mount

Step 2 - Copy the contents of the mounted image to a local directory called "esxi_cdrom":

$  cp -r esxi_cdrom_mount esxi_cdrom

Step 3 - Unmount the ISO after you have successfully copied it and change into the esxi_cdrom directory

$ sudo umount esxi_cdrom_mount
$ cd esxi_cdrom

Step 4 - Edit the boot.cfg and specifically the "kernelopt" line to not use the weasel installer but kickstart and also specifying the remote location of your ks.cfg. To get more details on the various kernel boot options, please take a look at the vSphere Boot Options documentation above.

You will also need to specify the static IP Address you wish to have the host automatically use when the ISO first boots up on the same line.

Step 5 - Once you have finished your edits and saved the boot.cfg, you will now change back to the parent directory and use the "mkisofs" to create your new bootable ISO. In this example, we will name the new ISO "custom_esxi.iso":

$ sudo mkisofs -relaxed-filenames -J -R -o custom_esxi.iso -b isolinux.bin -c boot.cat -no-emul-boot -boot-load-size 4 -boot-info-table esxi_cdrom/

You now have a new bootable ESXi 5 ISO called "custom_esxi.iso" which will now automatically boot up with the specified static IP Address and install based on the ks.cfg that was specified.

Create ESXi 5 Bootable ISO w/Local ks.cfg:

Similar to the above configuration, we will create a custom ESXi ISO that will boot with a static IP Address but use a local ks.cfg (kickstart) configuration file that will be included within the custom ISO.

Step 1 through 3 is exactly the same as above

Step 4 - By default, a basic ks.cfg is included in the ESXi 5 ISO in /etc/vmware/weasel/ks.cfg and we will create a custom *.tgz file that will include our ks.cfg within the ISO. First off by creating a temporary directory which will be used to store our ks.cfg:

$ mkdir -p temp/etc/vmware/weasel

Step 5 - Copy your ks.cfg file into the temp/etc/vmware/weasel:

$ cp ks_custom.cfg temp/etc/vmware/weasel

Step 6 - Create a *.tgz file containing the path to our ks.cfg using the "tar" utility. In this example, we will called it customks.tgz:

$ cd temp
$ tar cvf customks.tgz *

Step 7 -  Copy the customks.tgz from temp directory to your esxi_cdrom directory:

$ cp temp/customks.tgz esxi_cdrom

Step 8 -  Change into the "esxi_cdrom" directory and edit the boot.cfg just like the above, but we will be using the "file://" stanza to specify the path to our ks.cfg, static IP Address as well as adding our customks.tgz to the module list to ensure that it loads up which contains the actual ks.cfg file that is called in the boot.cfg.

Step 9 - Same as Step 5 above, you now just need to run the "mkisofs" utility to create your bootable ISO.

You now have a new bootable ESXi 5 ISO called "custom_esxi.iso" which will now automatically boot up with the specified static IP Address and install based on the ks.cfg that is included within the ISO.

Automating SSL Certificate Expiry Validation for vCenter Server + ESX(i) Hosts

As many of you know, it is a best practice to replace VMware's self-signed SSL certificates that are included in the vCenter Server (Windows & VCSA) and ESX(i) hosts to prevent or help reduce MiTM (Man in The Middle) attacks. If you are looking for more details on how to replace the default SSL certificates, you should take a look at the fantastic articles written by Michael Webster who details the process, provides some troubleshooting steps and best practices for SSL certificate replacement.

Replacing the default self-signed SSL certificate is just one part of the process, but you also need to check to ensure the certificates are still valid and have not expired. If you already have a process in place or a system that automatically does this for you, that is great. If you do not, you should definitely validate that your SSL certificates are valid on a regular basis.

I recently stumbled onto a nifty open source tool called ssl-cert-check that can help with validating expiration of SSL certificates found on vCenter Server(s) and ESX(i) hosts or any other SSL enabled host for that matter. This utility is just a shell script (specifically bournce shell) wrapping the common openssl utility found on most UNIX/Linux systems and does not require any login credentials to the remote hosts to validate the SSL certificate.

To use the script, you can visit the website here and download it to a system that has openssl installed (in my home lab, I used vMA).

It took me awhile to find the script, but it's located on the right side of the screen where it says "Website". You can also download it from the command-line using wget if you have direct/proxy access to the internet:

$ wget http://freecode.com/urls/353b752faa208fca12bc0091c742f764 -O ssl-cert-check

Note: Don't forget to set the execute permission on the script (chmod +x ssl-cert-check) else you will get permission denied when trying to run the script.

The script can be executed interactively by specifying the -s option for server and -p for the port. You can also specify the issuer of the certificate by using the -i option. Below is a screenshot of running the ssl-cert-check against a vCenter Server:

You can also run the script in batch mode by specifying -f option which accepts a list of servers in FQDN along with the port number. Using this feature of the script, you can easily run this script against all your vCenter Server(s) and ESX(i) hosts to ensure that their SSL certificates are still valid.

If you already have a list of hosts you want to check, then you can easily create a new file with the hostname and port. Though if you do not have one handy, I wrote a quick vSphere SDK for Perl script called generateESXiHostsList.pl that helps automate the creation of the output file containing all ESX(i) hosts when connecting to a vCenter Server. To use the script, you just need to have vCLI installed on a system or use vMA.

The script accepts one options which is --output which specifies the name of the output file to be created:

If we "cat" the file out, we can see it looks like the following:
vesxi50-1.primp-industries.com 443
vesxi50-2.primp-industries.com 443
vesxi50-3.primp-industries.com 443
vesxi50-4.primp-industries.com 443

Let's now run the ssl-cert-check against the list of ESX(i) hosts using the -f option and see if we have any hosts with expired certificates:

Uh oh, it looks like we have two hosts with some problems. We can see one host that already has an expired SSL certificate and another one that will be expiring in 10 days. We better take a look at these and get them replaced soon!

There are additional options in the ssl-cert-check script including the ability to email the results or run as a nagios check. You can easily schedule a cron job to automate this script to run every week and grepping for the keyword "Expiring" to alert you of any hosts that have expiring SSL certificates. As you can see, it is not only important to replace the default self-signed SSL certificates in your environment, but you need to validate on a routinely basis your your certificates are still valid.

VM Provisioning on Datastore Clusters in vSphere 5

Last year I wrote an article called Automating Storage DRS & Datastore Cluster Management in vSphere 5 and I provided a pretty comprehensive vSphere SDK for Perl script to help administrators automate Storage DRS configurations. These past few months I have noticed an increase in interest on the VMTN developer forums relating to Storage DRS. Majority of the questions has been related to which vSphere API methods to use and how to use these methods for cloning VMs to datastore clusters.

If you have cloned a VM before, the underlying vSphere API method being used is the CloneVM_Task(), but when cloning a VM to a datastore cluster, a different API must be used.

In vSphere 5, VMware introduced several new API methods in the StorageResourceManager and the two specific ones relating to provisioning VMs are RecommendDatastores() and ApplyStorageDrsRecommendation_Task(). The process to clone a VM to a datastore cluster is a two step process:

1. Call RecommendDatastores() which accepts very similar input as the CloneVM_Task(). In addition, you need to specify the datastore cluster also known as a Storage Pod in the vSphere API as well as the "type" (create, clone, relocate, reconfigure), in this example we will be performing a clone operation. This method will then generate a recommendation on where to place the VM which is based on the SDRS placement engine. No provisioning will occur at this point, just a placement recommendation.
2. To perform the actual provision of the VM, you will need to call ApplyStorageDrsRecommendation_Task() which accepts a recommendation ID that was generated from the first step. Once the recommendation is applied, the provisioning of the VM will start just like it did when you called the CloneVM_Task().

Note: The RecommendDatastores() will return multiple recommendations, the best one will be first entry in the array. This is the same algorithm used when performing this same operation in the vSphere Client, it also selects the first recommendation.

Now that we understand how the APIs work, let's take a look at how we can leverage this in a script for some automation! Here is a simple vSphere SDK for Perl script called datastoreClusterVMProvisioning.pl which allows you to clone an existing VM onto a datastore cluster. You will need a system that has vCLI 5.0 installed or you can use VMware vMA 5 to run the script. You will also need to connect to a vCenter Server 5 for all SDRS operations.

The script requires 4 input parameters:

• vmname - Name of the VM you wish to clone from
• clonename - Name of the cloned VM
• vmfolder - Name of a vCenter folder 
• datastorecluster - Name of a datastore cluster

Here is a screenshot of cloning an existing VM onto a datastore cluster:

The script is pretty straight forward and it can easily be adapted to include other configurations as required in your own environment.

Hopefully this gives you a better idea on how to leverage the new provisioning APIs for Storage DRS and start automating your VM deployments onto datastore clusters and get the benefits Storage DRS in your vSphere environment.

Automating Dead Space Reclamation in ESXi 5.0u1

VMware released vSphere 5.0 Update 1 last week, which mainly included bug fixes but it also brought back one very cool feature that was initially introduced with the release of vSphere 5.0 called Thin Provisioning UNMAP primitive for an ESXi host. You can read more about the details in this article by my colleague Cormac Hogan.

As you can see from From Cormac's article, the process of reclaiming of dead space on a thin provisioned LUN is currently a manual process, but does it have to be? The answer is, No of course, we can can definitely automate this!

Disclaimer: This script is not officially supported by VMware, please test this in a development environment before using on production system. This script is provided as an example on you can automate this manual process.

Before you proceed, please understand that the UNMAP operation can potentially take a few minutes up to a few hours depending on the size of your datastore and how your array handles this operation. You should consider performing this operation during a maintenance window or during off peak hours else you could impact VMs residing on the datastore. You should also ensure you have a VAAI-capable storage array before performing running this script.

I wrote a simple shell script called reclaimMyDeadsSace.sh which needs to be executed on the ESXi Shell via SSH. The script will also perform some validation such as ensuring you are running ESXi 5.0 Update 1 and that your host is in maintenance mode as a per-caution to ensure no running VMs are on the host during this process.

You will only need to run the script on one of the hosts connected to all the datastores you wish to reclaim dead space on. You may use scp or WinSCP to transfer the script to your ESXi host and ensure you set the execute permission on the script (chmod +x reclaimMyDeadSpace.sh)

The script can be executed in two ways:

1. Identify ALL VMFS3 and VMFS5 volumes and perform the reclaim based on the percentage entered by the user
2. Reclaim on specific datastores specified by the user as well as the percentage to be reclaimed (this is recommended, that way script does not choose all datastores including local ones)

Here is an example of selecting ALL VMFS3 and VMFS5 datastores to reclaim 60% of free space:

Here is an example of selecting just 4 datastores specified in a file and we will be reclaiming 60% of free space:

In this example, we created a file called "datastore_list.txt" (you may name the file anything you want) which contains the following:
iSCSI-1
iSCSI-2
iSCSI-3
iSCSI-4

So if you are using thin provisioned LUNs and would like to reclaim some of that dead space back and have a VAAI-capable storage array, be sure to check out the UNMAP functionality in ESXi 5.0u1.

How to Run WSX as a Standalone

This weekend I got chance to deploy the new Workstation Technology Preview 2012 in my lab and specifically play with the new WSX feature, which allows you to access your virtual machines from anywhere with just a browser. Currently WSX is only available for the Linux version of Workstation and is bundled together as part of the installer. I wanted to run WSX in one of my management VMs, and did not want the large disk footprint that came with Workstation. I did some digging and found it was quite easy to extract the WSX bits and run it on another Linux system, and in my case I tried it with vMA.

Disclaimer: This is mainly for educational and testing purposes as this is not officially supported by VMware.

The main prerequisite to install WSX is a Linux system that has Python 2.6 installed. You will still need to perform a full installation of Workstation to extract the WSX components, as recommended you can use the latest Ubuntu image.

Note: If you want to install Workstation Tech Preview in a VM, you may get an error for the version of VMware Tools not being up to date. You can by-pass that by running the following command: VMWARE_FORCE_INSTALL_IN_VM=yes ./VMware-Workstation-Full-e.x.p-646643.i386.bundle

Step 1 - You will need to create a few directories on the destination system in which you will be copying the WSX files to:

mkdir -p /etc/vmware/wsx
mkdir -p /usr/lib/vmware/{setup,scripts,lib,bin}
mkdir -p /var/lib/vmware/wsx/

Step 2 - You will now copy the following directory/files to destination system using scp:

scp /usr/lib/vmware/bin/vmware-wsx-server root@wsx.primp-industries.com:/usr/lib/vmware/bin
scp /etc/init.d/vmware-wsx-server root@wsx.primp-industries.com:/etc/init.d
scp /etc/vmware/bootstrap  root@wsx.primp-industries.com:/etc/vmware
scp -r /usr/lib/vmware/setup root@wsx.primp-industries.com:/usr/lib/vmware/
scp -r /usr/lib/vmware/scripts root@wsx.primp-industries.com:/usr/lib/vmware/
scp -r /usr/lib/vmware/lib/python2.6 root@wsx.primp-industries.com:/usr/lib/vmware/lib

Step 3 - Next you need to re-create the WSX config file which will be stored in /etc/vmware/wsx/config using the following command:

/usr/lib/vmware/lib/python2.6/site-packages/wsx/vmware-wsx-server --generate_config

If you wish to change the default port of 8888, you may edit the file before starting the WSX service.

Step 4 - Finally, you are now ready to start the WSX service by running the following command:

/etc/init.d/vmware-wsx-server start

Note: I ran into an odd issue with the initial login to WSX from the browser, in which I needed to create a secondary account other than the default "vi-admin". You need to login with "vi-admin" first, clear the cookie, so you can login with another user account before you add new servers. This was mainly looking at some of the errors from the logs and performing sqlite dump of WSX db.

Here are a few screenshot of accessing WSX from browser, iPad and iPhone:

The interface was pretty easy to use and it's pretty damn cool to be able to access your desktop from any platform that has a browser! Really looking forward to see where WSX is headed and hopefully it will be available in the future as a standalone installer and also with a logout button :)

Congratulations to Chris Greer for Winning Automating vSphere with vCenter Orchestrator

A week ago, I ran a simple contest to give a way a free copy of  Automating vSphere with vCenter Orchestrator Book. To enter the contest, you just had leave a comment with 5 things you wish or hope to automate using vCenter Orchestrator. I am happy to announce the winner of Cody Bunch's new book is Chris Greer! Congratulations Chris!

Chris's comment was the following:

1). I would like to put a web ui in front of the request process for a vm.
2). I want to call out to other services like request tracker via their rest interface
3). I want to automate vcloud director with automated task like license tracking
4). I want to automate network appliances like load balancers and firewalls when deploying specific templates (like web servers)
5). I want to be able to kick off vco workflows via a soap/rest call to extend current scripts
Bonus: I'd love to configure SRM but I don't think it's integrated yet.

Chris, hopefully the new Automating vSphere with vCenter Orchestrator book will help you accomplish these tasks!

Thank you to those who enter and if you did not win, you should still go and grab a copy of Cody's book (available in Kindle and paper back format) and learn how easy it is to leverage vCO in your vSphere environment.

How to Access USB Storage in ESXi Shell

While performing some experiments in my home lab, I needed to access a USB storage key directly on my ESXi host (not pass-through to VMs) and found it required a small trick after some tinkering. I thought I share the process in case this comes in handy for others.

Disclaimer: This is mainly for educational and testing purposes as this is not officially supported by VMware. Please use at your own risk.

Before I begin, you should know that only USB storage devices formatted with FAT16 can be accessed in the ESXi shell and is applicable to both ESXi 4.1 and 5.0.

Step 1 - Login to ESXi Shell via SSH and disable the USB Arbitrator service (this is automatically enabled by default to allow pass-through of USB devices to your VMs) using the following command: /etc/init.d/usbarbitrator stop

Step 2 - Plug-in your USB device to your ESXi host and you can verify by using the two ESXCLI commands: verifying the storage device using the command: esxcli storage core device list | grep -i usb or viewing the mounted filesystems using the command: esxcli storage filesystem list

Step 3 - Lastly, after you verify the USB device can be seen by the ESXi host, you can of course browse and access your USB device by looking under /vmfs/volumes/

Te re-enable pass-through of USB devices to your VMs, you just need to start the usbarbitrator service.

How to Deploy an OVF Located On ESXi Datastore Using ovftool

I have written several articles in the past about the awesome ovftool which is a versatile remote command-line utility for importing/exporting virtual machines in the OVF format across various VMware products. I mainly run ovftool in either the vMA or on my OSX desktop. When performing an import, the OVF files are local on the same system that has the ovftool installed.

I recently came across an interesting question about using the ovftool to deploy OVF files that are located on an ESXi datastore. My initial thought was that you would not be able to deploy the OVF files since the ovftool would not have access to the files locally. After finishing a recent article about ESXi datastore management using the vCLI's vifs utility, I then realized there might actually be a way to deploy OVF files that are stored on an ESXi datastore.

If you take a look on page 17 of the ovftool user guide, there is a table describing the various source locators that are supported. You can see that the source of an OVF file can be accessed by ovftool in 4 different methods including HTTP/HTTPs which is a key for this specific request.

Source Type	Default File	Extension Protocol
OVF	.ovf	File, HTTP, HTTPS, FTP
OVA	.ova	File, HTTP, HTTPS, FTP
VMX	.vmx	File
vvApprun	N/A	File
vCloud Director	N/A	HTTPS
vSphere	N/A	vSphere

Files and folders management for a datastore is exposed through the fileManager in the vSphere API and datastores are referenced as a URL or remote path.

A URL has the form

scheme://authority/folder/path?dcPath=dcPath&dsName=dsName

where

• scheme is http or https.
• authority specifies the hostname or IP address of the Center or ESX(i) server and optionally the port.
• dcPath is the inventory path to the Datacenter containing the Datastore.
• dsName is the name of the Datastore.
• path is a slash-delimited path from the root of the datastore.

Putting all this together, you can use the ovftool remotely to deploy an OVF file that is stored on an ESX(i) datastore. Below is an example walk through of this process.

Here is an OVF that is stored on a datastore located on an ESXi host:

To identify the URL path to your OVF, you can use a web browser to assist. Point your browser to the following address: https://[ESXI_HOST]/folder

When you first login, you will be brought to the root datacenter, in the case of directly connecting to an ESX(i) host, you will only see "ha-datacenter". Go ahead and select it and then you will be brought to a list of datastores the host has access to.

Select the datastore which contains the OVF file you wish to deploy from and then browse to the specific file.

Make a note of the URL path used to get to the OVF file and the OVF filename itself. Taking the example above, we end up with the following URL path:

https://vesxi50-4.primp-industries.com/folder/MyVM/MyVM.ovf?dcPath=ha-datacenter&dsName=iSCSI-1

To confirm the URL path, we can use ovftool to perform a simple "probe" on our OVF, this will provide you with a quick summary of the OVF.

Next we are ready to import the OVF file to our ESXi host. In this example, we will deploy the OVF to another datastore the ESXi host has access to and configure a specific portgroup to connect to the VM to after deployment. There are various options that can be passed to ovftool, please refer to the ovftool user guide for more details.

One the import has completed, you should now see the VM automatically registered in your ESXi host inventory.

You can see that this method allows you to import an OVF file stored on a datastore locally to the ESXi host as well as an OVF file stored on a remote datastore of another ESXi host. To help manage and deploy your OVF files, you should consider storing them on a centralized "media" datastore or even a WEB/FTP server that can be accessed by the ovftool.

Win a Free Automating vSphere with vCenter Orchestrator Book

Our good friend Cody Bunch has just recently released a new book called Automating vSphere with vCenter Orchestrator (available in Kindle and paper back format) that provides administrators with a complete walk through of installing, configuring and automating your vSphere infrastructure using vCenter Orchestrator. This is a must have book for all vSphere administrators looking to further automate and orchestrate your virtual infrastructure!

I still consider myself a beginner in vCO, but I have found that it has been very easy to create some really cool workflows such as this, this and this and you do not even need to be a developer to start using vCO! I am looking forward to reading Cody's new book.

Both Pearson and Cody Bunch was kind enough to provide me with an additional paper back copy of Automating vSphere with vCenter Orchestrator. Since sharing is caring, I will be giving away this copy to one lucky reader of virtuallyGhetto!

How do you win?

Just leave a comment below with the top 5 things you would like/hope to automate using vCO in your vSphere infrastructure. I will randomly select a winner a week from today. Even if you do not win, you should definitely still grab a copy of Cody's book and learn how easy it is to leverage vCO in your vSphere environment.

Datastore File Management using vCLI vifs

There are many useful scripts that are bundled with the VMware vCLI, one such script, that is not very well known is the vifs utility which provides datastore file management. When you right click on a datastore and browse using the vSphere Client, you can create a new folder, download/upload, delete and move files.

Using the vCLI's vifs utility, you can perform the same set of operations via the command-line and behind the scenes it uses the vSphere API fileManager to perform these operations. You can also browse datastore by just having access to a web browser, just point it to the following address: https://[ESXI_HOSTNAME]/folder and you can access the datastores by clicking through the links.

To browse the datastore using vifs, you will need vCLI installed on either a Windows/Linux system or you may use VMware vMA.

To browse a specific datastore for an ESXi host, you will need to first list the available datastores by using the following command: vifs --server [SERVER] --username [USERNAME] --listds

Once you have identified the datastore you are interested in, you will then use the --dir flag to list the contents of the directory and their sub-directories by using the following command: vifs --server [SERVER] --username [USERNAME] --dir '[DATASTORENAME]'

Note: The format of the datastore name must be in brackets '[datastorename]' which is how a datastore path is identified in the vSphere API. To list sub-directories, you will need a space between the datastore name and the directory name and do not forget to quote the parameter

Let's say you would like to download the .vmx configuration file for in the directory, you can use --get flag to by using the following command: vifs --server [SERVER] --username [USERNAME] --get '[DATASTORENAME] somedir/somefile.vmx' .

Note: In the example above, we are downloading the file in the current working directory denoted by the "." (period). If you wish to download it somewhere else or even renaming the file, you will need to specify the full path to the destination


If you wanted to automate the downloading of say all .vmx configuration files, it might be pretty tedious to run through the directory discovery, so here is a quick shell script called getVMVMX.sh that is more user friendly that allows you to easily download all .vmx configurations for a given datastore.

To use the script, you will need vCLI installed on either a Linux system or use VMware vMA and be sure to set the executable permission on the shell script. You will need to specify the credentials to the ESX(i) host and the specific datastore you wish to either "list" or "download" all .vmx configuration files.

Using the --listds flag, you will need to identify the datastore you wish to use. Next you will use the following command to "list" all .vmx configuration file: ./getVMVMX.sh [ESXI_SERVER] [USERNAME] "[PASSWORD]" [DATASTORE] list

To download all .vmx configuration file you will use the following command: ./getVMVMX.sh [ESXI_SERVER] [USERNAME] "[PASSWORD]" [DATASTORE] download [FOLDER] where FOLDER is a directory that will automatically be created for you to store all .vmx configuration files

Note: You can easily modify the script to add an additional "for loop" at the beginning to automatically download .vmx configurations for all datastores. I will leave that as an exercise for the reader.

So if you ever need to grab a vmware.log file for a specific VM or upload an ISO to datastore, you can do so from the command-line using the vifs utility that is bundled with the vCLI.