woohoo virtuallyGhetto is #25 on Top 25 VMware Bloggers!

I am very honored to be part of the Top 25 VMware Bloggers, a contest that was held by Eric Siebert who runs the well known vSphere-land website. This took me by surprised as I just recently started blogging and virtuallyGhetto is only 4months old. This year there was a total of 115 blogs that were on the ballot and a total of 860 votes. You can check out the full list above or watch the top 25 countdown in a very special episode of vChat Episode 8 with Eric Siebert, David Davis, Simon Seagrave and special guest John Troyer. 


I wanted to thank Eric for all his hard work for setting up the contest and tallying up all the votes! I also would like to thank all my readers and followers who voted and supported me! I hope to continue to share great scripts and content with the community.

Congratulations to all the top 25 bloggers and to other bloggers who made the top 100 list. Keep up the great work and content!

If you are interested in following the Top 25 VMware Bloggers on twitter, Raymon Epping has created twitter list that you can follow here: http://twitter.com/repping/vmware-top25-2010

Automating vCloud Director 1.5 & 5.1 and Oracle DB Installation

While playing around with the new VMware vCloud Director, I found myself rebuilding the vCD system twice. This meant I needed to automate the deployment so I did not have to do it for the third time if needed. The following script is used to deploy VMware vCloud Director and Oracle Express database on a single CentOS 5.5 64bit system. This setup is very similar to the one documented by Duncan Epping - Creating a vCD Lab on your Mac/Laptop, though our setup is running in our ghettoDatacenter.

This script will assume that you have installed CentOS 5.5 64bit OS on a system and it has direct or proxy access to the internet. You will also need to make sure you have provisioned two virtual NICs for the system, which the second one will automatically be configured for you.

UPDATE: The scripts below have been updated to support the latest release of vCloud Director 5.1.

For our setup, I used CentOS-5.5-x86_64-netinstall.iso to do a network installation using ftp://ftp.cs.ucsb.edu/mirrors/centos/5.5/os/x86_64/images and using all the defaults for "Server" type deployment. Since I can not re-distribute the installers for VMware vCloudr Director and Oracle Express, you will need to download the installers and transfer it to your CentOS system. The last step before you being is to transfer the vcd_setup.sh script and vcd.rsp response file to your CentOS system.

Download: vcd_setup.sh
Download: vcd.rsp

You will need make a few tiny edits the vcd.resp file to fit your environment which will be used to deploy vCD and Oracle Express, including the configurations.

Here is the sample vcd.resp that is included:


If you have read Duncan's vCD article or the vCD installation guide, you will be familiar with all these configurations. The only required changes you will need to make will be the secondary IP Address and the passwords for Oracle Database. Once you have made all the necessary edits, we can start the installation.

You should have the following 4 files located in the same directory on your vCD system:

• vcd_setup.sh
• vcd.resp
• oracle-xe-10.2.0.1-1.0.i386.rpm
• vmware-cloud-director-1.0.0-285979.bin

Note: Make sure you set the executable permission on the vcd_setup.sh, else you will get a permission denied error.

1. Start the installation and type y to begin:

The script will exit if it can not locate both vCD and Oracle Express installers, make sure those are available in the same working directory.

2. The YUM repository will be configured from a generic mirrorlist from CentOS and package dependency for vCD will begin:

3. The next section is disabling the firewall and selinux which is not a best practices, but for a lab, this is fine. NTP is also configured and enabled upon bootup:

4. Oracle Express will now be installed and configured based on the initial configuration file passed to the script (vcd.rsp). An oracle response file is created in /tmp which allows for a silent and unattended installation:

5. Per VMware's best practices, two separate tables spaces (CLOUD_DATA and CLOUD_INDX) are created along with a separate user account with the required privileges per the documentation using sqlplus and configuration file generated on the fly:

6. We now configure the secondary IP Address and generate the keystore for both HTTP and CONSOLE PROXY and stores it in /opt/keystore/certificates.ks:

Note: Make sure that both IP Addresses used for HTTP and CONSOLE PROXY have both their forward and reverse DNS properly configured. The "host" utility will be used to query their hostnames which is used in creating the keystore certificates.

7. vCloud Director will not be installed which actually extracts an embedded RPM for the actual installation:

8. We now configure vCloud Director from our response file and a configuration file is generated again for a silent and unattended installation:

Note: The script does not cover syslog server and syslog port configuration. If you need this, you can easily edit the script to support this.

9. Once the vCloud Director installation and schema creation is completed, vCloud Director is finished installing:

10. Now we can start vCloud Director and wait for the initialization to be completed which the script watches /opt/vmware/cloud-director/logs/vcloud-container-info.log for completion:

11. You can now open up your browser point it to your vCD system and if everything went accordingly, you should see vCloud Director landing page:

I hope this script will be helpful for those that want to kick the tires on vCloud Director but prefer not to go through all the manual steps of configuring an Oracle database and vCD. In total, the script took ~8minutes to complete from start to finish.

Automating ESXi 4.1 Kickstart Tips & Tricks

While testing the new kickstart functionality in ESXi 4.1, I ran into a few issues trying to convert a classic ESX 4.x deployment to ESXi 4.1. I thought I share some of the tips and tricks I have learned, so others will not encounter the same issues.

Before diving in and creating an ESXi 4.1 kickstart configuration, make sure you spend some time going over the documentation provided by VMware, specifically the ESXi Installable and vCenter Server Setup Guide. 

UPDATE: For ESXi 5, Check out ESXi5 Kickstart Tips & Tricks

Tip #1

If you are going to specify a ks.cfg (kickstart configuration file) in your pxelinux file, make sure that the kickstart entry is appended after the *vmkboot.gz* but before *vmkernel.gz* entry as highlighted in green in the screenshot. If you place it anywhere else in the boot line option, you will receive an error that is not easy to diagnose. Also you want to make sure you add triple dashes (---) after the kickstart line following the required syntax for the boot options as highlighted in orange in the screenshot.

Tip #2

While developing and testing your ks.cfg, you may want to use the new dryrun parameter which parses your kickstart configuration file looking for syntax and formatting errors. In dryrun mode, no installation will be performed but you will be provided with a log of whether your ks.cfg had any errors, warnings or was successful in being validated.

The following screenshot shows a warning where I purposely left out --hostname entry which is generally recommended within the "network" portion of the ks.cfg:

If there are other errors or warnings, they will be displayed within this screen and you can login to the host to view the log for more details (esxi_install.log):

To login to the host, you will press "enter" and you will be prompted for login (press Alt+F1 to go to login screen). The username by default will be "root" and the password is blank, just press enter for the password:

Once logged in, you will want to take a look at the esxi_install.log for more details on how your ks.cfg is being processed and if there are any errors or warnings discovered by the parser:

Tip #3

If you want to enable both local and remote (SSH access) Tech Support Mode on your ESXi host, you now have the ability to do this via host services. You can use the vim-cmd (vimsh) utility to enable these services and both local and remote TSM is disabled by default. 

Note: If you want to enable either local and/or remote TSM, you need to make sure you enable and start the service for you to actually be able to SSH into your ESXi host.

Tip #4

With classic ESX, if you needed to transfer additional packages or files to your host, you could easily mount an NFS volume, with ESXi, an NFS client is not available. If need to transfer files for configuration purposes, you can utilize the wget utility. 

The syntax for wget is the following:

wget http://webserver/file -O /tmp/file

Tip #5

I have been told by support that you could not configure syslog for your ESXi host without relying on external tools such as vCLI, PowerCLI or vSphere Client. I have found that you actually can configure syslog configurations, though you have to dig a little bit into vim-cmd (vimsh) as it is not available using any of the local esxcfg-* commands. There only three syslog options as provided via vSphere Client in the Advanced Host Configurations: Syslog.Remote.Hostname, Syslog.Remote.Port and Syslog.Local.DatastorePath

Here is the syntax for the syslog options:


vim-cmd hostsvc/advopt/update Syslog.Remote.Hostname string syslog.primp-industries.com

vim-cmd hostsvc/advopt/update Syslog.Remote.Port int 514

vim-cmd hostsvc/advopt/update Syslog.Local.DatastorePath string "[datastoreName] /logfiles/hostName.log"

Note: Currently you can only configure one syslog server for your ESXi host to forward logs to.

Tip #6

Another new new kickstart parameter introduced with ESXi 4.1 is --level that is used in conjunction with %firstboot stanza. This parameter specifies the specific order in which the kickstart firstboot configurations should run with respect to the other startup scripts when your ESXi host first boots. By default, if you leave this out, VMware will automatically create a script called firstboot_001 and number it 999 which will be the very last script to execute. It is a good idea to move any post configurations to the very end, since most of post configuration may rely on specific VMware CLIs and services which must be started up before executing. You of course can change level, but be careful about moving it too early in the boot process. 

Here is an example of changing the level to 998:

Once the host has booted up, you can login to see the script that was created from your %firstboot stanza under /etc/vmware/init/init.d

Note: As you can see, the firstboot script has now changed to 998. You will also notice two other scripts set at level 999 that handles updating the password if you decide to set a root password from the default blank, which you should. These custom scripts are generated after the initial build and upon the next reboot, these will be automatically removed. 

Tip #7

You may have noticed in Tip #6, we changed the --level to 998, by default all three of these init scripts are set to boot order 999. This was actually done on purpose, the reason being as described earlier, the root password is blank by default. One issue that I found while testing is the inability to enable "Management Traffic" for a VMkernel interface. You can easily enable vMotion and FT Traffic for a VMkernel interface using vim-cmd (vimsh), but you can not for Management Traffic. One way I solved this problem is creating a python script which connects to the local ESXi MOB and enables Management Traffic on a particular VMkernel interface. I have shared this specific script on the on the VMTN communities which can be found here. The script is actually based on modified version that was initially created by Justin Guidroz who blogged about it here.

Here is the snippet that would be included in the %firstboot in which does not require you to expose the root password as it is empty by default:

ESXi 4.1
ESXi 4.1 Update 1 ( Requires CSRF code update)
As you can see, we first create the python script and then we execute it. This allows us to call other utilities within the Busybox console without having to specify the interpreter to be python, we can just use busybox as the interpreter.

Tip #7a Here is an alternative solution to enable management traffic type on ESXi - Another way to enable management traffic on ESXi

Tip #8

If you tried to configure NTP by echoing your NTP servers into /etc/ntpd.conf and restarting ntpd, you will notice that the changes do not take effect. The only way I have been able to get this to work is by issuing another reboot which is specified at the very end of the %firstboot which will then be picked up upon boot up by the host.  

Tip #9

If you would like customize the DCUI Welcome Screen, take a look at my blog post How to add a splash of color to ESXi DCUI Welcome Screen.  

Tip #10

If you want to update the default datastore name from "datastore1" to something more useful such as [hostname]-local-storage-1, you can use vim-cmd (vimsh) to do so. Here is the syntax for the command if you want to use the short hostname and append "-local-storage-1" (this should be done in the %firstboot section of your ks.cfg): vim-cmd hostsvc/datastore/rename datastore1 "$(hostname -s)-local-storage-1"

Tip #11

SNMP is another one of those configurations that can not be configured and started up via normal services as you would have done in classic ESX. You can make the appropriate edits to the configuration file and you will need to reboot the host for the changes to take affect just like NTP configurations. You will need to edit /etc/vmware/snmpd.xml and add that to your firstboot section. Here is an example of snmpd.xml file:

<config>
  <snmpsettings>
    <communities>public1;private1</communities>
    <enable>true</enable>
    <port>163</port>
    <targets>192.168.1.5 public1;192.168.1.6@163 private1</targets>
  </snmpsettings>
</config>

Tip #12

A VMTN user recently posted an issue when applying patches during firstboot, that the init scripts were not being removed and the scripts continue to execute upon every reboot. The problem was that the boot.cfg were not being properly updated under /vmfs/volumes/Hypervisor{1,2}. I did some testing and found that if you created a second customization script and perform the patching as the very last step and issued a reboot, that you would not run into this problem.  Here is a small snippet of what your ks.cfg would like look with 2 custom init scripts, one configured at 998 and the other configured at 9999:

Tip #13

Here is a post on Automating Active Directory Domain Join in ESXi Kickstart

Tip #14

Here is a post on Automating Active Directory User Management in ESXi Kickstart As you can see, there are quite a few hacks I had to go through to get an equivalent kickstart build for ESXi 4.1 compared to classic ESX. I am sure there are other gotchas, but these are the ones I had encountered. Overall, ESXi 4.1 has greatly improved in terms of automating an unattended installation and configuration from ESXi 4.0, but there is definitely more work that needs to be done by VMware before users can easily transition from classic ESX to ESXi.    

Tip #15
Here is a post on How to automatically add ESX(i) host to vCenter in Kickstart

In additional to VMware's documentation which is still limiting, here are additional tools and links that will be useful when creating your ks.cfg for ESXi 4.1:

• http://communities.vmware.com/blogs/vmwareinsmb/2010/07/13/esxi-41-scripted-installation-via-pxe-and-kickstart
• http://www.kendrickcoleman.com/index.php?/Tech-Blog/esxi-41-kickstart-install-wip.html
• http://labs.vmware.com/flings/vmware-auto-deploy

How to add a splash of color to ESXi DCUI Welcome Screen

Earlier this year I created a simple vSphere SDK for Perl script that allows you to update ESXi's DCUI (Direct Console User Interface) banner with multiple lines of text. I originally thought you could not customize the text color or the background color, though recently I found out that was not the case. While doing some testing on ESXi 4.1, I noticed two files (support,welcome) located under /etc/vmware and looking at the contents of support, it made realize we might be able to change the colors.

Here is the contents of /etc/vmware/support, notice the special formatting of the variables including color tags:

I decided to use one of my favorite UNIX utility, "strings" to take at the dcui binary that is located under /sbin in the Busybox Console (Tech Support Mode) and discovered you can control both the font color and background color. There are also special variables that can be used to display information about the ESXi host such as the product version or IP Address.

Here are the supported colors:
white
black
dark-grey
light-grey
yellow

Here are the special variables:
assettag
BIOSversion
BMCversion
CIM_Chassis
CPLDversion
esxproduct
esxversion
hostname
ip
license
memory
OMC_MCFirmwareIdentity
OMC_SMASHFirmwareIdentity
OtherIdentifyingInfo
PLSAversion
serial-number
servicetag
ssl-thumbprint
supportperiod
supportstart
VersionString
VMware_HHRCSoftwareIdentity

There are two ways of updating the DCUI welcome banner: using local or remote esxcfg-advcfg or manually editing /etc/vmware/welcome file.

Here is an example of using vCLI's esxcfg-advcfg:

Here is what that looks like on the DCUI:

As you can see, this is not easy if you want to update multiple lines. You would need to add a lot more spaces to force newlines, but this becomes tedious and pretty much unreadable. The second method is edit the welcome file that is located in the Busybox Console, which requires you to enable ESXi's Tech Support Mode. I wrote a quick Perl script called generateDCUIScreen.pl which accepts an input file and allows a user to customize the output and the script generates the "welcome" file which is uploaded to your ESXi host.

Here is an example of the input file:


The script will parse the input file which will contain definitions for:

• bgcolor and color as described above
• special variables as described above (must use braces for variables to be translated)
• custom text
• "=space=" string which is unique to my script which generates the newlines

The script requires that you have Perl, but you do not need to have vSphere SDK for Perl. For ease of use, I executed the script using vMA.

Here is an example execution using the input file from above:

You will need to scp the new"welcome" file to your ESXi host under /etc/vmware which is empty by default. For the changes to take effect, you will need to run the following command at the console:

kill $(ps | grep dcui | awk '{print $1}')

This will kill dcui utility and watchdog process will spawn a new instance causing the change to take effect Note: A reboot will also do the job, but be sure to run /sbin/auto-backup.sh before doing so, that way the change will be backed up.

Here is what DCUI screen looks like:

As you can see, you can control variety of pre-defined variables including hostname and IP Address and custom text for your organization. This is useful for those that do not want to expose all the information that available on the default DCUI screen, which may be a security concern for some organizations. A few things to note, I was not able to fill the entire screen like the default DCUI banner and the "welcome" file is character sensitive and you need to use tabs or white spaces to force the background to get filled. There is also a limit in the number of characters per row before it wraps to the next line.

I am sure there is someone out there that will create some interesting ASCII art, but here is my 5min free hand attempt at it ;)

This can easily be integrated into a scripted install using the new ESXi 4.1 kickstart feature.