Automate Update Manager Operations using vSphere SDK for Perl + VIX + PowerCLI + PowerCLI VUM

I recently saw an interesting question on the VMTN developers forum asking about vSphere SDK for Perl and VUM API integrations. As it stands today, VMware has not publicly exposed or documented the VMware Update Manager APIs, though they have released a few PowerCLI VUM cmdlets for PowerCLI users. This is one feature, in my opinion, that PowerCLI has over the other vSphere SDKs. This is quite unfortunate, since automating VUM operations is something I have looked forward to and the process can be slow from clicking around in the GUI. I think VMware sometimes forget that the world does not run on Windows and that VMware != Windows.

I was hoping with the release of vSphere 4.1, the VUM APIs would finally be exposed as a standard web service like the vSphere API provided with proper SDKs for the various languages. My reply to the VMTN user was exactly this: this type of integration does not exist today. If you would like to automate VUM operations, you must use the PowerCLI VUM cmdlets. The follow-up comment was how one might integrate between vSphere SDK for Perl and PowerCLI VUM cmdlets. Initially, I did not recommend this but it could potentially work with some type of WMI call or hooks from some Perl modules. The solution might get a little messy and it would probably be faster to just set up the PowerCLI VUM cmdlets on a Windows host.

As I thought about the question later that evening, I realized that there is potentially another method if you wanted to use the vSphere SDK for Perl but utilize the PowerCLI VUM cmdlets. The following solution will demonstrate the use of vMA, vSphere SDK for Perl, VIX Perl API, PowerCLI and PowerCLI VUM cmdlets. VMware VIX is used for guest management for VMs running on a VMware hypervisor, whether it be Workstation, Fusion or ESX(i). There is a Perl SDK for the VIX API which I use to generate a PowerCLI script that is transferred directly into a Windows VM running both PowerCLI and PowerCLI VUM. This dynamic PowerCLI script will contain the host information and VUM baseline provided from the vSphere SDK for Perl script. The script will remediate the host and then it will be deleted by VIX. This configuration is not ideal, but if you wanted to automate some of the available VUM operations but continue to use your vSphere SDK for Perl scripts, then you can.

The dynamic PowerCLI script that is generated is the following:

Connect-VIServer -Server [vc-server] -Protocol https -User [vc-user] -Password [vc-password]
$vmhost = Get-VMHost [vi-host]
$baseline = Get-Baseline [host-baseline]
$baseline | Attach-Baseline -Entity $vmhost -Confirm:$false
$vmhost | Scan-Inventory
$baseline | Remediate-Inventory -Entity $vmhost -Confirm:$false

Note: This is just one example, you can easily customize the script to generate other VUM operations or more complex PowerCLI script. Once the operation is completed on the ESX(i) host, the script is deleted via VIX.

Before we get started, you will need to have a Windows VM that has both PowerCLI and PowerCLI VUM installed which resides in the same infrastructure as your VUM Server. You will also need to have VMware vMA available and installed with VMware VIX which will be documented down below.

1. Download VMware VIX 1.10

2. Transfer VMware-VIX-1.10.1-266898.x86_64.bundle installer to vMA

3. Install VIX:

VMware-VIX-1.10.1-266898.x86_64.bundle

================================================================

[vi-admin@kate ~]$ sudo sh VMware-VIX-1.10.1-266898.x86_64.bundle

Password:
Extracting VMware Installer...done.
You must accept the VMware VIX API End User License Agreement to
continue. Press Enter to proceed.
.....
The product is ready to be installed. Press Enter to begin
installation or Ctrl-C to cancel.

Installing VMware VIX API 1.10.1
Configuring...
[######################################################################] 100%
Installation was successful.

To complete the next section, you will need GCC to be installed which is not part of the default vMA installation. If you are using vMA 4.1, it is running CentOS and you can setup CentOS YUM repository. This will require that your vMA host can proxy out to internet or to specific site. I will document the changes that need to be made to talk to CentOS repository for package installs.

4. Create repository file:

[vi-admin@kate ~]$ sudo vi /etc/yum.repos.d/centos-base.repo

Add the following to the repo file:

5. Install GCC using yum:

[vi-admin@kate ~]$ sudo yum -y --nogpgcheck install gcc.x86_64

6. CD to vmware-vix directory and extract VIX Perl

[vi-admin@kate ~]$ cd /usr/lib/vmware-vix/

[vi-admin@kate vmware-vix]$ sudo tar -zxvf vix-perl.tar.gz
vix-perl/
vix-perl/vix.h
vix-perl/includeCheck.h
vix-perl/vm_basic_types.h
vix-perl/VixBinding.xs
vix-perl/VixBinding.pm
vix-perl/ModuleList
vix-perl/lib/
vix-perl/lib/API/
vix-perl/lib/API/API.pm
vix-perl/lib/API/Host.pm
vix-perl/lib/API/Job.pm
vix-perl/lib/API/VM.pm
vix-perl/lib/API/Snapshot.pm
vix-perl/lib/API/PropertyList.pm
vix-perl/lib/API/Constants.pm
vix-perl/lib/Simple.pm
vix-perl/Makefile.PL
vix-perl/typemap
vix-perl/ppport.h
vix-perl/README
vix-perl/samples/
vix-perl/samples/findhosttest.pl
vix-perl/samples/powertest.pl
vix-perl/samples/vmrun.pl
vix-perl/libvixAllProducts.so

7. Install VIX Perl

[vi-admin@kate vmware-vix]$ cd vix-perl

[vi-admin@kate vix-perl]$ sudo perl Makefile.PL
Writing Makefile for VMware::VixBinding

[vi-admin@kate vix-perl]$ sudo make

[vi-admin@kate vix-perl]$ sudo make install

If you did not want to go through all these manual steps after installing VIX, I actually wrote a quick shell script that will automate the configuration of the CentOS repository, GCC installation and extraction and installation of VIX Perl.

Download installVIXPerl.sh

Executing installVIXPerl.sh:

[vi-admin@kate ~]$ chmod +x installVIXPerl.sh

[vi-admin@kate ~]$ sudo ./installVIXPerl.sh

9. Download patch-host.pl script and transfer it to vMA

10. You will need to edit 4 variables defined within the script:

$psvm_username = Is the username to your Windows system running PowerCLI
$psvm_password = Is the password to your Windows system running PowerCLI
$powercli_bin = Is the full path to your powershell binary, if it is a default installation, you can leave this unchanged
$powercli_options = Is the path to vSphere PowerCLI psc file, you can also leave this as the default

Note: If you do decide to change either $powercli_bin or $powercli_options, make sure you add an extra "forward slash" which needs to be escaped in the Perl script.

The script requires you to connect to your vCenter server and provide the following parameters:

--vihost = Name of ESX(i) host to be patched

--baseline = Name of the VUM basline to attach

--psvm = Name of the Windows VM running both PowerCLI and PowerCLI VUM

Here is an example patching an ESXi host that is running on 4.0u2 and using the script to apply vSphere 4.1 baseline:

[vi-admin@kate]$ ./patch-host.pl --config --server reflex.primp-industries.com --username primp --vihost esxi4-3.primp-industries.com --psvm powerclivm --baseline vSphere4.1-Upgrade

Entering maintenance mode for esxi4-3.primp-industries.com
Successfully entered maintenance mode for esxi4-3.primp-industries.com
Guest login successfully!
Copy patch-host-esxi4-3.primp-industries.com-22174.ps1 to guest successfully!
PowerCLI/VUM script executed successfully!
Script removed successfully!
Successfully patched esxi4-3.primp-industries.com using VUM baseline: vSphere4.1-Upgrade

One thing I did notice, after the script is transferred to the Windows VM, it takes about 15-20 seconds before the script executes. I am not exactly sure why this occurs, as this does not happen when running the PowerCLI script logged into the VM. This may be due to the way VIX is executing the script within the guest.

The following resources were used in creating this script:

• Referenced growdisk.pl vSphere SDK for Perl/VIX script written by Matt Lamantia
• Referenced Alan's video on Getting Started with Update Managed cmdlets

Script: ghettoVCB Email Support (Experimental Support)

Check out the latest update to ghettoVCB which now support backup logs to be emailed upon competition. This feature is in experimental support and require the availability of nc (netcat) utility which is included in ESX(i) 4.0+. One can utilize netcat for variety of networking tasks, including communicating to an email server. The reason this feature is in experimental support is it may not be compatible with all email servers. The email functionality has been tested against a default installation of Postfix and is provided as-is.

For more details, please take a look at the ghettoVCB documentation.

Presenting at VMworld Developers Day 2010

If you are attending VMworld Developers Day 2010, come by and check out the presentation I will be giving alongside the security and virtualization expert, Edward Haletky.

Our presentation is entitled: vGhetto - Communities Building Great Solutions (PPC-07)

Level: Advanced

Description: The vSphere APIs offer a very rich interface allowing developers and administrators to create any number of scripts and tools to aid in the management of their VMware environment. We will present to you how the vGhetto Scripts and Client were created using vSphere API in a platform agnostic manner. The vGhetto Client is a multi platform user interface for running tasks that previously could only be run from a command line. We will also show you how to implement the vGhetto Scripts on various platforms that includes Windows, Linux, and Mac OSX including a few live demonstrations.  While we have chosen the vSphere SDK for Perl, all other SDKs (PowerCLI, VI Java, etc.) can also work in creation of an invaluable set of tools targeted at a developer or administrator.

Speaker: William Lam, Edward Haletky

If you are interested in automation and want to learn how the vGhetto Scripts can help you administer and manage your VMware environment. We will also be giving a cool demo of the vGhetto Client, you won't want to miss this session. Hope to you see you here!

For more details on this and other sessions at VMworld Developers Day 2010, check out this blog post here.

I would also like to mentioned that Wil Van Antwerpen, who is not listed in the abstract, is also a key member in the vGhetto Open Source Project. Unfortunately, Wil will not be joining us at VMworld this year.

vSphere 4.1 Is the Gift That Keeps On Giving

While doing some testing on ghettoVCB earlier this week, I noticed a new command line argument to vmkfstools utility called "--fix" in vSphere 4.1. From the man pages for vmkfstool, it states the following:


-x, --fix -[check|repair]
This option will check and/or repair the virtual disk in case of an unclean shutdown.

Here is an example of running the command against a VM's VMDK:

[root@esx4-1 ~]# vmkfstools --fix check /vmfs/volumes/esx4-1-local-storage-1/dummy/dummy.vmdk
Disk is error free

What surprised me next while looking up this new parameter in the man pages, I discovered another new argument called "--miscop":


-J, --miscop [setuuid | getuuid]
´setuuid´ option creates a unique identifier (UUID) for the
virtual disk and stores the UUID in the descriptor file of the
virtual disk. If the descriptor file already contains a UUID,
it will be overwritten with a new one. Please make sure that the
virtual disk does not have a UUID before using this option.
´getuuid´ option displays the UUID of the virtual disk.

The "--miscop" command is listed in the man pages but is not displayed when running "vmkfstools --help".

At this point, I thought there might be more hidden commands that VMware is holding out on us. I decided to use a well known UNIX/Linux utility called "strings" which looks for printable string in files and apply that to the vmkfstools binary. After sifting through the massive output, I found the following additional command line parameters that are not documented:

• dumpfs
• numfiles
• force
• recursivelock
• recover
• vmfsscan
• physicalmapping
• logicalmapping
• allocateblock
• clearlazyzero
• parseimage
• createarro
• createmirrordisks
• createmultiextent
• trackvdisk
• activehosts

Here are some of the command syntax which I have been able to verify:

dumpfs can be used by specifying either "-D | --dumpfs" and specifying a VMFS volume, file or folder.


[root@esx4-1 ~]# vmkfstools -D /vmfs/volumes/esx4-1-local-storage-1/

Lock [type 10c00001 offset 4292608 v 33, hb offset 3440640
gen 11, mode 0, owner 00000000-00000000-0000-000000000000 mtime 2509]
Addr <4, 0, 0>, gen 1, links 4, type dir, flags 0, uid 0, gid 0, mode 1755
len 1260, nb 1 tbz 0, cow 0, zla 1, bs 1048576

[root@esx4-1 ~]# vmkfstools --dumpfs /vmfs/volumes/esx4-1-local-storage-1/

Lock [type 10c00001 offset 4292608 v 33, hb offset 3440640
gen 11, mode 0, owner 00000000-00000000-0000-000000000000 mtime 2509]
Addr <4, 0, 0>, gen 1, links 4, type dir, flags 0, uid 0, gid 0, mode 1755
len 1260, nb 1 tbz 0, cow 0, zla 1, bs 1048576

activehosts can be used by specifying "--activehosts" and specifying a VMFS volume


[root@esx4-1 ~]# vmkfstools --activehosts /vmfs/volumes/esx4-1-local-storage-1/
Found 1 actively heartbeating hosts on volume '/vmfs/volumes/esx4-1-local-storage-1/'
(1): MAC address 00:50:56:92:3f:86


For the other parameters, I have not been able to figure out the additional arguments to make them work. If anyone or VMware has further insight into these other options, I would love to know what they are used for.

Magikmon MKBackup for ghettoVCB

I was contacted awhile back ago by Alain Spineux, about his MKBackup utility and he was in the very early stages of integrating the software with my ghettoVCB script. Per Alain's website:

MKBackup is a free front-end for common backup tools like MS Windows ntbackup, and it successor wbadmin, Un*x tools like tar, but also popular ghettoVCB to backup Virtual Machine on VMware ESX(i) host.

MKBackup is developed in Python and is available for Microsoft Windows, for Linux and other Un*x systems.

MKBackup is driven by a CLI and jobs are defined in an INI file. Its main feature is to send an email report including log files and all hints that could help the user to estimate the real status of the backup.

It looks like Alain has done some more work and the recently released version 0.9.3. You can get a better overview of the tool at his website and there is also a dedicated support forum.

I have personally not used MKBackup, but it could be very useful and compliment ghettoVCB. Not only can you schedule backups on a Windows or UNIX/Linux host using MKBackup, but it also generates backup reports based on the output from ghettoVCB logs. The reports can then be monitored using a soon to be released utility called MagiKmon which can provide email alerts on successful or failed backups.
 
Here are a few screenshots that Alain has kindly provided with the use of the latest version of ghettoVCB:

Script: Updated ghettoVCB and ghettoVCBg2 to Support vSphere 4.1

Check out the latest update to both ghettoVCB and ghettoVCBg2, which now supports vSphere 4.1! There are no new features introduced with these updates but ghettoVCB does include additional logging and debugging functionality to help identify issues that may arise. Unfortunately, with ghettoVCBg2 I had to perform some additional code changes due to undocumented changes in vMA's vi-fastpass Perl library. This initially caused the script to break and the reason was deprecated methods that were removed in the latest release of vMA 4.1. Luckily the fix was not too bad and I had to spend a few hours looking at the new methods that were implemented.

Hopefully everyone enjoys these two releases and if you run into any troubles, please post in respective script's forum.

vMA 4.1 - Authentication Policy (fpauth vs adauth)

I recently wrote an article about vMA 4.1 and Active Directory Integration and today I noticed there were some confusion on the expected behavior of the two types of authentication policy: vi-fastpass authentication versus Active Directory authentication. There are actually a few things to consider:

• What user context are you trying to execute a command against a target?
• What authentication policy was used to add the target to vMA?
• Is vMA host joined to an Active Directory Domain?

USER CONTEXT	FPAUTH or ADAUTH	vMA in AD DOMAIN
vi-admin	fpauth	no
DOMAIN\username	adauth	yes

I will try to explain the following two scenarios listed above.

In this example, vMA was not joined to an Active Directory Domain and we are adding a vCenter target to vMA using a local administrator account on vCenter server (by default, fpauth is assumed):

[vi-admin@tancredi ~]$ sudo vifp addserver manaslu.primp-industries.com
Enter username for manaslu.primp-industries.com: administrator
administrator@manaslu.primp-industries.com's password:
This will store username and password in credential store which is a security risk. Do you want to continue?(yes/no): yes


We can verify the target was added using fpauth by running the following command:

[vi-admin@tancredi ~]$ vifp listservers -l

esx4-1.primp-industries.com ESX fpauth
esxi4-3.primp-industries.com ESXi fpauth
manaslu.primp-industries.com vCenter fpauth

Next, we will set the fastpass target to the newly added vCenter server:

[vi-admin@tancredi ~]$ vifptarget -s manaslu.primp-industries.com

[vi-admin@tancredi ~][manaslu.primp-industries.com]$

If we run "esxcfg-nics -l" against an ESX(i) host that is being managed by this vCenter, we would do the following (note: user context is vi-admin):

[vi-admin@tancredi ~][manaslu.primp-industries.com]$ esxcfg-nics -l --vihost esxi4-3.primp-industries.com

Name PCI Driver Link Speed Duplex MAC Address MTU Description
vmnic0 02:00.0 e1000 Up 1000Mbps Full 00:50:56:ac:69:95 1500 Intel Corporation PRO/1000 MT Single Port Adapter

In this first example, we are relying solely on vi-fastpass authentication, where a vi-adminXX account is created on the target. The credentials to this account is generated by vMA and stored in the local credential store.

In this example, vMA has been joined to an Active Directory Domain and we are adding a vCenter target using Active Directory credentials:

[vi-admin@tancredi ~]$ sudo vifp addserver reflex.primp-industries.com --authpolicy adauth
Enter username for reflex.primp-industries.com: PRIMP-IND\primp

Note: As of writing this, there is a typo in vMA 4.1 documentation on the syntax to use when specifying the username when prompted. You will need to use DOMAIN\username, if you decide to use the --username, then you need to add a second "slash" to escape the first (e.g. DOMAIN\\username)

We can verify the target was added using adauth by running the following command:

[vi-admin@tancredi ~]$ vifp listservers -l

esx4-1.primp-industries.com ESX fpauth
esxi4-3.primp-industries.com ESXi fpauth
manaslu.primp-industries.com vCenter fpauth
reflex.primp-industries.com vCenter adauth

Next, we will set the fastpass target to the newly added vCenter server but before we do so, we need to login to vMA using a valid Active Directory account.

[primp@tancredi ~]$ vifptarget -s reflex.primp-industries.com

[primp@tancredi ~][reflex.primp-industries.com]$

Now if we run "esxcfg-nics -l" against an ESX(i) host that is being managed by this vCenter, we would do the following (note: user context is DOMAIN account):

[primp@tancredi ~][reflex.primp-industries.com]$ esxcfg-nics -l --vihost himalaya.primp-industries.com

Name PCI Driver Link Speed Duplex MAC Address MTU Description
vmnic0 06:00.0 e1000e Up 1000Mbps Full 00:30:48:d9:58:6a 1500 Intel Corporation 82574L Gigabit Network Connection
vmnic1 07:00.0 e1000e Down 0Mbps Half 00:30:48:d9:58:6b 1500 Intel Corporation 82574L Gigabit Network Connection

In this second example, we are relying solely on Active Directory authentication, where credentials of the user that is logged into vMA are being used. Unlike in the first example, if you were in the vi-admin context and tried to execute the same command, you will notice you are prompted for credentials. This is the intended and expected behavior of the two scenarios.

However, if you do not want to join vMA to an Active Directory Domain but would still like to perform an unattended authentication from vi-admin context, then you need to setup a Kerberos ticket for the target. The details on configuring this is outlined in vMA 4.1 user guide, please refer to the document for more details.

One thing to note which I actually ran into, is that when you join your vMA host to Active Directory Domain, you must reboot vMA after joining to the domain. If you do not, you will run into issues when trying to add a target using adauth authentication policy.

resxtop & vi-fastpass Downgraded Feature In vMA 4.1

For those that use VMware's vMA vi-fastpass authentication (vifpinit which is now vifptarget) which allows a user to perform an operation on an ESX(i) or vCenter host without having to provide credentials each time, may notice a change with the resxtop command. In vMA 4.0, if you initialized a fastpass target you could run resxtop against the host without having to provide additional credentials to the host.

In vMA 4.1, this functionality has actually been downgraded, that is right, a feature de-enhancement.

As you can see, even if you have initialized a fastpass target, you will need to specify both the username and password each time you call resxtop. resxtop only supports manual credentials and does not support other types of authentication mechanisms such as a configuration file, session file or passthrough auth. This can definitely slow things down if you are trying to troubleshoot multiple hosts and want to switch between them utilizing the fastpass authentication.

What is actually more interesting is this seems to be a feature that has been in the works over the 3 major releases of vMA, which was formally known as VIMA. Let's take a trip down memory lane regarding this feature in the vMA/VIMA release notes:

VIMA 1.0 - Oct 27, 2008 - Feature not supported and marked as a known issue.

vMA 4.0 - May 21, 2009 - Feature supported but may run into an issue with non-fastpass targets which has been resolved in this release.

vMA 4.1 - July 13, 2010 - Feature has been removed, requires both --username and --password each time.

 
Even though I consider this a bug, VMware did document this change as a "known issue" in the release notes. I am not sure if this feature will be resolved in a future release, but it just seems odd that we are taking one step backwards in terms of functionality.

vMA 4.1 - Active Directory IntegrationTip

The latest release of vMA 4.1 now supports Active Directory integration which can be used to centralize all authentication within a Windows environment. To join a vMA host to your Active Directory domain, you just need to use one simple command called domainjoin-cli which is part of Likewise's "Open" product.

Here is an example of vMA host joining an AD domain:

By default, Likewise "Open" is configured to not assume the current Active Directory Domain as the default. This means if you are authenticating against vMA via SSH connection, you will need to specify both the username and the full domain. (e.g. ssh username@domain.com@vMA-host)

Here is an example of logging into vMA using AD credentials:

This can be pretty tedious to type out everytime, especially if you have a very long domain name. However, this can be easily modified to assume the default domain.

You will need to edit /etc/likewise/lsassd.conf and uncomment "assume-default-domain = yes" and then save your changes:

sudo vi /etc/likewise/lsassd.conf

You will need to reload the configurations for the changes to take effect by running the following utility:

sudo /opt/likewise/bin/lw-refresh-configuration

Now, you can login by just specifying the username without having to provide the full AD domain name.

I actually wrote an article about a month ago on configuring Likewise "Open" AD intergration on vMA before the release of vSphere 4.1. The article goes through the process of setting up "Open" on vMA 4.0 and also documents the change of the default domain. For more Likewise commands and details, check out the article above.

Update1:
If you would like to add an AD group to sudoers file, you need to edit /etc/sudoers file. You need to make sure you escape the initial forward slash and any white spaces that maybe in the group name. In this example, we have a group called "VI Admins" that you would like all users to be able to login to vMA using their AD credentials and perform operations using sudo.

1. Edit /etc/sudoers using vi-admin account, make sure you use 'sudo':

[vi-admin@kate ~]$ sudo vi /etc/sudoers

2. Add the following towards the bottom of the file:

%PRIMP-IND\\VI\ Admins ALL=(ALL) ALL

Note: We're escaping both the initial forward slash and the space

3. Verify user can now sudo by querying sudo operatoins the user is allowed to execute:

[primp@kate ~]$ id
uid=1058014289(primp) gid=1058013696(domain^admins) groups=1058013696(domain^admins),1058014440(vi^admins)

[primp@kate ~]$ sudo -l
Password:
User primp may run the following commands on this host:
(ALL) ALL

ESXi 4.1 - Major Security Issue

On Thursday July 15th, a user raised a question on the VMTN forums regarding an ESXi 4.1 password issue. The problem was described as the following:

Hi all
It seems that authentication only requires the first 8 characters to be correct. My root password is 11 characters long, but so long as the first 8 characters are correct, I can put whatever I like after that and it still authenticates me. Tested this on three ESXi boxes, all running 260247 (release)

I performed a few quick tests to validate the user's claim and in fact this was the case with a new installation of ESXi 4.1. Though in my lab, I had upgraded from ESXi 4.0 Update 2 to ESXi 4.1 and I was not able re-produce the issue. I then tried to reset my password to the same initial password and to my surprise, I hit the exact problem as described by the user. I notified VMware of the issue and was told that they were looking into the problem. As of today, there is still not official word from VMware on whether this is a bug or an expected behavior.

VMware does have a KB article (KB1012033) documenting both ESX and ESXi password requirements, by default a minimum of 8 characters is required but can be changed by modifying the PAM module pam_passwdqc.so which checks for the quality of a password.

Further looking into the issue over the weekend, I have found the following:

• pre ESXi 4.1 (e.g. ESXi 3.5, 4.0, 4.0u1, 4.0u2) does not have this password issue
• password issue affects ALL methods of authentication to an ESXi host: vSphere Client, local and remote Tech Support Mode, /bin/login, SSH localhost and vSphere MOB. (did not get a chance to validate AD logins)
• password issue will NOT affect hosts that were upgraded to ESXi 4.1, unless you change or update your password afterwards, in which case only the first 8 characters will be checked

During my testing, I provisioned a newly built ESXi 4.0 Update 2 (called esxi4-4) and ESXi 4.1 (called esxi4-3) and used a modified SSH login expect script to run my tests. You can download my modified version of the script here. Both hosts had their password created using the DCUI and was set to "VMware123"

Here is an example of SSHing to both host and running "vmware -l" command to show that hosts accepts the password that was created:

As you can see, I am able to login to both hosts using the defined password of "VMware123" and displaying the output of "vmware -l" command.

In the next example, I will purposely provide the wrong password of "VMware12" which should fail:

This should have failed for both hosts, but did not in the case of ESXi 4.1 and still allowed me to login. If you tried to login to the ESXi 4.1 host using "VMware1", access is denied. Basically, if you have a password that is greater than 8 characters, only the first 8 characters are validated and any combination there after is accepted! This in my mind is a big security hole, especially if this is an unexpected behavior that users are not made aware of.

I also performed one additional test by modifying the default password requirement from 8 characters to 9 by editing /etc/pam.d/system-auth:

I then used the passwd utility to create a new root password, I attempted to use the password "VMware12" which should fail as it does not meet the minimum password requirement, which it does:

Once I used password that was 9 characters or greater, the utility allowed me to set the new password:

From what I can understand, the password in fact is being set properly and the rules that govern the length are being regulated. The problem seems to be with the authentication module where it is validating the password and only checking for the first 8 characters. To me this sounds like a authentication bug and one that can severely impact any security policies put in-place for an organization.

I hope that VMware provides either a clarification on the issue and potentially a fix or patch for this problem. In the meantime, if you have security policies in your environment pertaining to password requirements and plan on upgrading to ESXi 4.1, ensure you do not change the password. If you do, change it before the upgrade.

UPDATE1:

It looks Didier Pironet found the solution and documented the steps in his follow-up blog post. I am still hoping VMware will address this security issue as soon as possible and provide a patch. The above work around does resolve the problem but the change is not persisted through a reboot without manual hacks to force a backup of PAM configuration file.

UPDATE2:

VMware just released KB article regarding the issue: KB1024500

UPDATE3:

VMware has just updated the KB article KB1024500 with slightly more details pertaining to the solution of the problem. They still have not confirmed the bug that is identified in the PAM module where DES encryption is used over MD5, but can be assumed via the solution provided. The problem actually exists in both vSphere ESX and ESXi 4.1 and the KB includes a fix to both including a way to persist the changes on ESXi. The security issue apparently is not important enough to get an immediate patch but may eventually get a permanent patch sometime in the future.

UPDATE4: VMware just first the first patch set (p01) for both ESX and ESXi 4.1 that resolves this security issue along with other bugs that have been identified since the release of vSphere 4.1. Here is the specific KB article regarding the fix - http://kb.vmware.com/kb/1027024 and here is the link to the details p01 patch set - http://kb.vmware.com/kb/1027027 As with anything, please test and validate in your test environment prior to rolling out to production.

How to get vSphere 4.1 to work with CapacityIQ 1.0.3 using unsupported method

Part of my process before upgrading to new software in a production environment, whether it's a major or minor release, is to let it bake in a development lab. While finishing up the upgrade of my vSphere 4.1 lab, I had one additional component to install, CapacityIQ. To my surprise (but not really), it is incompatible with vSphere 4.1.

CapacityIQ is distributed as a VMware virtual appliance that runs on CentOS 5.2. As part of the installation you are asked to configure both a password for both the root and ciqadmin account. Since this is just a linux host, I decided to SSH in and poke around to see where this check might be taking place. It only took a few minutes using the linux command find to locate some Perl modules that seem to be in the very early stages of a client side stub to a CapacityIQ API.

There is a Perl module called CIQExtension.pm which actually does the validation of your vCenter Server build. Within the script, there are two hardcoded values 2.5.0 and 4.0.0 which are the version that are supported by CapIQ.

What I did was add an extra "OR" statement to allow for vCenter Server 4.1.0:

Once I saved the changes, I went back to the CapIQ admin page to try to re-register my vCenter 4.1 and voila, it fully registered and started CapIQ!

I also verified by checking the status tab:

At this point, I was optimistic that the change has worked and decided to login to vCenter server to verify by loading the CapIQ application:

There you have it! CapacityIQ 1.0.3 running on vSphere 4.1

Even though there are new performance statistics with the release of vSphere 4.1, these stats will most likely not be used or seen by CapIQ, which leads me to believe that this should work. However, I would not recommend doing this in a live production environment without consulting with VMware first. If you do work for VMware or part of the CapacityIQ team, I would love to know if this hack works or do we just need to wait for another release of CapIQ?
 
I was aware of the VMware View Issue but I was not able to find any documentation or KB article documenting this compatibility issue with the latest version of CapacityIQ 1.0.3. This tends to happen more often where there is a disconnect between the core VMware products (ESX, ESXi, vCenter) and the rest of the auxiliary products that VMware provides. Sometimes, I wonder if the various product teams actually talk to each other with regards to integration and compatibility.

UPDATE: 

While browsing around, I also found an undocumented Perl script called ciq-admin.pl which can also be used to administrator and configure your CapacityIQ. Using the utility, you can register your vCenter with CapIQ and there is a hidden flag which allows you to bypass the version check. This also allows you to get vSphere 4.1 working with CapIQ without having to manually edit any files.

To use this utility, you must login as the ciqadmin user. If you are logged in as root, you can just switch user.

The syntax will be the following:

/usr/lib/ciq/ciq-admin.pl register --vc-server [server] --force --user [username] --password [password] --noversioncheck


Here is an example of registering vSphere 4.1 vCenter using the CLI:

You can verify on the command line that operation is successful by running the status command, which is the same status tab in the CapIQ admin page :

New way of enabling and disabling services using vSphere 4.1

While checking out the PlanetV12n feed, I noticed a new video from David Davis about the new vSphere 4.1 Tech Support Mode. In the short video, David goes over the new method of enabling "hidden" unsupported Busybox Console, also known as Tech Support Mode. In the past, you had to be on the console of your ESXi host, type ALT+F1, and then "unsupported" to gain access. Once in, to enable remote SSH access or Remote Tech Support Mode, you had to edit /etc/inetd.conf and restart inetd service. This was pretty tedious if you needed access for a short period of time. In the video, David goes over the new method showing how it can be done using the DCUI and the old method is no longer required.

What surprised me after watching the video was that he did not mention the other method of enabling and disabling Tech Support Mode both local and remote. One issue I had with past releases of ESXi is that you could restart some services such as ntp or vmware-vpxa via the vSphere API, but others were just not available. In vSphere 4.1, VMware introduces a few new services around their Likewise Active Directory integration but also includes controlling both local and remote Tech Support Mode as well as DCUI itself.

These services can be enabled and disabled using the vSphere Client, here is a screenshot:

To enable or disable TSM, just click on the service and then click on options:

You will then have the option to configure the startup policy including enabling or disabling the service:

If you needed to perform this operation against one or two host, it is not that big of a deal. Though if you needed to enable remote Tech Support Mode (SSH access) across few dozen hosts, then can still be tedious. Luckily I wrote a script (hostServiceMangement.pl) last year that allowed you to enable and disable supported services using the vSphere API. Without any modifications, it supports vSphere 4.1 and can take advantage of the new services that are available for control.

Here is an example of listing the services on an ESXi 4.1 host:

Here is an example enabling remote Tech Support Mode:

Here is an example of disabling remote TSM:

The script can be executed on a host that has vCLI 4.1 installed or on vMA 4.1 and can bulk update a list of ESX/ESXi host or individual host. For more details, please check out the documentation for hostServiceManagement.pl.

vSphere 4.1 was not always 4.1?

During the BETA of vSphere 4.1, there were some inconsistencies in some of the build numbers which was eventually resolved. It looks like something was still missed in the final RTM release of vSphere 4.1 ....

Here is a screenshot of the python libraries on classic ESX 4.1 used for esxupdate. At some point, it looks like this release was going to be named vSphere 4.5 as most people had anticipated, but then was revised to 4.1.

I'm sure there will be few more easter eggs, happy hunting :)

Script - New vSphere Health Check Script 4.1.0

Check out the latest update to the popular vSphere Health Check Script, now at v4.1.0, which supports the new release of vSphere 4.1 and is compatible with previous releases of vSphere.

Here is what's new in v4.1.0:

 For more details, please visit the documentation for vSphere Health Check Script.

New vSphere 4.1 CLI Utilities Marketing Did Not Tell You About Part 3

Following part1 and part2 of new vSphere 4.1 CLI Utilities, here are some new vimsh commands:

For ESX use /usr/bin/vmware-vim-cmd
For ESXi use /bin/vim-cmd

1. vmsvc/power.suspendResume is used for vMotion and sVMotion tasks before switching over to the new VM.

[root@esx4-1 ~]# vmware-vim-cmd vmsvc/power.suspendResume
Insufficient arguments.
Usage: power.suspendResume vmid

Suspend & resume the specified virtual machine.

2. vmsvc/queryftcompat allows you to query a given Virtual Machine to check for Fault Tolerance compatibility.

[root@esx4-1 ~]# vmware-vim-cmd vmsvc/queryftcompat
Insufficient arguments.
Usage: queryftcompat vmid

Query FT compatibility for a VM.

3. vimsvc/auth/lockdown_is_enabled allows you to query if the host has the lockdown feature enabled, this is only applicable to ESXi.

~ # vim-cmd vimsvc/auth/lockdown_is_enabled
false

4. vimsvc/auth/lockdown_is_possible allows you to check if you can put the host into lockdown mode.

~ # vim-cmd vimsvc/auth/lockdown_is_possible
true

5. vimsvc/auth/lockdown_mode_enter allows you to enter lockdown mode on an ESXi host.

~ # vim-cmd vimsvc/auth/lockdown_mode_enter

6. vimsvc/auth/lockdown_mode_exit allows you to exit lockdown mode on an ESXi host.

~ # vim-cmd vimsvc/auth/lockdown_mode_exit

The next two commands refer to the following services which can also be configured using the vSphere Client:

7. hostsvc/start_local_tsm allows you to enable Local Tech Support Mode on an ESXi host.

~ # vim-cmd hostsvc/start_local_tsm

8. hostsvc/start_remote_tsm  allows you to enable Remote Tech Support Mode (SSH access) on an ESXi host.

~ # vim-cmd hostsvc/start_remote_tsm

9. hostsvc/stop_local_tsm allows you to disable Local Tech Support Mode on an ESXi host.

~ # vim-cmd hostsvc/stop_local_tsm

10. hostsvc/stop_remote_tsm allows you to disable Remote Tech Support Mode (SSH access) on an ESXi host.

~ # vim-cmd hostsvc/stop_remote_tsm

New vSphere 4.1 CLI Utilities Marketing Did Not Tell You About Part 2

Continuing from Part1 of new vSphere 4.1 CLI Utilities, here are few more:

1. vmkload_app64 is a 64bit version of the VMkernel application loader.

[root@esx4-1 bin]# /usr/lib/vmware/bin/vmkload_app64 -h

Usage: vmkload_app [options]
--nocore|-n Disable core dump for this world
--enableDebug|-d Enable userworld debug wait after core
--break|-b [wid] Break a userworld into the debugger
(only specify world id when not creating
or connecting to a userworld)
--kill|-k Send signal 'signum' to a running userworld
(only works with vmx worlds)
--ouputCartelID|-o Output cartel ID and other info to
--setsid|-S Run vmkload_app in a new session
--workingdir|-w Use as userworld working directory (defaults to current)
--env|-v Add environment variable to
userWorldApp's environment
--ipSocketType|-i Set the type for IPv4 sockets, where
is one of 'proxied', 'vmktcp', or 'costcp'
--sched.group= Set scheduler group
--sched.cpu.units= Set units (mhz or pct) for cpu rates
--sched.cpu.min= Set minimum cpu rate
--sched.cpu.max= Set maximum cpu rate
--sched.cpu.minLimit= Set upper bound for minimum cpu rate
--sched.cpu.shares= Set cpu share allocation
--sched.cpu.affinity= Set affinity towards pcpus
--sched.mem.min= Set minimum memory allocation (MB)
--sched.mem.max= Set maximum memory allocation (MB)
--sched.mem.minLimit= Set upper bound for minimum memory growth (MB)
--sched.mem.shares= Set memory share allocation
--sched.swap.scope= Set swap scope (none, private, system)
--sched.swap.dir=[dir] Set directory for swap file
--sched.swap.file= Set swap file name
--sched.memSizeLimit= Set upper bound on address-space size (MB)
--help|-h This message

2. vmware-usbarbitrator is a utility for allowing VMs to connect to the host's USB devices.
[root@esx4-1 bin]# /usr/lib/vmware/bin/vmware-usbarbitrator

3. vprobed looks to be a utility for running the vProbe daemon.

[root@esx4-1 bin]# /usr/lib/vmware/bin/vprobed -h

Error: vprobed takes no arguments.
4. vmkiscsiadm was a pretty well known tool used to configure and troubleshoot iSCSI on ESX. However, it has been removed with the release of vSphere 4.1.

5. vmkiscsi-tool has been updated with some new options:


[root@esx4-1 ~]# /usr/sbin/vmkiscsi-tool -h
vmkiscsi-tool -h --help
-A --Authentication
-C --connection
-D --discovery
-E --session
-I --iSCSIname
-L --Lun
-M --MTU
-N --Network: network properties
-O --Associate: Associate/Disassociate route.
-P --Phba
-R --discoveryStatus : Print discovery status.
-S --static: Static Discovery Targets
-T --Target
-U --Route: Route table.
-V --Nic
-W --Parameter
-X --Reset
-Y --SetNicList
-c --ipconfig: enable/disable DHCP, ARP redirect
-d --dnsserver
-e --ethernet: Link Status
-g --gateway
-i --ipAddress
-k --Alias
-n --iSNS
-p --Pnp: Physical Network Portal properties (pnic)
-q --Lnp: Logical Network Portal properties (vmknic)
-s --subnetmask
-v --version
Subcommands
-l --list
-r --remove
-a --add
-m --authMethod : specify method for add/remove
-b --mutual CHAP
-j --persist changes
-y --per dynamic discovery
-x --per static discovery
-z --per network portal (binded vmknic)
-t --per target
-u --per isid(session)
-f --flag: set a discovery or authentication flag
adapterName
Combine -l with an option to display the current information.

6. vmkiscsi-test is a new iSCSI utility to test various components and provides a summary at the end with the checks that ran, failed and passed:

[root@esx4-1 ~]# /usr/sbin/vmkiscsi-test


CUnit - A Unit testing framework for C - Version 1.1-1
http://cunit.sourceforge.net/


Suite: general - General Tests
Test: IMA_NullParameters ... passed
Test: IMA_VMW_NullParameters ... passed
Test: IMA_BadOids ... passed
Test: IMA_VMW_BadOids ... passed
Suite: Info - Library and Plugin Tests
Test: IMA_GetLibraryProperties ... passed
Test: IMA_GetPluginOidList ... passed
Test: IMA_GetPluginProperties ... passed
Suite: Adapter - Adapter Tests
Test: IMA_GetPhbaOidList ... passed
Test: IMA_GetPhbaProperties ... passed
Test: IMA_VMW_GetPhbaProperties ... passed
Test: IMA_GetLhbaOidList ... passed
Test: IMA_VMW_GetLhbaOidList ... passed
Test: IMA_GetLhbaProperties ... passed
Test: IMA_VMW_GetLhbaProperties ... passed
Suite: Portals - Portals Tests
Test: IMA_GetNetworkPortalOidList ... passed
Test: IMA_VMW_GetNetworkPortalOidList ... passed
Test: IMA_GetPnpOidList ... passed
Suite: Discovery - Discovery Tests
Test: IMA_GetDiscoveryProperties ... passed
Test: IMA_SetIsnsDiscovery ... passed
Test: IMA_SetSlpDiscovery ... passed
Test: IMA_SetStaticDiscovery ... passed
Test: IMA_SetSendTargetsDiscovery ... passed
Test: Static Discovery ... passed
Test: Dynamic Discovery ... passed
Suite: Properties - Standard Properties Tests
Test: IMA_GetDataPduInOrderProperties ... passed
Test: IMA_GetDataSequenceInOrderProperties ... passed
Test: IMA_GetDefaultTime2RetainProperties ... passed
Test: IMA_GetDefaultTime2WaitProperties ... passed
Test: IMA_GetErrorRecoveryLevelProperties ... passed
Test: IMA_GetFirstBurstLengthProperties ... passed
Test: IMA_GetIFMarkerProperties ... passed
Test: IMA_GetImmediateDataProperties ... passed
Test: IMA_GetInitialR2TProperties ... passed
Test: IMA_GetMaxBurstLengthProperties ... passed
Test: IMA_GetMaxConnectionsProperties ... passed
Test: IMA_GetMaxOutstandingR2TProperties ... passed
Test: IMA_GetMaxRecvDataSegmentLengthProperties ... passed
Test: IMA_GetOFMarkerProperties ... passed
Suite: VMW Properties - VMware Properties Tests
Test: IMA_VMW_ArpRedirectProperties ... passed
Test: IMA_VMW_MtuProperties ... passed
Suite: Auth - Auth Tests
Test: IMA_GetInUseInitiatorAuthMethods ... passed
Test: IMA_VMW_GetInUseInitiatorLocalAuthMethods ... passed
Test: IMA_SetInitiatorAuthMethods ... passed
Test: IMA_VMW_SetInitiatorLocalAuthMethods ... passed
Test: IMA_GetInitiatorAuthParms ... passed
Test: IMA_VMW_GetInitiatorAuthParms ... passed
Test: IMA_SetInitiatorAuthParms ... passed
Test: IMA_VMW_SetInitiatorAuthParms ... passed
Test: IMA_VMW_GetMutualAuthParms ... passed
Test: IMA_VMW_SetMutualAuthParms ... passed
Test: IMA_GetLhbaMutualAuthParmsList ... passed
Test: IMA_GetMutualLocalAuthParms ... passed
Test: IMA_GetMutualLocalAuth ... passed
Test: IMA_RemoveLhbaMutualAuthParms ... passed
Test: IMA_AddLhbaMutualAuthParms ... passed

--Run Summary: Type Total Ran Passed Failed
suites 8 8 n/a 0
tests 55 55 55 0
asserts 495 495 495 0
Asserts Skipped: 0
Tests Skipped: 0
Ignore Flags:

7. vmfs-support is a new shell script that uses vmkfstools -D and recursively dumps information about all files given a VMFS folder or file (This utility is only found on ESXi 4.1).

~ # ash /sbin/vmfs-support
Usage: vmfs-support /

~ # ash /sbin/vmfs-support /vmfs/volumes/iSCSI-1/

vmkfstools -D /vmfs/volumes/iSCSI-1/.fbb.sf
Lock [type 10c00001 offset 4294656 v 3, hb offset 3866624
gen 73, mode 0, owner 00000000-00000000-0000-000000000000 mtime 2656]
Addr <4, 0, 1>, gen 1, links 1, type sys, flags 0, uid 0, gid 0, mode 400
len 98304, nb 1 tbz 0, cow 0, zla 1, bs 1048576
vmkfstools -D /vmfs/volumes/iSCSI-1/.fdc.sf
Lock [type 10c00001 offset 4296704 v 1, hb offset 0
gen 0, mode 0, owner 00000000-00000000-0000-000000000000 mtime 16020]
Addr <4, 0, 2>, gen 1, links 1, type sys, flags 0, uid 0, gid 0, mode 400
len 31129600, nb 30 tbz 0, cow 0, zla 1, bs 1048576
vmkfstools -D /vmfs/volumes/iSCSI-1/.pbc.sf
.....
.....
.....

8.  The well known nc utility has been added to unsupported Busybox console of ESXi 4.1.

~ # /bin/nc
usage: nc [-46DdhklnrStUuvzC] [-i interval] [-p source_port]
[-s source_ip_address] [-T ToS] [-w timeout] [-X proxy_version]
[-x proxy_address[:port]] [hostname] [port[s]]

9. vdu is a new utility that provides disk utilization information, similar to that of UNIX/Linux du (This utility is only found on ESXi 4.1).

~ # /bin/vdu
For './bin':
tardisk SYS1: 28694376 ( 47 inodes)
tardisk SYS2: 135168 ( 18 inodes)
heap : 208 ( 16 inodes)
For './etc':
tardisk SYS1: 4630016 ( 220 inodes)
heap : 65184 ( 35 inodes)
tardisk SYS2: 10790 ( 22 inodes)
tardisk vpxa.vgz: 6144 ( 5 inodes)
tardisk aam.vgz: 4096 ( 2 inodes)
ramdisk MAINSYS: 65536 ( 4 inodes)
For './lib':
tardisk SYS1: 41208050 ( 146 inodes)
ramdisk MAINSYS: 16384 ( 1 inodes)
tardisk SYS2: 29168640 ( 95 inodes)
For './lib64':
tardisk SYS1: 6955539 ( 23 inodes)
For './opt':
tardisk SYS1: 0 ( 1 inodes)
tardisk vpxa.vgz: 44070912 ( 24 inodes)
tardisk aam.vgz: 12457984 ( 188 inodes)
For './productLocker':
tardisk SYS1: 24
For './sbin':
tardisk SYS1: 87665431 ( 145 inodes)
heap : 8192 ( 4 inodes)
ramdisk MAINSYS: 106496 ( 10 inodes)
tardisk SYS2: 151552 ( 3 inodes)
For './tmp':
heap : 34816 ( 33 inodes)
ramdisk updatestg: 8192 ( 1 inodes)
For './usr':
tardisk SYS1: 36028127 ( 381 inodes)
tardisk SYS2: 252928 ( 39 inodes)
ramdisk MAINSYS: 53248 ( 6 inodes)
heap : 23 ( 1 inodes)
For './var':
tardisk SYS1: 2137 ( 29 inodes)
tardisk SYS2: 28048896 ( 51 inodes)
ramdisk MAINSYS: 3850240 ( 56 inodes)
heap : 63758 ( 58 inodes)
ramdisk hostdstats: 2023424 ( 2 inodes)
tardisk vpxa.vgz: 0 ( 2 inodes)
For './vmfs':
tardisk SYS1: 6 ( 2 inodes)
For './vmimages':
tardisk SYS1: 47 ( 3 inodes)
For './vmupgrade':
tardisk SYS1: 19
For './.ssh':
heap : 2048 ( 2 inodes)
ramdisk MAINSYS: 8192 ( 1 inodes)
For './.rnd':
heap : 2048
For './bootbank':
heap : 50
For './altbootbank':
heap : 50
For './store':
heap : 50
For './scratch':
heap : 50
For './locker':
heap : 7
For '.':
heap : 176484 ( 156 inodes)
tardisk SYS1: 205183772 ( 999 inodes)
tardisk SYS2: 57767974 ( 228 inodes)
tardisk vpxa.vgz: 44077056 ( 31 inodes)
tardisk aam.vgz: 12462080 ( 190 inodes)
ramdisk MAINSYS: 4100096 ( 78 inodes)
ramdisk updatestg: 8192 ( 1 inodes)
ramdisk hostdstats: 2023424 ( 2 inodes)

New vSphere 4.1 CLI Utilities Marketing Did Not Tell You About Part 1

With the new release of vSphere 4.1, there are new additions to the CLI utilities that administrators can leverage for configurations and troubleshooting. Although, not all were treated equally from an announcement and documentation standpoint. 

Here are some of the new and/or subtle changes with the CLI utilities:

1. vmkfstools undocumented "-D" option now outputs to console along with an entry in /var/log/vmkernel, whereas in the past, to locate the entry within the vmkernel logs for identifying locked files by a particular host as documented in this article. 

Here is an example of the old version of vmkfstools and -D option:

Here is an example of the new version of vmkfstools and -D option:

 2. storageRM is a debugging utility for Storage I/O Control at the host level.

[root@esx4-1 ~]# /usr/lib/vmware/bin/storageRM -h

Default Values:
SleepTime: 4000
Threshold: 35
Gamma: 0.25
Beta-per-host: 4.00
LowerBound: 4
UpperBound: 64
Storage I/O Control-- This tool does flow control at the host
in order to maintain disk I/O latency close to a
threshold and queue sizes converge at values
proportional to the beta parameter.

The following histogram related options are available:
-a, --print the list of all luns, their latency threshold,
queue depths and if Storage I/O Controlis set/unset
-b, --Beta per host value
-d - put debugging info in the given file
-f, --force the run without checking version or checksum
-g, --defGamma value for use in control equation
-h, --help will print the usage
-l, --lower bound on the length of lun queue, (default 4)
-n, --no anomaly detection is done
-r, --reset issue queue for all luns to default
-s, --sleep time in ms for periodic execution
-t, --threshold on the latency (in ms), for rate control
-u, --upper bound on the length of lun queue (default 64)
-v, --debug log level value
storageRM Usage:
storageRM [options]

3. net-lbt is a debugging utility for the new Load-Based Teaming feature.

[root@esx4-1 ~]# /usr/lib/vmware/bin/net-lbt -h

Usage: [-d] [-t time] [-v] [-s threshold]
-d run in daemon mode
-t daemon sleep period in seconds, minimum 10 seconds
-v run with verbose logging
-s saturate threshold [10, 100], i.e. 60 for 60% of line rate

4. net-dvs is a debugging utility for Distributed vSwitch. 

[root@esx4-1 ~]# /usr/lib/vmware/bin/net-dvs -h

Warning: This is an unsupported command. Use at your own risk.
net-dvs -a [ -P maxPorts] switch_name
net-dvs -d switch_name
net-dvs [ -A | -D ] -p port switch_name
net-dvs [ -s name=value | -u name ] -p port switch_name
net-dvs -l [ switch_name ]
net-dvs -i (init database)
net-dvs [-S | -R | -G ]
net-dvs -T
net-dvs -v "vlanID[;t|p[0-7][;min-max,min-max...]]
net-dvs -V "primaryVID,secondaryVID,i|c|p;primaryVID,secondaryVID,i|c|p..."
net-dvs -m "sid;dname;snaplen;

[oiveld];encapvlan;wildcardsIn,wildcardsOut;dstPort1,dstPort2,...;srcInPort1,srcInport2,...;srcOutPort1,srcOutPort2,...;:sid2;dname2..."
net-dvs dvswitch -k "respool1_id;respool2_id;..."
net-dvs dvswitch -p dvport -K "respool1_id:shares:limit;respool2_id:shares:limit;..."
net-dvs dvswitch -p dvport -z "respool_id"
net-dvs dvswitch -j [activate|deactivate]
net-dvs -L uplink_name1[,uplink_name2,...] -t team_policy_type -p port switch_name
net-dvs dvswitch -H "red|yellow|green:some message" switch_name
net-dvs -o "depth,param|classname;depth,param|classname;... -p port|globalPropList switch_name
net-dvs --mtu mtu_value [-p dvport] switch_name
net-dvs --x 0|1 -p dvport switch_name
net-dvs --vlan vlanID -p dvport switch_name
net-dvs --reset -p dvport switch_name
net-dvs --cap cap_value -p dvport switch_name
net-dvs --states -p dvport switch_name

5. remoteDeviceConnect is a new utility that allows you to mount various remote devices including floppy and USB. 

/usr/lib/vmware/bin/remoteDeviceConnect: option requires an argument -- h

VMware remote Device Connect
Usage: /usr/lib/vmware/bin/remoteDeviceConnect OPTIONS filename
Options:
-h : hostname (localhost)
-p : Port to connect to (63079, 902 for authd)
-t : (Req.) cd-raw, cd-iso, cd-normal, cd-raw-ex, floppy
-d : (Req.) Device node to connect to (floppy0, ideX:Y)
-n : the VM's floppy drive number
-f : fileType
-A : use Authd to connect
-U : username for authd (your username)
-V : the VM to use. (NULL)
-P : password.
Examples:
remoteDeviceConnect -t floppy -d floppy0 -f device /dev/fd0
remoteDeviceConnect -t floppy -d floppy0 -f file image.flp
remoteDeviceConnect -t usb "path:2/1 vid:0x0547 pid:0x2131"

6. sensorD looks to be a debugging utility that can connect to an ipmi device.

[root@esx4-1 bin]# /usr/lib/vmware/bin/sensord

sensord: failed to open ipmi device: No such file or dir
sensord: unsupported hardware

7. statedumper looks to be a debugging utility for output information about the system and its states.

[root@esx4-1 bin]# /usr/lib/vmware/bin/statedumper -h

statedumper [-f filename] [-s off] [-e off] [-b] [-o] [-r] [-x]

The following options are supported:
-e - end dump at offset
-f - use filename rather than the default state.log
-o - output entry offsets
-r - output all registers, output 64 bits with -r64
-s - start dumping at offset
-b - filter on branch count, use -s and -e for start/end
-x - dump extra debug data

8. vmkeventd looks to be a utility for capturing VMkernel events

[root@esx4-1 bin]# /usr/lib/vmware/bin/vmkeventd -h

/usr/lib/vmware/bin/vmkeventd: invalid option -- h
Usage: /usr/lib/vmware/bin/vmkeventd [-d]

9. analyze-esx-init-boot.py looks like a debugging utility to analyze the COS boot up logs

[root@esx4-1 ~]# /usr/sbin/analyze-esx-init-boot.py -s -S /vmfs/volumes/datastore1/esxconsole-4c27dd75-a38d-5044-5670-005056927558/logs/sysboot.log -V /var/log/messages
ERROR: Could not find 'cpu 0: early measured tsc speed' in the log
This is one of the very first log messages after boot
POST boot times will not be relevant
Unable to find vsish. Please ensure that you have debugging tools installed and your PATH is correctly setup.
VMKernel: 0.0000 secs
POST Tests: 0.0000 secs
Init Scripts: 0.0000 secs

ESX Boot Time: 0.0000 secs
Hardware/BIOS: 0.0000 secs

Total Boot Time: 0.0000 secs

Serial port output was on

Script - Automate VAAI Configurations in vSphere 4.1 (vaaiHWAccelerationMgmt.pl)

With the release of vSphere 4.1, we finally get to see the first revision of the vStorage API for Array Integration (VAAI) features. This initial release provides the following SCSI driver primitives with VAAI supported storage arrays:

• Write Same/Zero - Eliminating redundant and repetitive write commands, tells array to repeat via SCSI commands.
• Full Fast/Copy - Leverage array to mass copy, snapshot and move blocks via SCSI commands.
• Atomic Set and Test - Stop locking LUNs and start locking blocks.
• Thin Provisioning Stun - Reporting array TP state to ESX.

The above definitions were taken off of an EMC presentation found here. VMware has also published a new VMware KB article regarding VAAI FAQ, take a look here at KB1021976.

Though these features are not specific to any one storage array vendor, the vendors themselves are required to implement these primitives within their array OS software for them to be available. If all prerequisites are met, and you have an ESX or ESXi host running on vSphere 4.1 and VAAI supported storage array, these new storage operations will now offload to the array versus running within the VMkernel.

However, if you do not have an array that supports VAAI, the new version of ESX and ESXi will still try to use these features. As I understand from an earlier discussion of VAAI, there is one additional operation that is performed and it's impact is supposed to be insignificant (please correct me if I'm wrong). Though if you would like to disable these VAAI features or would like to see the difference between a non-VAAI and VAAI operation, it is controlled with the following three advanced host configurations.

VMFS3.HardwareAcceleratedLocking - Atomic Test and Set
DataMover.HardwareAcceleratedMove - Full/Fast Copy
DataMover.HardwareAcceleratedInit - Write Same

By default, all three of these configurations are enabled with a default installation of vSphere 4.1. The following vSphere SDK for Perl script allows a user to enable or disable VAAI configuration on a set of hosts defined in an input file. The script allows you to connect to vCenter if your hosts are being managed by vCenter or directly to a specific ESX or ESXi host and provide the following parameters:

--hostlist = Lists of ESX(i) hosts to perform operation _IF_ they're being managed by vCenter (default is ALL hosts in vCenter)

--operation = Operation to perform (query|enable|disable)

Download: vaaiHWAccelerationMgmt.pl

Here is an example of the host input file:

[vi-admin@makalu scripts]$ cat hosts
esxi4-2.primp-industries.com
esxi4-3.primp-industries.com
Here is an example of querying for VAAI configurations:

Here is an example of disabling VAAI configurations:

Here is an example of disabling VAAI configurations:

For more information about vStorage APIs, take a look here.

Script - Automate Storage I/O Control in vSphere 4.1 (siocManagement.pl)

Storage I/O Control is a new feature of vSphere 4.1 which allows a user to define the QoS prioritization for the I/O activity on a single host or a cluster of hosts. SIOC supports only VMFS volumes and the latency threshold is configured on a per VMFS datastore.

Currently, the only method of configuring SIOC is using the vSphere Client while connected to your vCenter Server, which can be tedious if you manage more than 1 VMFS datastore:

 The following vSphere SDK for Perl script allows a user to bulk update SIOC across your vSphere infrastructure based on an input file that consists of the name of the VMFS datastores and the latency thresholds to be configured. The script requires that you connect to your vCenter server and provide the following input parameters:

--datastore_inputfile = Is the name of the datastore input file which contains the name of your VMFS datastore and the latency value

--operation = There are four supported operations (query|enable|disable|update)

--vihost = Name of the ESX(i) host to perform the operation on (you only need to perform the operation on 1 host within a cluster and the changes are taken place across all hosts)

Download: siocManagement.pl

Here is an example of the datastore input file:

[vi-admin@makalu scripts]$ cat datastorelist
# datastorename;latency_value
# e.g.
# mydatastore1; 35

iSCSI-1;20
iSCSI-2;15
iSCSI-3;35
esxi4-3-local-storage-1;40


Note: The latency threshold must be between 10-100 ms. The default value when you enable SIOC is 30 ms,

Example of query operation:

Example of enable operation:

Example of disable operation:

Example of update operation:

Before attempting to make any changes, make sure you consult with your storage vendor.