How to inject custom drivers into an ESXi 4.1 image using vibddi?

Over the holiday break, I spent some time cleaning up some of the development virtual machines in our ghettoDatacenter. I came across the VMware Auto Deploy appliance that I deployed awhile back ago. I did not think I had a use for it since we already have an automated deployment system using PXE and kickstart. Auto Deploy was launched relatively recently from the VMware Flings lab. It was originally slated for release as part of vSphere 4.1 but during the transition from the BETA to RC, it was dropped and never made it into the GA release of vSphere 4.1

I decided to give the documentation one last read before deleting and to my surprise, I stumbled across an interesting gem, vibddi. vibddi (pronounced vib d-d-i) stands for VIB (vSphere Instalaltion Bundle) Disk Dump Image, which is actually a Perl utility that was created to help customize ESXi images more easily.

If you ever had a need to customize an ESXi image and inject custom drivers or configurations, you know it can be long and complex process. There are many tutorials on the internet including a recent post by Eric Sloof on injecting drivers into an ESXi installer. vibddi is meant to expedite the process and make it much simpler to inject custom drivers into an ESXi image.

****Disclaimer Since this tool is not very well documented and it is most likely not officially supported by VMware, please use test and validate the images generated prior to using in an production environment Disclaimer****

To run vibddi, you need to use sudo. Here are the available options:

[vi-admin@autodeploy ~]$ sudo vibddi -h
Password:

vibddi: Query and update vibs on a VMvisor dd image or device

Usage:
vibddi -h --- Print this

vibddi -i <VMvisor-dd> -q --- Query vibs installed on the image

vibddi -i <VMvisor-dd> -c --- Check bootbank filesystems on the image

vibddi -i <VMvisor-dd> -v <vib-path> [ -g ] [ -n ] --- Update the image with a single vib

vibddi -i <VMvisor-dd> -m <metadata-URL> -b <bulletin-ID> [ -p <proxy> ] [ -g ] [ -n ] --- Update the image with an online bulletin

vibddi -i <VMvisor-dd> -o <bundle-path> [ -g ] [ -n ] --- Update the image with an offline bundle

vibddi -i <VMvisor-dd> -e <export-path> [ -a <alternate-conf> ] --- Export boot modules from the image

vibddi -i <VMvisor-dd> -t <kernel-opt> --- Add/Remove a VMkernel option

vibddi -i <VMvisor-dd> -x --- Transform image to ThinESX format

vibddi -i <VMvisor-dd> -l <license-path> --- Install a license file (vmware.lic) on the image

vibddi -d <VMvisor-dev> -q --- Query vibs installed on the device

vibddi -d <VMvisor-dev> -c --- Check bootbank filesystems on the device

vibddi -d <VMvisor-dev> -v <vib-path> [ -n ] --- Update the device with a single vib

vibddi -d <VMvisor-dev> -m <metadata-URL> -b <bulletin-ID> [ -p <proxy> ] [ -n ] --- Update the device with an online bulletin

vibddi -d <VMvisor-dev> -o <bundle-path> [ -n ] --- Update the device with an offline bundle

vibddi -d <VMvisor-dev> -e <export-path> [ -a <alternate-conf> ] --- Export boot modules from the device

vibddi -d <VMvisor-dev> -t <kernel-opt> --- Add/Remove a VMkernel option

vibddi -d <VMvisor-dev> -x --- Transform image to ThinESX format

vibddi -f <iso-path> -k <kickstart-path> --- Add a customized kickstart file to the ThinESX/Recovery CD ISO

Where:
VMvisor-dd - The VMvisor dd image that is going to be customized

VMvisor-dev - The VMvisor device that is going to be updated

vib-path - The local file path to the vib

metadata-URL - The URL to the metadata.zip file (Ex. http://www.oem.com/depot/metadata.zip)

bulletin-ID - The bulletin ID to install

bundle-path - The local file path to the offline bundle

proxy (OPTIONAL) - Proxy used to download vib, for update operation only

-g (OPTIONAL) - Generate customized ThinESX/Recovery CD ISOs

-n (OPTIONAL) - Bypass signature check, for update operation only

export-path - Directory to export boot modules

alternate-conf (OPTIONAL) - Alternate export configuration file

kernel-opt - VMkernel option

license-path - vmware.lic file (Format: 00000-00000-00000-00000-00000)

iso-path - The local file path to the ThinESX/Recovery CD ISO

kickstart-path - The local file path to the kickstart file

Here are a few examples of using the vibddi tool:

Mount ESXi 4.1 ISO to extract the DD image:

[vi-admin@autodeploy scratch]$ sudo mount -o loop VMware-VMvisor-Installer-4.1.0-260247.x86_64.iso /mnt/iso/

Unzip the DD image and extract to current directory:

[vi-admin@autodeploy scratch]$ sudo bunzip2 -c /mnt/iso/imagedd.bz2 > imagedd

You now should have the DD image called imagedd located in your current working directory.You can name the file anything you want, but I'm using the suggested name as noted in the Auto Deploy documentation.

To list vibs installed on the image, you'll use the following command:

sudo vibddi -i [imagedd] -q

Here is an example of the vibs installed with default installation of ESXi 4.1:

To inject the image with an offline bundle, you'll use the following command:

sudo vibddi -i [imagedd] -o [offline_bundle] -n

Note: The -n flag should be used when performing updates as it bypasses the signature checks, else you will get an error.

Here is an example of injecting the Cisco Nexus 1000 Virtual Ethernet Module offline bundle as part of the default ESXi 4.1 installation:

We can confirm the Cisco VEM is part of the default image by running the query command again:

To inject the image with a single VIB, you'll use the following command:

sudo vibddi -i [imagedd] -v [vib] -n


Here is an example injecting the Cisco Nexus 1000 Virtual Ethernet Module VIB as part of the default ESXi 4.1 installation:

To inject VMkernel boot parameters, you'll use the following command:

sudo vibddi -i [imagedd] -t [vmkernel_option]


Note: Here is a list of a few VMkernel options documented by Dave Mishchenko. The -t argument only accepts one VMkernel option at a time. If you want to updated more than one option, you will need to run the command for each VMkernel option.

With a default installation of ESXi 4.1, there are no VMkernel options defined. To see whether or not these have been defined, you will need to login to Tech Support Mode and view boot.cfg:

~ # cat bootbank/boot.cfg
kernel=b.z
kernelopt=
modules=k.z --- s.z --- c.z --- oem.tgz --- license.tgz --- m.z --- state.tgz --- vpxa.vgz --- aam.vgz
build=4.1.0-260247
updated=1
bootstate=0


Here is an example of injecting the following two VMkernel options: noACPI and nopowerManagement:

To inject a license file, you'll use the following command:

sudo vibddi -i [imagedd] -l [license_file]


Note: The license file must contain a single entry using the following format - 00000-00000-00000-00000-00000

Here is an example of injecting license file:

To inject a custom kickstart configuration, you'll use the following command:

sudo vibddi -f [esxi-iso-path] -k [kickstart_file]


Here is an example of injecting a custom kickstart file:

Note: This actually injects a custom ks.cfg into the ESXi .iso which can then be used to deploy an ESXi host including the custom configurations found in the kickstart file. A brand new .iso will be created in the current working directory which includes the timestamp of kickstart injection as part of its filename.

We now can loop mount the new .iso and verify the custom kickstart has been injected:

Note: I'm using the sample ks.cfg found on Kendrick Colemans's site.

You can also extract certain items from the DD image, you'll use the following command:

sudo vibddi -i [imagedd] -e [export-path]


Here is an example of extracting the entire DD image to a temporarily directory:

To check the bootbank filesystem, you'll use the following command:

sudo vibddi -i [imagedd] -c


Here is an example of verifying bootbank filesystem:

Once the imagedd has been updated with all the drivers, you will need to compress the image back to .bz2 using bzip2. From here, you will have two options: A) copy the modified imagedd.bz2 over to your PXE/TFTP server used for automated kickstart installation B) Create a new ESXi .iso, there are a bunch of tutorials online such as here and here.

If you need to troubleshoot or would like to view the process of vibddi, you can take a look at the logs stored in /var/log/vibddi.log. You can also see the injection process which includes both informational and debugging logs in /var/log/esxupdate.log.

As you can see, this tool is extremely useful for injecting and customizing ESXi images. Hopefully one day VMware will officially release this tool and make it available on both UNIX/Linux and Windows platform so that everyone can benefit. For now, if you want to use vibddi, you will need to download and use Auto Deploy appliance. Looks like I'll be keeping this appliance around ;)

What is the VMware VIX API and it's future?

Some recent twitter conversations about vSphere SDK improvements, specifically around VMware VIX API and few emails around the use cases was the motivation for this post.

I will provide a very high level of what the VMware VIX API is, since there is a awesome presentation by Matt LaMantia at TechExchange in VMworld 2010 SF that goes into more details about VIX (video below). The VIX API (Virtual Infrastructure eXtension) is an API that provides guest management operations inside of a virtual machine that maybe running on VMware vSphere, Fusion, Workstation or Player. These operations are executed on behalf of the vmware-tools service that must be running within the virtual machine and guest credentials are required prior to execution.

The current incarnation of the VIX API is a little odd because it provides guest management operations which is unique to the VIX API but it also provides virtual machine operations. These virtual machine operation overlaps with some of the functionality provided by the vSphere API which is used for managing your vSphere infrastructure. Part of this overlap was due to origins of VIX which started from Workstation on the desktop. I almost consider this VMware API sprawl :)

Currently if you want to automate your vSphere infrastructure and want hooks into your virtual machine and guestOSes you need to leverage both the vSphere API and VIX API. I have seen this cause some confusion in the communities. Users wanting to standardize on VIX API and realize that 80% of what they want to do is available in VIX but the portion requires the use of the vSphere APIs. Don't get me wrong, the functionality of VIX is very powerful. When I first learn in version 1.6.2 of VIX where it officially supported vSphere, a whole new set of possibilities just opened up for administrators and developers.

There are several ways of using the VIX API today. If you are using PowerCLI, there are actually a few cmdlets that directly integrate with VIX, which requires you to install the VIX client libraries local to your PowerCLI installation. One such cmdlet is the Invoke-VMScript and here are a few example use cases:

• http://www.virtu-al.net/2010/02/05/powercli-changing-a-vm-ip-address-with-invoke-vmscript/
• http://www.jasemccarty.com/blog/?p=477

If you don't prefer the dark side ;) and want to run something like the vSphere SDK for Perl, C or COM, the VIX client libraries also includes client side bindings to these languages. If you are a Java shop and want to leverage Steve Jin's VI Java there is also an open source project called VIX Java Toolkit that can be used. Here is an example use case with vSphere SDK for Perl:

• http://www.virtuallyghetto.com/2010/07/automate-update-manager-operations.html

You can also just install the VIX client libraries which also includes a pre-compiled binary called vmrun that provides majority of the VIX operations all bundled into one utility. It also supports VMware vSphere, Workstation, Fusion and Player. Here is an example use case:

• http://professionalvmware.com/2008/12/vmware-vix-changing-ips-of-a-guest-vm/

There is also a very interesting application called VGC (Virtual Guest Console) created by the VMware Lab guys also known as flings. This is a graphical interface similar to VMware Remote Console, but it is 1000x better and provides integration to all the VIX operations. The application allows you to view running processes in a guest, kill a particular process, deploy application to a guest, download/upload files and browse guest filesystem and much much more. I would highly recommend taking a look at this tool!

As you can see, the possibilities are endless! but you still have to use two separate APIs. I along with others in the community have asked for a consolidation of the VIX API and merging with the vSphere API, makes the most sense. I think our feedback has finally been heard and if you watch the presentation given by Matt, you will see that future of VIX API is exactly that, consolidatation with the vSphere API. Though we will not see it anytime soon until the next major release of vSphere dubbed vSphere.next (MN) and vSphere.next^2 (OP), it is a change I am looking forward to.

One other interesting thing to point out, during the first release of the VIX API that supported vSphere which was 1.6.2, there was a tiny bug that was identified regarding licensing. As you may or may not know, VMware requires that you have a "paid" license to be able to leverage the various APIs and CLIs for automation, configuration and management of your vSphere infrastructure. This is generally not an issue when dealing with vCenter or ESX, but with ESXi, you have option of a free license.

In the VIX 1.6.2 release, you actually have full VIX read and write operations against an ESXi host using the free license. This was of course fixed in subsequent releases and the trick only works with ESXi 4.0 or older. You will notice on the VIX landing page, there is no mention of the release notes for 1.6.2 release other than 1.7.1 superseding it. You however still can download 1.6.2 release under VIX 1.6 and you can still see the release notes if you search on VMware's site or on Google.

For more details about VIX and downloads, take a look at the following:

• http://www.vmware.com/support/developer/vix-api/
• http://blogs.vmware.com/vix/2008/07/what-is-vix-and.html

PPC15 Guest Operations Using VMware VIX APIs and Beyond:


Guest Operations using VMware VIX APIs and Beyond from heyitspablo on Vimeo.

How to obtain GID and LWID from esxtop?

Last week I remember seeing a tweet from Duncan Epping regarding the use of VMware vsish to obtain the VMX Cartel ID, also known as the LWID (Leader World Id) for a virtual machine world. This VMX Cartel ID is then used to obtain the GID (Group Id) which was documented by Duncan on his esxtop page to limit the view in esxtop to a particular VM of interest:

VMWID=`vm-support -x | grep  |awk '{gsub("wid=", "");print $1}'`
VMXCARTEL=`vsish -e cat /vm/$VMWID/vmxCartelID`
vsish -e cat /sched/memClients/$VMXCARTEL/SchedGroupID

As you can see from the above, Duncan utilizes the vm-support to obtain the WID (World ID) and then using vsish to query for the VMX Cartel ID (LWID). Finally, using LWID, he was able to obtain the GID (Group ID). This example only applies to ESXi, since classic ESX does not include vsish tool.

Here is an example of where to locate the values in esxtop under CPU section:

I remember during some of my skunk work adventures, there were other methods of obtaining these IDs. Due to my VCAP-DCD cramming last week, I did not get a change to further investigate. Now that I have some time, I thought share some of these other methods for obtaining the GID and LWID in both ESX and ESXi.

ESXi

Obtaining VMX Cartel ID (LWID)

vmdumper - This will list all running VMs including the path to the .vmx configuration file, the display name of your VM and the VMX Cartel ID

~ # /sbin/vmdumper -l 

wid=16881       pid=-1  cfgFile="/vmfs/volumes/4cdeeb09-1ad4c18a-5ff9-003048d9586a/scofield/scofield.vmx"       uuid="42 34 e6 bc f1 83 c2 db-fb fb 08 73 b7 0c 26 40"       displayName="scofield"  vmxCartelID=16880 

ps - The VMX Cartel ID is actually both the PID and CID within process status in ESXi

~ # ps -Cc | grep "scofield" | grep -v grep | awk '{print $2}' 

16880

vscsiStats - The VMX Cartel ID is also used in identifying the VMs when displaying vscsi information

~ # /sbin/vscsiStats -l | grep "scofield" | awk '{print $4}' | sed 's/,//g' 

16880

esxcli - You can obtain pretty much the same information as in vmdumper using the new "vm" option in esxcli

~ # /sbin/esxcli vms vm list | grep -A3 "scofield" | grep Cartel | awk '{gsub("VMX Cartel ID:", "");print $1}' 

16880

Obtaining GID

Method1 - Using vscsiStats and esxcfg-info to extract VM's GID

~ # VM_NAME=scofield;VMX_CARTEL=$(/sbin/vscsiStats -l | grep "${VM_NAME}" | awk '{print $4}' | sed 's/,//g');esxcfg-info -r -F perl > /tmp/esxcfg-info.txt;grep -B1 "vm.${VMX_CARTEL}" /tmp/esxcfg-info.txt | head -1 | awk '{print $3}' | sed 's/,//g'

4639

Method2 - Using esxcli and esxcfg-info to extract VM's GID

~ # VM_NAME=scofield;VMX_CARTEL=$(/sbin/esxcli vms vm list | grep -A3 ${VM_NAME} | grep Cartel | awk '{gsub("VMX Cartel ID:", "");print $1}');esxcfg-info -r -F perl > /tmp/esxcfg-info.txt;grep -B1 "vm.${VMX_CARTEL}" /tmp/esxcfg-info.txt | head -1 | awk '{print $3}' | sed 's/,//g'

4639

Method3 - Using esxcli and vsish to extract VM's GID

~ # VM_NAME=scofield;VMX_CARTEL=$(/sbin/esxcli vms vm list | grep -A3 ${VM_NAME} | grep Cartel | awk '{gsub("VMX Cartel ID:", "");print $1}');/sbin/vsish -e cat /sched/memClients/${VMX_CARTEL}/SchedGroupID
4639

Note: You just need to substitute VM_NAME variable with the display name of of the virtual machine you are interested in. There are actually multiple commands being executed in this one line. If your VM has spaces, make sure you put quotes around it

ESX

Obtaining VMX Cartel ID (LWID)

vmdumper - This will list all running VMs including the path to the .vmx configuration file, the display name of your VM and the VMX Cartel ID

[root@himalaya ~]# /usr/lib/vmware/bin/vmdumper -l
wid=29250   pid=-1     cfgFile="/vmfs/volumes/4a48004d-f9af7fa0-5bbf-003048d9586b/STA202G/STA202G.vmx"  uuid="42 30 d1 75 c5 3e 81 2a-14 15 1f 86 bb 5b b9 e5"       displayName="STA202G"  vmxCartelID=29249

vscsiStats - The VMX Cartel ID is also used in identifying the VMs when displaying vscsi information

[root@himalaya ~]# /usr/lib/vmware/bin/vscsiStats -l | grep "STA202G" | awk '{print $4}' | sed 's/,//g'

29249

esxcli - You can obtain pretty much the same information as in vmdumper using the new "vm" option in esxcli

[root@himalaya ~]# /usr/sbin/esxcli vms vm list | grep -A3 STA202G | grep Cartel | awk '{gsub("VMX Cartel ID:", "");print $1}'

29249

Obtaining GID

Method1 - Using esxcli and esxcfg-info to extract VM's GID

[root@himalaya ~]# VM_NAME=STA202G ;VMX_CARTEL=$(/usr/sbin/esxcli vms vm list | grep -A3 ${VM_NAME} | grep Cartel | awk '{gsub("VMX Cartel ID:", "");print $1}');esxcfg-info -r -F perl > /tmp/esxcfg-info.txt;grep -B1 "vm.${VMX_CARTEL}" /tmp/esxcfg-info.txt | head -1 | awk '{print $3}' | sed 's/,//g'

197

Method2 - Using sched-stats to extract GID

[root@himalaya ~]# VM_NAME=STA202G;VMX_CARTEL=$(/usr/lib/vmware/bin/vscsiStats -l | grep "${VM_NAME}" | awk '{print $4}' | sed 's/,//g');/usr/bin/sched-stats -t groups | grep "vm.${VMX_CARTEL}" | awk '{print $1}'

197

Method3 - Using /proc information from sched

[root@himalaya ~]# VM_NAME=STA202G ;grep "${VM_NAME}" /proc/vmware/sched/drm-stats  | awk '{print $1}'

197

As you can see, there are more than one way to obtain the same exact values and I am sure there are probably a few others. For command simplicity, I would probably recommend method #3 for ESXi and method #3 for ESX. As with anything, be careful when you using these methods as they are not really supported by VMware unless directed by their support organization.

New sponsor: Veeam

virtuallyGhetto is proud to announce our very first blog sponsor, Veeam! Veeam provides management and data protection software for your virtual infrastructure. They have a variety of products that include both paid and free solutions such as the popular Veeam Backup & Replication, Veeam Reporter and Veeam FastSCP. Check out their site if you would like to learn more.

Thank you for your support Veeam and welcome!

How to configure and use vMA's vi-fastpass with fpauth and adauth on vSphere 4.1

From time to time, I see users posting on the VMTN forums with some questions and confusion around the proper implementation and functionality of vMA's vi-fastpass. The confusion is further enhanced with the new Active Directory functionality and integration with vMA's new vi-fastpass type called adauth.

The vi-fastpass component found in vMA is a credentials caching mechanism to allow you to connect to your ESX(i) or vCenter servers. Prior to vMA 4.1, vMA 4.0 only supported one type of vi-fastpass which is just called fpauth (fastpass authentication). This fpauth basically allows you to manage an ESX(i) or vCenter server under vMA by creating a vi-adminXX and vi-userXX account. The password for these two accounts are obfuscated using a simple XOR cipher. A user can now initialize one of these managed targets and execute either vCLI or vSphere SDK for Perl scripts without having to specify credentials each and every time, this works because the vi-adminXX credentials are being used to connect to your target. This can make running a simple command across n-number of hosts simple without having to provide the credentials for every host.

In vMA 4.1, a new vi-fastpass type has been introduced called adauth, in which you can leverage your Windows Active Directory credentials to authenticate against a managed target without having to store an insecure cipher in vMA. The concept works exactly the same as fpauth, but there are some additional pre-requisites that must be met before you take advantage of adauth. You need to ensure that your ESX(i) or vCenter server is joined to your Active Directory and that your vMA host is also joined to that same Active Directory domain. Once this is all configured, you will now be able to login to vMA using your AD credentials and execute a vCLI command or vSphere SDK for Perl script against an ESX(i) or vCenter host and your AD credentials will be used to authenticate against your target without using any stored cached credentials. This helps to centralize your user management and is much more secure than using the standard fpauth where the credentials are stored in a simple XOR cipher which can easily be decrypted.

The following will describe instructions on setting up and verifying both standard fpauth and adauth using vMA 4.1 and vSphere 4.1 (ESXi and vCenter server). Before continuing, it is highly recommended that you take a look at both the vSphere 4.1 and vMA 4.1 documentation with regards to Active Directory integration.

Configuring vMA vi-fastpass using fpauth (fastpass authentication):

1. Login to vMA using vi-admin credentials, note that your user context is with the vi-admin account.

2. Use the vifp addserver command to add an ESX or ESXi host, you will be using the root account to do so and provide the credentials to that account. If you do not receive any errors, you have successfully added the target.

3. Using vifp addserver again, we now add a vCenter server and we will be using a local account found on the host. In this example, we will be using the administrator account and provide the credentials to that account.

Note: You do not need to add both your ESX(i) and vCenter server to utilize vi-fastpass, this is just an example of adding both types of supported targets.

4. Next, we use vifp listservers to display the managed targets and the type of vi-fastpass we are using denoted by fpauth or adauth.

5. Earlier we mentioned that when using fpauth, the stored credentials are encrypted using a simple XOR cipher. You can view these entries by taking a look at /home/vi-admin/.vmware/credstore/vmacredentials.xml.

6. To use a target, you need to first initialize the target by using the vifptarget -s command. This will change your command prompt in which it will display the current target that has been initialized.

7. As you can see from the screenshot, we now have initialized the host esxi4-1.primp-industries.com and now we can run a vCLI command without having to specify the credentials to the host. In this example, we will just list the number of vmnics using esxcfg-nics -l command.

8. We can also perform the same operation against the ESXi host by going through vCenter. To do so, you will need to initialize your target to your vCenter server and specify the --vihost parameter to the name of a managed ESX or ESXi host under vCenter.

As you can see, you perform this vi-fastpass using fpauth against an ESX(i) or vCenter server. When you initialize a target, the command prompt tells you which host context you are currently in. If you decide you want to switch context, you can issue the vifptarget -c command which clears the target.


Configuring vMA vi-fastpass using adauth (Active Directory authentication):

1. Login to your ESX or ESXi host using the vSphere Client, you will need to first join your ESX(i) host to your AD domain. Click on Configuration tab and on the lower left hand side, click on Authentication Services and click on Properties. You will need to select Active Directory as the type and specify the Domain and click Join Domain. You will now be prompted to provide credentials that have the permissions to add a new host to the domain.

2. If you joined the host to your AD domain successfully, you should see the following displaying the domain.

3. When you join an ESX or ESXi host to AD, it actually does a group lookup for "ESX Admins", if it exists, it will assign the Administrator role to it. If it does not exist, it will periodically query your AD until one is created. In my lab environment, it was not created automatically and I had to manually create this group. For more details, take a look at this blog post by Maish - http://technodrone.blogspot.com/2010/07/esxi-41-active-directory-integration.html

4. Next, you will need to either add the individual users or group that will utilize the vi-fastpass with adauth. You need to do this for each and every ESX or ESXi host under the Permissions tab. If you have more than one user, I would suggest creating and Active Directory group and adding all the users into that group so you add only the group. You will right click and Add Permission and specify your AD domain and select either the individual users or groups and make sure you select Administrator role. Once you have added the appropriate users, you should see the following.

Note: At this point you should be able to login to your ESX(i) host using AD credentials using the vSphere Client and remember to use DOMAIN\username. If you are unable to login using your AD credentials, you will not be able to proceed with the next step as this is required for vi-fastpass adauth to function.

5. You will login to your vMA host using the vi-admin account. You will now join your vMA host to your Active Directory domain using the domainjoin-cli join command. You will need to use sudo and specify the name of your domain and the username which as the permissions to join a host to your AD domain.

6. To confirm you have successfully joined vMA to your AD domain, you can run the domainjoin-cli query command to list the domain. You can also verify the new computer name in your AD server.

7. Now we add an ESXi host to vi-fastpass but instead of using fpauth, we will leverage adauth. You will need to use the vifp addserver command and specify --authpolicy adauth parameter. You will need to specify an AD account that will be utilizing the vi-fastpass, make sure you specify the DOMAIN_NAME\username. If you decide to use the --username parameter, make sure you escape the forward slash with another slash (e.g. --username PRIMP-IND\\primp). You would repeat this if you plan on adding a vCenter Server that is joined to the same domain.

8. We can also verify that the vi-fastpass targets have been added using adauth by using the vifp listservers -l command which displays the managed targets and the type of vi-fastpass they have been added with.

9. Since we are using adauth, we can confirm that no cached entries should be stored locally within the vMA host by looking at /home/vi-admin/.vmware/credstore/vmacredentials.xml which should be blank now.

10. Now, we are ready to login to vMA using our AD credentials. Depending if you implemented this vMA AD Intergration Tip, you may not need to specify the full domain when logging in. If you did not, you would to need to specify the full DOMAIN\username@vMA-hostname when logging in.

11. You will now use the vifptarget -s command to initialize an ESX(i) server just like in our fpauth use case and you now should be able to run esxcfg-nics -l without having to provide any additional credentials.

12. We can also use the vifptarget -s command to initialize a vCenter server just like in our fpauth use case and specifying --vihost parameter to either your ESX(i) host and now should be able to run esxcfg-nics -l without having to provide any additional credentials.

Hopefully by outlining the entire process for both vi-fastpass fpauth and adauth, implementing this functionality will be a breeze now.

Additional Caveats: There have been some users who have not been able to get full vi-fastpass with adauth working with ESXi host. There have been a few threads on the VMTN forums including this one which refers to a potential VMware bug. I have not been able to reproduce the issue noted by the user, but I have seen several cases in which the above workflow is not the behavior that users have seen. If you run into any issues, I would recommend before trying again to unjoin both your ESX(i) host and vMA from your AD domain. You will also want to make sure you go into your AD server and delete these entries before trying again.

Hidden HA and VPXA Configurations

Applying the strings method as described in my last article, I decided to also take a look at /opt/vmware/vpxa/vpx/vpxa binary to see if there were anymore hidden goodies. To my surprise, I was able to locate additional HA and VPXA advanced configuration options. While going through and testing some of the HA advanced options, I found that only 19 out of 47 have not been documented and much of the documentation I found online was from Mr. Duncan Epping's blog. This really shows how open Duncan has been around the advanced options with VMware HA, if only VMware as a whole could be so open with the other advanced options that are used throughout VMware but left undocumented.

I do have to stress, these are configurations that are not documented and probably not supported unless directed by VMware. You should be very careful if you decide to play with some of these options and ensure you do not test on a production environment, don't say I did not warn you :)

Here is the list of 47 HA Advanced Configurations, the ones with links have been publicly documented by either Duncan or VMware. The rest have not been documented and I have done some testing which will be discussed further.

das.allowNetwork
das.allowVMotionNetworks
das.appMonFailureInterval - #
das.bypassNetCompatCheck
das.checkVmStateDelay - # (default 120)
das.consoleNode - [fqdn]
das.consolePerm - [PERM_USER,PERM_OPER,PERM_ALL]
das.consoleUser - [username]
das.defaultFailoverHost
das.disableUWSwapRequirement
das.failureDetectionInterval
das.failureDetectionTime
das.failureInterval 
das.ignoreRedundantNetWarning 
das.includeFTcomplianceChecks
das.iostatsInterval
das.isolationAddress 
das.isolationShutdownTimeout
das.maxFailures
das.maxFailureWindow
das.maxftvmrestartcount - # (default 5)
das.maxFtVmsPerHost
das.maxvmrestartcount
das.minUptime
das.perHostConcurrentFailoversLimit
das.perHostVMLimit - #
das.powerOffOnIsolation
das.preferredPrimaries
das.primaryCount - [2,3,4,5] (default 5)
das.restartPriority 
das.sensorPollingFreq
das.sim.enabled
das.sim.numVms
das.sim.powerOpsPerMinute
das.sim.resourceOpsPerMinute
das.sim.sendInterval
das.slotCpuInMHz
das.slotMemInMB 
das.slotNumVcpus
das.trace - [on,off] (default on)
das.traceLevel - [0,1,2,3,4,functiontracing] (default 3)
das.traceOutput - [stdout, eventLog, file] (default file)
das.useDefaultIsolationAddress
das.vmCpuMinMHz 
das.vmFailoverEnabled
das.vmMemoryMinMB

You can actually use AAM's (VMware HA) cli utility to view some of these hidden configurations, by using /opt/vmware/aam/bin/Cli

Here is a screenshot of the default VMware HA configuration using the getrule VMWareClusterManager command to list cluster rules, make a note of the parameters with red arrows:

 Note: I found it interesting that the "VMWareClusterManager" had an upper case "W" in VMware. It looks like this name was not vetted by VMware marketing :)

Next, we add some of the advanced HA configurations using the vSphere Client under Advanced Options (HA) section:

Note: das.primaryCount only supports values 2 through 5, if you try to specify any other value, you will get an error. VMware has selected 5 primary nodes for a specific reason, you should not try to change it from the default.

We reconfigure the cluster with these new HA options and re-run getrule VMWareClusterManager and now we see the values get updated along with some new ones being added:

Note: You can also see some of these changes in /var/log/vmware/aam/aam_config_util_addnode.log

You can also use AAM cli to set some of these configurations:

Usage: setRuleVar|srv <ruleName> <name> <value>
AAM> setRuleVar VMWareClusterManager ft_Trace on

The advanced VPXA configurations applies to the vCenter Agent configuration file located in /etc/opt/vmware/vpxa/vpxa.cfg on an ESX(i) host. There was a total of 47 with only 8 being used in the default vpxa.cfg file.

Here is the list of 47 VPXA Configurations, the ones listed in blue are used by default, the ones with links are documented by VMware and the rest are undocumented:

vpxa.alarm.checkFrequency
vpxa.alarm.powerOnSilence
vpxa.bundleVersion
vpxa.cfg
vpxa.checkAAMPeriod
vpxa.checkNodeResourcePeriod
vpxa.connectTo
vpxa.disableOldQuickStats
vpxa.editionCheckTime
vpxa.ft.cleanupTimeout
vpxa.getchanges.delay
vpxa.getchanges.envbrowserRefreshRate
vpxa.getchanges.timeout
vpxa.healthSystem.throttleInterval
vpxa.heartbeat.interval
vpxa.HostdCnxBypassProxy
vpxa.hostdMonitorPeriod
vpxa.hostIp
vpxa.hostKey
vpxa.hostPort
vpxa.ignoreAamErrorCount
vpxa.ignoreAamErrorTime
vpxa.ignoreAamShutdownCount
vpxa.licenseExpiryNotificationThreshold
vpxa.maxEventReportingDelaySeconds
vpxa.maxFds
vpxa.memoryCheckerTimeInSecs
vpxa.mmapThresholdInKB
vpxa.nodeResourceThreshold
vpxa.partialVmUpdates
vpxa.preserveServerIp
vpxa.respool.changeThreshold
vpxa.respool.changeThresholdInMB
vpxa.respool.enableOverheadLimitTightLoop
vpxa.serverIp
vpxa.serverPort
vpxa.soapAdapterOverNamedPipe
vpxa.soapAdapterPort
vpxa.sslVersion
vpxa.StopMemInMB
vpxa.timeDiffThreshold
vpxa.useSsl
vpxa.vmapTimeoutCountThreshold
vpxa.vmapTimeoutTimeThreshold
vpxa.vmImages
vpxa.vmotion.vmIdAcquireTimeout
vpxa.WarnMemInMB

Note: The VPXA parameters is pretty difficult to reverse engineer and figure out what values each parameter can accept. I did notice the "periods" within each key actually represents the XML node separation within the vpxa.cfg.

Taking vpxa.vmotion.vmIdAcquireTimeout would translate to the following:

<vpxa>
    <vmotion>
         <vmIdAcquireTimeout>600</vmIdAcquireTimeout>
    </vmotion>
</vpxa>

Again, please use extreme caution if you decide to experiment with these parameters and happy hacking :)