How to Automate the disabling of the VAAI UNMAP primitve in ESXi 5

I just saw an interesting article on Jason Boche's blog about VMware's recall of the new VAAI UNMAP primitive for vSphere 5. VMware released a new KB article KB2007427 documenting the details along with a recommendation to disable the VAAI UNMAP primitive for now (a patch will be released in the future to automatically disable this until the issue is resolved).

One thing that caught my in the VMware KB article is to disable the VAAI UNMAP primitive, you need to manually login to ESXi 5 Tech Support Mode (SSH) to run the local version of esxcli command. This is trivial if you have several hosts, but it can be time consuming if you have several hundred hosts to manage. Even though the "VMFS3.EnableBlockDelete" is a hidden parameter that can not be seen using any of the supported utilities, you can enable and disable the property using the remote version of esxcli which is part of vMA 5 or vCLI 5.

Here is an example of the command when connecting directly to an ESXi 5 host:

esxcli --server himalaya.primp-industries.com --username root system settings advanced set --int-value 1 --option /VMFS3/EnableBlockDelete

Here is an example of command when connecting to vCenter Server:

esxcli --server reflex.primp-industries.com --vihost himalaya.primp-industries.com --username administrator system settings advanced set --int-value 1 --option /VMFS3/EnableBlockDelete

As you can see, you can easily wrap this in a simple for-loop to disable the VAAI UNMAP primitive. So here is a script to help with exactly that called vaaiUNMAP.sh

The script accepts 4 parameters:

1. A list of ESXi 5 hosts to enable or disable VAAI UNMAP primitive
2. Name of the vCenter Server managing the ESXi 5 hosts
3. vCenter auth file which contains the username/password 
4. 0 to disable or 1 to enable VAAI UNMAP primitive

The auth file is just a file that contains the following:

VI_USERNAME=administrator
VI_PASSWORD=y0mysuperdupersecurepassword

Here is an example of disabling VAAI UNMAP primitive on 3 ESXi hosts being managed by a vCenter Server:

To help extract all ESXi 5 hosts from your vCenter Server, you can use the following vSphere SDK for Perl getESXi5Hosts.pl

Here is an example of running the script and just save the output to a file:

One of the unfortunate thing about the VMFS3.EnableBlockDelete is that it is a hidden parameter along with others, so it will not automatically display when using local or remote ESXCLI, but thanks to Craig Risinger, you can still get the information using the remote ESXCLI by specifying the --option parameter which is great because you do not need to login to the ESXi Shell to retrieve the information.

esxcli --server himalaya.primp-industries.com --username root system settings advanced list --option /VMFS3/EnableBlockDelete

I would also recommend adding an entry into your ESXi 5 kickstart to automatically disable VAAI UNMAP by default until a fix is released.

%firstboot --interpreter=busybox

#disable VAAI UNMAP primitive
esxcli system settings advanced set --int-value 0 --option /VMFS3/EnableBlockDelete

How to Install VMware VSA with Running VMs

For those of you who want to quickly test out the new VMware VSA (vSphere Storage Appliance) will notice that you can not just throw a few ESXi 5 hosts that have running virtual machines on them. If you try to proceed with the VSA installation, you will see an error message regarding the presence of virtual machines whether they are running or not.

This can make it difficult to evaluate or test the new VSA if you do not have additional hosts that can be easily re-deployed as vanilla ESXi 5 installations. While working on the previous article How to Install VMware VSA in Nested ESXi 5 Host Using the GUI, I decided to test out the behavior of a few other configuration variables found in the dev.properties file for the VSA Manager. It turns out that you can actually disable the host audit check which includes the validation of running virtual machines by changing the host.audit variable from "true" to "false" using the same trick documented here. You will need to restart the VSA Manager and then the vCenter Server service for the change to go into effect.

**** DISCLAIMER: This is not supported by VMware and there maybe specific checks that are now bypassed by disabling the host.audit parameter. Please use at your own risk and test before deploying on actual systems **** 

One interesting observation made while testing this in a nested ESXi configuration is that even though there is a message warning the user that any data found on the local VMFS volumes will be deleted, I did not see any process that was kicked off to do so. This does not mean this was not the original intention, but there was no reformatting of the local VMFS or removal or powering off of the running virtual machines. While testing both a "supported" and "ghetto" installation of the VSA, I found that several advanced settings were updated as part of the VSA installation, you should see the same if you look in the vmkernel.log of one of the ESXi 5 hosts:
  

2011-09-23T17:36:33.030Z cpu0:3475)Config: 346: "HostLocalSwapDirEnabled" = 0, Old Value: 0, (Status: 0x0)
2011-09-23T17:38:00.971Z cpu0:3258)Config: 346: "HeartbeatPanicTimeout" = 60, Old Value: 900, (Status: 0x0)
2011-09-23T17:38:07.069Z cpu1:2851)Config: 346: "EnableSVAVMFS" = 1, Old Value: 0, (Status: 0x0)
2011-09-23T17:38:07.090Z cpu1:2851)Config: 346: "VmkStressEnable" = 0, Old Value: 1, (Status: 0x0)
2011-09-23T17:44:22.163Z cpu1:3477)Config: 346: "SIOControlFlag2" = 1, Old Value: 0, (Status: 0x0)

One that sparked my curiosity is EnableSVAVMFS which is a hidden setting found on the ESXi host but one can view it using vsish. Per the limited documentation found in vsish, this parameter is to enable some sort of optimization with the local VMFS volume.

Thanks to @VMwareStorage (Cormac Hogan, VMware Technical Marketing for Storage) for the quick answer to my question on twitter, it looks like this parameter does the following:

"Forces linear allocation of VMDKs on local VMFS for VSA. Improves mirroring performance across VSAs apparently"

There was nothing in the vmkernel.log that would indicate the local VMFS was reformatted or files had to be delete to support the VSA installation. I can understand why VMware wanted a vanilla installation which included no running VMs to simplify the installation process. Another reason that I can think of is by having some initial storage consumption, it can offset the amount of "available" storage that needs to be setup on VSA cluster. The amount of available storage per host must be equal on all two or three node cluster, to ensure there is sufficient space for replication. As long as you understand by having running virtual machines on one or more ESXi nodes, the node with the smallest amount of free physical storage is what the rest of the VSA nodes will be configured to.

You potentially may also find yourself in a chicken and the egg problem if VSA installation fails to install and reverts it's changes, which includes putting the ESXi host into maintenance mode. This will cause it to fail on the node that is running the vCenter Server and VSA Manager, another reason you would want to run the management system outside of the VSA cluster.

Without further ado, I recorded a quick 6minute video demonstrating the installation of the new VMware VSA on ESXi 5 hosts that has running virtual machines which includes the vCenter Server and VSA Manager running on one of the nodes (video is awesome when you bump up the audio):

Installing VMware VSA with Running VMs from lamw on Vimeo.

Not only is this not supported, but it is also NOT a best practice to run the vCenter Server and VSA Manager within the VSA cluster because you may potentially have issues with replication if vCenter and VSA Manager goes down. In my testing, I found that I could take down vCenter and VSA Manager and the NFS volumes continue to function and the cluster continues to churn away. Any virtual machines running on the VSA volumes will automatically be restarted by vSphere HA. Once the VSA manager has recovered, it'll automatically ensure the volumes have all synchronized and re-protect the VSA cluster.  

Note: It is important to understand that even though you can install the VMware VSA with running virtual machines using the hack above, the requirement of a vanilla ESXi 5 installation is still 100% mandatory. You MUST still have only a single vSwitch (vSwitch0) with only a single vmnic (vmnic0) connected to the vSwitch with only two default portgruops that must exists: "VM Network" and "Management Network", there is no workaround for this requirement. If you you have a host that you plan on running VMs prior to VSA installation, make sure they are on the "VM Network" portgroup as additional portgroups are not supported prior to installation of the VSA.

I am hoping that some of these requirements are relaxed in a future release of the VMware VSA and possibly a version that would work with the vCVA (vCenter Virtual Appliance). For now if have limited hardware or would like to use existing ESXi 5 host with running virtual machines (needs to be configured like a vanilla installation of ESXi 5) then you can run everything on either a two or three node cluster, just be aware of the caveats.

For more in-depth information and details about the new VSA, please check out the VMware Storage Blog - vSphere Storage Appliance Links and be sure to follow Cormac Hogan on twitter at @VMwareStorage

How to Query VM Disk Format in vSphere 5

Prior to vSphere 5, it was not trivial to identify the particular disk format for a given virtual machine's disk. Using the vSphere Client, you would see a virtual machine's disk be displayed as either thin or thick. The problem with this is that the "thick" format can be either:

• zeroedthick - A thick disk has all space allocated at creation time and the space is zeroed on demand as the space is used
• eagerzeroedthick - An eager zeroed thick disk has all space allocated and wiped clean of any previous contents on the physical media at creation time. Such disks may take longer time during creation compared to other disk formats.

Users would not be able to distinguish the exact type using the vSphere Client or the vSphere 4 APIs. With the release of vSphere 4, VMware did introduce a new property in the vSphere 4 API called eagerlyScrub which was supposed to help identify whether a virtual disk was allocated as an eagerzeroedthick disk. Unfortunately there may have been a bug with the property as it never gets modified whether a disk is created as zeroedthick or eagerzeroedthick.

The only method that I was aware of to truly figuring out the disk format would be to manually parse the virtual machine's vmware.log file to identify the disk type which I wrote a script for in 2009.

During the vSphere 5 beta, I had noticed the vSphere Client UI now properly displays all three virtual machine disk format: zeroedthick (displayed as flat), thin and eagerzeroedthick (displayed as thick).

Seeing that VMware now displays the three different formats, I wanted to see if it was possible to extract this using the vSphere 5 APIs and not have to rely on the hack of reading the vmware.log files. It turns out that the eagerlyScrub property is now functioning properly when a VMDK is provisioned or has been inflated/converted to the eagerzeroedthick format. I wrote a simple vSphere SDK for Perl script called getVMDiskFormat.pl which allows you to extract the disk formats of all virtual machines connecting to either vCenter or directly to an ESX(i) host.

The script allows for two types of output: console (directly on the console) or csv (creates .csv file)

If you select csv output, by default it will be stored in a file called "vmDiskFormat.csv". You also have the option of specifying the filename by using the --filename flag and providing a name of your choosing.

You can then load the csv file into excel and easily sort through the various disk format types.

All this is already included in the latest version of the VMware vSphere Health Check Report 5.0 if you want a centralize report that includes virtual machine disk format.

How to Install VMware VSA in Nested ESXi 5 Host Using the GUI

We upgraded the ghettoDatacenter to vSphere 5 this weekend and one of the things I wanted to play with was the new VMware VSA (vSphere Storage Appliance). Since we only have a single host, running nested ESXi would be our only option and this would allow us to easily deploy three vESXi 5.0 hosts and vCenter to tinker with the new VMware VSA.

UPDATE (09/16/12): You can use the same process outlined in this article to run the new VMware VSA 5.1 (vSphere Storage Appliance) in a nested ESXi configuration. Below is a screenshot of running VSA 5.1 in Nested ESXi 5.1.

One caveat in using vESXi hosts to test the VSA is during the selection of your ESXi hosts, VSA expects the hosts to be EVC capable. The VSA will create a vSphere Cluster and automatically enable EVC baseline based on your cpus. As you may or may not know, EVC can not be supported in a vESXi host and this would prevent you from selecting these hosts.

Luckily this issue was solved by a Vijay in his blog post here. There is a configuration file called dev.properties located in C:\Program Files\VMware\Infrastructure\tomcat\webapps\VSAManager\WEB-INF\classes which contains a line specifying whether or not EVC should be configured "evc.config". This configuration file appears to be used by VMware internally for some type of development but by changing the parameter from true to false, the VSA will support non-EVC capable hosts and not enable EVC for the VSA vSphere cluster.

Note: This is most likely not supported by VMware, please use at your own risk along with modifying any other values within this file.

Now, you might wonder if Vijay had already documented this process in his blog, why am I repeating it? Well the issue that Vijay had identified by tweaking this configuration file was that the VSA GUI installer did not detect the change and had to rely on an alternative method of installation using the commandline. Though not ideal, this method does work but for first time evaluators of the VMware VSA, the various commandline options can overwhelm or confuse users. It would be great if one could using the VSA GUI to perform the installation which is much more intuitive and that is reason for this article.

For the VSA to detect the new changes, you will need to restart the VMware VSA and then the vCenter Service under the Windows Services utility. I am not sure why both services need to be restarted, but I guess the VSA extension is not updated when just the VSA is restarted which is unfortunate. 

Once both services have started up, open a new vSphere Client session to your vCenter Server and proceed with the VSA installation. During the selection of the hosts, you will now have the option of selecting your vESXi 5 hosts and a warning message is presented stating "Unsupported Hardware" but the installer will allow you to continue on. 

After you have selected either two or three of your vESXi hosts, you will be prompted once more that this configuration is not supported nor in the VMware HCL, go ahead and click OK.

After this, you will be able to go through the rest of the VSA installation as long as meet the default requirements of the VMware VSA noted in the documentation.

So if you have some interests in the new VMware VSA and do not have physical hardware to test with, you should consider deploying a couple of vESXi hosts and kick the tires with the new VSA.

VMware officially releases vibddi for vSphere 4.1

There were several product releases last week that got a lot of buzz on the inter-tube:

• VMware Fusion 4
• VMware Workstation 8
• VMware vCloud Director 5
• VMware Site Recovery Manager 5
• VMware View 5
• VMware vFabric 5
• Microsoft Windows 8

However, VMware actually released an additional product last week which snuck under the radar, vibddi.

I actually wrote about this unsupported and undocumented utility last year: How to inject custom drivers into an ESXi 4.1 image using vibddi? vibddi (pronounced vib d-d-i) stands for VIB (vSphere Installation Bundle) Disk Dump Image and it is a utility to help users easily customize ESXi images with custom drivers. This utility first appeared in the vSphere Auto Deploy appliance and it looks like VMware has finally released it as an official tool to support vSphere 4.1 image customization. You also may have heard about the new Image Builder tool with the release of vSphere 5, the origins of that utility actually came from vibddi.

If you are still using vSphere 4.1 and need to inject or modify drivers, I would highly recommend you take a look at the tool as it is extremely simple to use. For more details, please check out the new VMware KB article 2003316 documenting the details of the utility or my blog post. If you are using vSphere 5, you will need to use Image Builder as vSphere 4.1 is not supported and vice-a-versa with ESXi 5 with vibddi.

Note: There are some changes in the latest vibddi utility compared to the one found in the vSphere Auto Deploy such as injecting custom kickstart configuration file or license file. If you rely on these features, you may want to use the older version or manually update these after the system build.

How to Run Windows 8 on vSphere 5

There's been a lot of hype/talk about Windows 8 and if you wanted to test drive the new OS, you might consider using the latest release of VMware Fusion 4.0.1 or VMware Workstation 8 as Windows 8 is an officially supported guestOS. Though what if you wanted to run it in your vSphere 5 environment? Well you can with a small hack.

Even though it's not listed as a supported guestOS, you can manually tweak the .vmx configuration to get ESXi 5 host to recognize the guestOS type. You just need to create a generic Windows 2008 system and then from the commandline or by exporting the .vmx using the datastore browser and then edit the configuration file. You will need to make the following change to the guestOS paramater:

guestOS = "windows8srv-32"
guestOS = "windows8srv-64"

One you have made this change, you will need to re-register the virtual machine or reload the configuration using vim-cmd vmsvc/reload operation.

Another method just using the vSphere Client without any modifications to the .vmx is to just create a virtual machine and select any guestOS type. Once the virtual machine has been created, there is actually an option in the guestOS to select Windows 8 32 or 64bit that can be selected. If you wish to automate through the commandline, then you can use the method above or you can just use the vSphere Client.
 
Note: This is not officially supported from VMware of course, use at your own risk.

UPDATE1: It looks like when Windows 8 64bit is booting up for installation, the virtual machine core dumps with the following error:

vcpu-0| MONITOR PANIC: vcpu-0:NOT_IMPLEMENTED vmcore/vmm/intr/apic.c:1804

Something similar occurs with Windows 8 32bit that gets past the panic but an error message is thrown on the screen regarding HAL initialization failure. Currently there are no workarounds and I've reached out to some of the folks at VMware to see if there's any tweaks that can be made to support this. As I mentioned earlier, this is an unsupported OS/hack, so it may not work at all. Sorry to get everyone's hope up, the new Fusion 4.01 and Workstation 8 might still be your best bet to test out the new Windows 8.

UPDATE2: VMware has released a KB article http://kb.vmware.com/kb/2006859 regarding Windows 8 and vSphere 5 support. You can subscribe to the KB article for the latest update on running Windows 8 on ESXi 5.

UPDATE3:  I recently saw a tweet by Raphael Schitz and it looks like you actually CAN run Windows 8 on ESXi 5. Raphael was able to run Windows 8 by first running Xenserver as a virtual machine and then creating a Windows 8 VM that would run as a nested guestOS within Xenserver virtual machine (pESXi 5 -> Xenserver VM -> Windows 8 VM).

Note: You may need to reboot the system one additional time if it does not automatically load.

Here is a screen shot of Windows 8 64bit running on the latest release of Xenserver 6 running on ESXi 5:

How to Use Custom VM Icons in the vSphere 5 Client

Ever get tired of the same old virtual machine icons in the vSphere Client? Ever thought about changing it? Well with vSphere 5, you can! Here is a screenshot of some the custom icons I added for several virtual machines in my development lab.

One of the major enhancement in the vSphere 5 API is the introduction of the vCenter Solutions Manager, vSphere ESX Agent Manager, and vServices SDK. These various interfaces allows ISVs, partners and end users to easily extend the functionality of the vCenter Server and provide solutions that are tightly integrated via extensions/plugins that are aware of features such as vSphere HA, DRS and DPM.

Here what each of the interfaces provides:

vCenter Solutions Manager - is a view in the vSphere client where you can monitor and interact with the solutions that register with a vCenter Server instance. Solutions Manager shows three standard tabs for each running solution. The tabs list the virtual machines that a solution deploys and manages, show the health, name, company URL, version of the solution, and show any vServices that the solution provides.

vSphere ESX Agent Manager - automates the process of deploying and managing vSphere ESX agents. The services that ESX Agent Manager provides include out-of-the-box integration of agents with vSphere features such as DRS, AddHost, High Availability, DRM, and maintenance mode. All of these features can be difficult to integrate with manually. ESX Agent Manager also allows users to monitor the health of ESX agents, and blocks users from performing certain operations on ESX agents that might affect the virtual machines that use them. For example, ESX Agent Manager can prevent an ESX agent virtual machine from being powered off or moved from an ESX host that contains other virtual machines that use that agent.

vServices SDK - is a service that a solution provides to specific applications that run inside virtual machines and vApps. A solution can provide several types of vServices. Virtual machines or vApps can have dependencies on several types of vServices. A vService is similar to a virtual hardware device upon which virtual machines and vApps can depend. Instead of providing a piece of virtual hardware, vServices typically provide access to a service across a network. By providing a vService, a solution can expose application-aware services to virtual machines and vApps. For example, a vService can provide a backup service or a logging service to virtual machines and vApps.

For more details about these interfaces and how to implement them, take a look at the documentation here.

Even though these APIs are really meant for ISVs and partners to consume, if you would like to add a splash of color to your environment, you can use the following trick. At at high level we will be adding a new extension(s) which includes a mapping of a particular solution (e.g. logical name) to a particular icon that a virtual machine or vApp can be associated with. The icons must be 16x16 PNG format referenced by a URL. Once the custom extension has been created, you will then need to reconfigure the virtual machine(s) and associate the managedBy property to the solution's logical name key to update the virtual machine's icon.

You will need access to either the vCLI or vMA and the following two vSphere SDK for Perl scripts: registerCustomSolution.pl and updateVMManagedBy.pl You will also need access to vCenter Server 5 as this is not supported on an ESXi 5 host.

There is a very simple example in registerCustomSolution.pl which creates two extensions: one that includes custom icons and tabs that can link to any webpage (vGhetto) and one that only includes custom icons (Custom Application). You will need to edit the script so it fits your environment and there is a special variable in the script called "editedScript" which is set to 0 that prevents the script from running. This will ensure you do not accidentally create these extensions based on information in my development environment. Once you have updated the script, go ahead and change the value to a 1.

When you are ready, you will use the registerCustomSolution.pl script to create the extensions, here is an example:

To verify that the extensions were created properly with the information you provide, you can use pluginExtensionManager.pl to list all registered extensions. The command is the following: ./pluginExtensionManager.pl --operation list

You should see at the bottom of the output the extensions that were just registered and the associated configurations URL + icons. It is important to make note of the extension key (e.g. com.vmware.vGhetto) and the solution type string as that will be needed in the next section.

Now all we need to do is associate a particular virtual machine with the solution to update the virtual machine's default icon. You will use the updateVMManagedBy.pl script and using the extension key and type from the output from the previous screen.

To verify the icons have been updated, you will need to login to the vCenter Server and check out your virtual machines.You will also notice that on the right hand side of the virtual machine summary screen, there is a new "Managed By" section which includes a link to the vCenter Solution Manager.

Note: If you would like to reset or revert back to the original icons, you just need to use the  updateVMManagedBy.pl script and specify a empty string for the key. If you would like to unregister and remove the extension all together, you can use pluginExtensionManager.pl and perform remove operation.

Another way to view all the vCenter Solutions is on the home page and by clicking on the vCenter Solutions Manager icon.

From here you will see all registered vCenter extensions including some information about the vendor, version and health of the extension.

Here is a drill down into one of the extensions that contains several tabs to some URLs

As you can see, you can link to some useful URLs that can easily be accessible through the vSphere Client without having to go to your browser. Another neat feature of the tabs is to include any web management interfaces for a particular solution/vApp so that you can easily configure and manage the system from a single pane of glass.

In addition to this, you can also get a summary of the registered virtual machines with a given solution by clicking on the "Virtual Machines" tab and selecting "Managed By" box, the "Server" and "Agents" are reserved for ESX Agents.

There are also two caveats to be aware of if you decide to create custom icons for your virtual machines. The first is when you edit a virtual machine, you will get an annoying pop-up that states changes to the solution is not recommended. Under normal circumstance, where a solution/extension is provided by a 3rd party, you definitely do not want to manually tweak the virtual machine but in this scenario, it is fine.

The second thing I noticed is the custom icons do not properly show up in the nextgen vSphere Web Client, a default icon is used instead for virtual machines who have custom icons. I am not sure if this is a display bug with the vSphere Web Client or with the APIs.

So there you have it, if you are bored at looking at the same old icons and would like to differentiate some or all of your virtual machines, you now have the option to use custom icons.

How to Automate the Deployment & Configuration of vShield Manager 5

If you have ever worked with VMware vShield Manager, you know that deployment and configuration of the virtual appliance is pretty much a manual process. You can automate the deployment of the vShield Manager OVA using the various vSphere SDK's or the ovftool, but the initial IP address configuration for vSM still needs to be configured manually using the remote console for the very first time.

An easy solution to this problem would be for VMware to create the vSM OVA to support IP address configuration out of the box as part of the deployment options (but why make things easy). In any case, I will demonstrate how you can easily automate both the deployment and the initial configuration of vShield Manager to your vSphere environment.

Before I begin, I can not take credit for coming up with the idea of automating vShield Manager deployment, the credit goes to Alan Renouf. Alan recently contacted me and ask if it was possible to automate the IP configuration. The answer is yes and here is a solution.

One of the main challenges in figuring out how to automate the IP address configuration of vShield Manager was due to the vtysh integrated shell daemon for Zebra that launches by default as part of the "admin" user account. This interface is used to manage the kernel routing and management table and made it very difficult to interface with for any type of automation. I decided manually go through a vSM configuration and then using Knoppix LIVE-CD, I was able to mount up the vSM filesystem and look around to get a better understanding of what was going on. After some investigating, it looks like IP address configuration is stored in /common/configs/cli/zebra.conf, here is an example of what the configuration looks like:

Armed with this knowledge, it was pretty straight forward in developing an automated way of deploying and configuring vShield Manager. I created a script called deployvShieldManager.sh which utilizes guestOpsManagement.pl, vCLI and ovftool. It's recommended that you use vMA and install ovftool to quickly get started. At a high level, the script is doing the following:

1. Deploy vShield Manager OVA using ovftool
2. PowerOn vSM and wait 2 minutes for VMware Tools to be ready on the system
3. Create new zebra.conf, backup the old zebra.conf and upload new zebra.conf using the new vSphere 5 VIX integration
4. PowerOff vSM to force the configurations to be read in upon next bootup
5. PowerOn vSM and it is now ready for use
   

At the top of the script, there are several configuration variables that need to be edited by the user to specify the vSM configuration, including vCenter and ESX(i) host to deploy vSM.

Here is a list of variables that need to be configured at the top of the deployvShieldManager.sh script:

Once you are done updating the variables, you are now ready to execute the script. Before the script performs any changes, it will first prompt the summary of configurations you have specified in the script. Once you are satisfiy, you may than proceed by typing "y" or "yes" to start, or if you would like to cancel, type "n" or "no".

Note: The script will perform some basic validation such as existence of the vShield Manager OVA, ovftool, etc. else you will get an error message and the script will exit.

Next, the script will perform the deployment of vSM using ovftool and proceed with the configuration of vSM once it has been deployed.

Note: If it takes longer to poweron vSM in your environment to get it into a ready state, you may want to tweak the sleep period from 120 seconds (2minutes) to something longer.

At this point, you now should see a new vShield Manager VM deployed and if you take a look at the summary page, you should see the new IP address and hostname configurations.

Now all that is left is to point your browser to the vSM address and you should be prompted to login to vShield Manager management interface.

Instead of manually deploying vShield Manager in your environment, you can now automate the initial deployment and configuration for general use or with VMware vCloud Director. For further automation and configuration of vShield manager, once vSM is online and accessible, you can leverage the vShield REST API.

Cool Undocumented Features in vCloud Director 1.5

While working on the updated script in Automating vCloud Director 1.5 & Oracle DB Installation, I did some digging in my lab deployment and noticed a few interesting things about the new vCloud Director 1.5 installation.

The first thing I noticed after configuring a new Provider vDC and the vCloud Agent (stored in /opt/vmware/vcloud-director/agent) is pushed out to the ESXi 5 hosts, a new esxcli module is added for vCloud Director under /usr/lib/vmware/esxcli-vcloud

There are 6 namespaces that ranges from simple configuration query, network fence management, account manage and also something called "esxvm" which I'll go into a little bit later. I am not sure why this is not in the vCloud Director documentation, I was not able to find any reference to the new esxcli operations. You may also notice the use of legacy "vslauser" (Virtual Software Lifecycle Automation) throughout vCloud Director, even though it was re-written from the ground up, it looks like VMware decided to either keep the name or some of the code related to the service account.

Here is an example of running "esxcli vcloud about get" command:

Here is an example of running "esxcli vcloud fence getfenceinfo" command:

Lastly, here is an example of what "esxvm" namespace provides:

As you can see above, there are two operations: disable/enable support for 64-bit nested virtual machines. This is exactly the same configuration as I blogged about in How to Enable Support for Nested 64bit & Hyper-V VMs in vSphere 5 but using esxcli interface with vCloud Director 1.5. Let's take a look at what happens when we run the "enable64bitnested" operation.

No surprise, we see that it automatically appends the required vhv.allow = "TRUE" flag which enables the support of running nested 64-bit virtual machines within a physical ESXi 5 host.

You might be asking, why is this in vCloud Director? Well if you attended VMworld 2011 or previous VMworlds and took part in the hands on labs, you will know that VMware utilizes vPods or nested ESXi to deploy their labs. I suspect, this functionality was added into vCloud Director so that VMware can easily leverage nested ESXi for hands on labs or vSel deployments just like they did with Lab Manager previously.

While look into this, I recall a very interesting article by Jason Boche - Deploy ESX & ESXi With Hidden Lab Manager 4 Switch in which Jason identifies a hidden flag in the Lab Manager database that enables a special feature in deploying nested ESX(i) VMs including customization through the use of a special version of VMware Tools for ESX(i). I was curious to see if something similar existed in the new vCloud Director that provided similar functionality.

Looking at the SQL install scripts located in /opt/vmware/vcloud-director/db/{oracle/mssql}, I noticed an interesting config called "extension.esxvm.enabled" in NewInstall_Data.sql file

As you can see from the insert statement, by default this value is set to "false" and we can also confirm this after vCloud Director has been installed and configured by querying the database. Let's go ahead and update this value to "true" and let's see what happens. 

Once you have verified the value has been successfully updated, I decide to use the same trick that Jason had identified with the special "Uber Admin Screen" to load the changes. To my surprise, the trick still worked but the page was not super Uber .... To enable the screen, you will need to click on the "About" page and then click CTRL+U (ctr + shift + u), which will toggle the "Uber Admin Screen".

The available options are quite limited as you can see but there are some new hidden options such as a new debug and console toggle. When you enable these options, you will see them at the bottom right of your screen including a counter of the amount of memory being used for your vCloud Director deployment.

After toggling the hidden database feature, I was not able to see any additional pages relating to nested ESXi hosts, even after restarting vCloud Director. Through some testing, I found that the "extension.esxvm.enabled" actually controlled whether or not nested 64bit VM was enabled when the vCloud Agent was pushed out to ESXi 5 hosts. Instead of manually adding vhv.allow = "TRUE" or using esxcli vcloud esxvm enable64bitnested, vCloud Director will automatically configure the ESXi hosts for you. I still suspect there is probably a hidden interface in managing vESXi hosts and leveraging a specialized version of VMware Tools to automate the deployment of nested ESXi, but I have not found out yet.

UPDATE: Take a look at this blog post for the full details on building your own vSEL - The Missing Piece In Creating Your Own Ghetto vSEL Cloud

Automating vCloud Director 1.5 & Oracle DB Installation

Here is an update to my vCloud Director 1.0 & Oracle Express Database installation script to include support for the new vCloud Director 1.5. There were a few slight modification I had to make as the new installer had an additional question around the type of database to use and there were also some path changes that had to be handled for the new version of vCloud Director. I also added support for vCloud Director 1.0.1 and have functionally tested all three version of vCloud Director with the use of a local Oracle Express Database.

It is important to note, the script is merely to help users quickly get vCloud Director setup for testing and evaluational purposes. For actual production/development deployment, I would recommend you go through the actual installer and using a remote database versus a local embedded. This script only deals with vCloud Director, you will still need to manually setup things like vShield or other products that you wish to use with vCloud Director.

One question I have gotten quite a bit is about setting up the initial vCloud Director Linux virtual machine prior to running the vCloud Director application installation script. The following outlines the steps in building out a CentOS virtual machine using network based installation.

You will need to download the following:

• CentOS 64bit Netinstall ISO
• Oracle Express 10g or 11g
• vCloud Director 1.5
• vcd_setup.sh
• vcd.rsp

Step 1 - Create a new 64bit virtual machine

• Select "Custom" Configuration
• Name your vCD Virtual Machine
• Select Datastore
• Select Virtual Machine Version 7 or 8 depending on version of vSphere
• Select "Linux" and "Red hat Enterprise Linux 5 (64-bit)" as Guest Operating System type
• Select the number of vCPU to be configured
• Select the amount of vMem to be configured
• Select 2 vNICs for the Virtual Machine and configure both on the same network
• Select the SCSI Controller (used defaults)
• Create a new Virtual Disk and configure the size

Step 2 - Power On the blank virtual machine

Step 3 - Attach the CentOS-5.5-x86_64-netinstall.iso to the vCD virtual machine and press CTRL+ALT+INSERT to reboot the system

Step 4 - When you see the "boot" prompt, just hit enter or wait for it load

Step 5 - Select language of choice

Step 6 - Select keyboard type:

Step 7 - Select installation type "FTP"

Step 8 - Select "eth0" which will be used for the initial connection to go out to the internet to pull in the CentOS image

Step 9 - Select "Manual Configuration" for eth0

Step 10 - Assign the appropriate IP Address for eth0

Step 11 - Select an FTP mirror from here to perform network based installation, in this example, I'm using one from UCSB where FTP Site name is "ftp.cs.ucsb.edu" and the CentOS Directory is "mirrors/centos/5.5/os/x86_64"

Step 12 - If successful, it should start retrieving the image

Step 13 - Click "Next" to start the installation

Step 14 - Select "Yes" to create new partition

Step 14 - Click "Next" to continue

Step 15 - Click "Yes" to clear all partitions

Step 16 - Verify network configuration for eth0, leave eth1 as uncheck, this will be configured per the vCloud Director Installation Script

Step 17 - Configure your timezone

Step 18 - Configure your root password

Step 19 - Select "Server" installation type and click "Next"

Step 20 - Click "Next" to start the installation and you can now sit back and relax

Step 21 - Once the system has completed, you will need to click on "Reboot"

Step 22 - SSH to the system and run "ifconfig eth0" and "ifconfig eth1" and eth0 should be configured since you are able to login into it and eth1 should not be configured with anything

Step 23 - You need to upload the vCloud Directory binary, Oracle Express RPM and the vCD setup script and response file

Step 24 - You will need to edit the vcd.rsp to setup you wish to deploy, here is an example of deploying vCloud Director 1.5 using Oracle 11g database

Step 25 - Start installation of vCloud Director + Oracle Express database using vcd_setup.sh script

Note: For more details on what is going on during the installation, please take a look here

Step 26 - Assuming you followed the directions, in about 10-15minutes you should get a successful message on the installation of vCloud Director and now be able to point your browser to your vCD system