Having Difficulties Enabling Nested ESXi in vSphere 5.1?

I noticed there were a few folks having some difficulties enabling Nested ESXi (VHV Virtual Hardware Virtualization) in the latest release of ESXi 5.1 and I thought I share some additional info and tips on troubleshooting your setup in case you are running into similar problems.

*** DISCLAIMER **** This is not officially supported by VMware, do not bother asking if it is supported or calling into VMware support for details or help.

If you wish to run nested ESXi or other hypervisors on ESXi 5.1 and run 32-bit nested virtual machines, you must meet the following hardware requirement:

• CPU supporting Intel VT-x or AMD-V

If you wish to run nested 64-bit virtual machines in your nested ESXi or other hypervisors, in addition to the requirement above, you must also meet the following hardware requirement:

• CPU supporting Intel EPT or AMD RVI

If you only meet the first criteria, you CAN still install nested ESXi or other hypervisors on ESXi 5.1, BUT you will only be able to run 32-bit nested virtual machines. When you create your virtual machine shell using the new vSphere Web Client, in the expanded CPU view, the "Hardware Virtualization" box will be grayed out. This is expected as you do not have full support for VHV, but you can still continue with your installation of ESXi or other hypervisors.

In ESXi 5.0, you may have been able to run 64-bit nested virtual machines without EPT/RVI support but performance was extremely poor. With ESXi 5.1, VHV now requires EPT/RVI.

Note: During the installation of ESXi, you may see the following message "No Hardware Virtualization Support", you can just ignore it. 

If you are using sites such as Intel's ark.intel.com to check your CPU requirements, be aware that it is COMMON even for the hardware vendors to publish incorrect information about their websites. However, there is a quick way you can validate on your ESXi host whether you have full VHV support.

In vSphere 5.1, there is a new capability property called nestedHVSupported which specifies whether your physical ESXi 5.1 host has full VHV support. This property will only be true IF your CPU has both Intel-VT+EPT or AMD-V+RVI. A quick and easy way to validate this is using the vSphere MOB to retrieve the value.

To check nestedHVSupported property, please enter the following into a web browser (substitute the IP Address/hostname of your ESXi host):

https://himalaya.primp-industries.com/mob/?moid=ha-host&doPath=capability

After you login, search for the nestedHVSupported property on the page and you should see a value of either true or false. As mentioned earlier, if it is false, you might still be able to install nested ESXi or other hypervisors but you will not be able to run nested 64-bit virtual machines. I would also recommend taking a look at your system BIOS to ensure things like Intel-VT/EPT and AMD-V/RVI are enabled and sometimes it might just be as simple as a BIOS upgrade (you can always confirm by contacting the hardware vendor if you have further questions).

For proper networking connectivity, also ensure that either your standard vSwitch or Distributed Virtual Switch has both promiscuous mode and forged transmit enabled either globally on the portgroup or distributed portgroup your nested ESXi hosts are connected to. 

Additional Resources: 

• How to Enable Nested ESXi & Other Hypervisors in vSphere 5.1 
• How to Enable Nested ESXi & Other Hypervisors in vCloud Director 5.1

Creating Custom VIBs For ESXi 5.0 & 5.1 with VIB Author Fling

VMware Labs just released a really cool new Fling called VIB Author which is a tool that allows you to easily create custom VIBs for your ESXi 5.x hosts. If you have tried to create custom ESXi firewall rules or add custom scripts to your ESXi host, you may have noticed they are not persisted after a system reboot and you had to play all sorts of games to get the files to persist. The VIB Author tool now solves that problem and you can even take your custom VIB and integrate them into an Auto Deploy Image Profile using Image Builder. Before you jump right in, be sure to read over the important note in the documentation before getting started.

So how does the VIB Author tool work?

You will need to provide two pieces of input: payload which is set of files you wish to include in your VIB and the descriptor.xml which contains the metadata for your files. From that, VIB Author can produce either a VIB and/or an offline bundle (can be used with Image Builder).

VIB Author is distributed only as an RPM and you will need to install the VIB Author tool on a 32-bit Linux system (sorry, no 64-bit support). In my home setup, I went with CentOS 6.2 i386 as it was free to download & easy to setup or you may choose go with SUSE Linux Enterprise 11 SP2 which is the recommended platform per the documentation.

To install the RPM, run the following command:

rpm -ivh vmware-esx-vib-author-5.0.0-0.0.844296.i386.rpm

In the example below, I will show you how to create a custom VIB that contains several different configurations:

• Custom Firewall Rule
• Custom Startup script (adds a static route)
• Custom Files (ghettoVCB)

Disclaimer: The example below is not officially supported by VMware, please thoroughly test this in a development environment before using in production.

Here is the directory structure for the example that we will be going through:

Step 1 - Create your stage directory structure which we will then populate with your payload files as well as the descriptor.xml file.

mkdir -p stage/payloads/payload1

Step 2 - Create your descriptor.xml file which should be placed in the stage directory. For more details on the parameters within the descriptor.xml, please take a look at the documentation.

Here is an example of my descriptor.xml file:

Step 3 - Create the directory structure and store the files you wish to include under payload1. Ensure the the directory structure matches the absolute path of how you want the files to appear on the ESXi host. For example, if you wish to create a file call foo in /etc/vmware/foo then your directory structure should look like stage/payloads/payload1/etc/vmware/foo

Note: In the documentation, there is a list of default supported paths, if you venture off of this supported list, then you will need to issue the -f flag when creating your VIB as well as installing your VIB on your ESXi host

So for our examples we have the following files:

stage/payloads/payload1/etc/vmware/firewall/virtuallyghetto.xml
This one should be pretty straight forward, we are just creating a custom ESXi firewall rule and you will need to place your configuration file under /etc/vmware/firewall, please take a look at this article for more details on creating your own firewall rules.

stage/payloads/payload1/etc/rc.local.d/999.addStaticRoute.sh
This is a custom shell script that adds a static route to an ESXi host upon bootup under /etc/rc.local.d. There maybe other startup scripts that could be executed and you do not want to conflict with any system defaults. I recommend you label yours with a high number such as 999 to ensure it is one of the last scripts to execute. 

stage/payloads/payload1/opt/ghettoVCB/{ghettoVCB.conf,ghettoCB-restore.sh,ghettoVCB.sh}
This is a custom set of files that I would like to store in ESXi under /opt directory and the files are my free ghettoVCB backup script.

Here is a copy of my directory structure (stage.zip) which can be used as a reference.

Step 4 - Now we ready to create our VIB and/or offline bundle by specifying our stage directory as input. In this example, we will generate both a VIB as well as an offline bundle containing the same contents. Run the following command:

vibauthor -C -t stage -v virtuallyghetto.vib -O virtuallyghetto-offline-bundle.zip -f

Note: Since we added some files outside of the default supported paths, we also need to specify the -f flag to force the creation.

We can also extract information about our VIB by using the -i option in VIB Author, to do so, run the following command:

vibauthor -i -v virtuallyghetto.vib

Finally, we are now ready to copy the VIB over to our ESXi host and install our custom VIB.

To install VIB run the following command:

esxcli software vib install -v /vmfs/volumes/[datastore-name]/virtuallyghetto.vib -f

To install the offline bundle run the following command:

esxcli software vib install -d /vmfs/volumes/[datastore-name]/virtuallyghetto-offline-bundle.zip -f

Note: You need to specify the -f flag to force the installation since we created files in an unsupported path. I have been able to test the VIB and offline bundle installation on both ESXi 5.0 as well as ESXi 5.1

To confirm we have succesfully installed our custom VIB, we can query it by running the following command:

esxcli software vib list | grep virtuallyghetto

So there you have it, in just a few steps, you can create your own custom VIBs!

Lossless OVF Export in vSphere 5.1 & vCloud Director 5.1

Did you know in vSphere 5.1, the new vSphere Web Client now provides a lossless OVF export feature for virtual machines? Previous to this, when you exported a virtual machine only the basic configurations were captured to ensure the virtual machine could be re-imported into another environment that supports OVF. The virtual machine's "virtual hardware personality" such as the BIOS UUID, MAC addresses, VMware specific advanced settings (Extra Configurations), boot order, PCI slot numbers, etc. were not captured as part of the export.

One of the main reasons for doing this is to ensure greater portability of the virtual machine across different environments. By having some of these specific properties tied to the virtual machine, it could limit where a virtual machine could be imported to. However, if you wish to preserve some of these settings during an export, you now have a new advanced option in vSphere 5.1 to specify additional configurations to capture for an OVF export.

When you select the "Enable advanced options" box, you will be provided with a list of configurations that you can preserve and export with the virtual machine.

Note: As the warning message states, only enable the configurations that you absolutely need, else you should stick to the defaults to ensure you have the best portability for your virtual machine export.

There is also a similar feature in vCloud Director 5.1 for vAppTemplates. When you initiate a download of a vAppTemplate, you now have an additional check box to "Preserve identity information" of the vAppTemplate prior to export. This is just a single check box for vCloud Director, you can not specify which configurations you want to preserve.

All these new OVF features can also be accessed with the latest ovftool 3.0.1 release which includes several new enhancements as well as support for both vSphere 5.1 and vCloud Director 5.1. You can find more details in the ovftool 3.0.1 user guide, but here is a summary of what's new in this release.

What Is New in OVF Tool 3.0.1

• Support for vSphere 5.1 and 5.0.  
• Support for vCloud Director 5.1, 1.5, and partial support for vCloud Director 1.0
•    The following new ovftool options: --annotation, --exportFlags, --shaAlgorithm, --sourcePEM, --targetPEM, --vCloudTemplate, --sourceSSLThumbprint, --targetSSLThumbprint, --fencedMode, --noSSLVerify, --vService.

Feature Highlights

• Full support for vCloud Director 1.5 and 5.1
• Includes full OVF 1.0, and 1.1 support and backward-compatible mode for importing existing OVF 0.9 packages
• Supports both import and generation of OVA packages (OVA is part of the OVF standard, and contains all the files of a virtual machine or vApp in a single file.)
• Directly converts between any vSphere, vCloud Director, VMX, or OVF source format to any vSphere, vCloud Director, VMX, or OVF target format
• Accesses OVF sources using HTTP, HTTPS, or FTP, or from a local file 
• Deploys and exports vApp configurations on vSphere 4.0 targets and later and on vCloud Director 1.5 targets and later
• Provides options to power on a VM or vApp after deployment, and to power off a virtual machine or vApp before exporting (caution advised)
• Show information about the content of any source in probe mode
• Provides context sensitive error messages for vSphere and vCloud Director sources andtargets, showing possible completions for common errors, such as an incomplete vCenter inventory path or missing datastore and network mappings
• Provides an optional output format to support scripting when another program calls OVFTool
• Uses new optimized upload and download API (optimized for vSphere 4.0 and later)
• Signs OVF packages and validates OVF package signatures
• Validates XML Schema of OVF 1.0 and OVF 1.1 descriptors
• Import and export of OVF packages into a vApprun1.0 workspace. For more information about vApprun, see http://labs.vmware.com/flings/vapprun.

2gbsparse Disk Format No Longer Working On ESXi 5.1

I was recently made aware of an issue with my ghettoVCB script that after upgrading to ESXi 5.1, the ability to clone (or in this case backup) using the 2gbsparse disk format with vmkfstools was no longer working. The error that users were seeing was "The system cannot find the file specified." and I also confirmed this behavior by manually creating a VMDK and then trying to clone using the 2gbsparse format.

To give you some background, the 2gbsparse disk format is not a VMFS virtual disk format, it is part of the hosted desktop product (VMware Fusion, Workstation, Server & Player) disk format. This disk format was created to prevent cross-platform file system compatibility issue as pointed out in this VMware KB article. This issue does not exists on VMFS and hence this extra disk format is not necessary.

After some investigation, I found to use the 2gbsparse format in vmkfstools, you will need to load a specific VMkernel module called "multiextent". 2gbsparse was never officially supported on ESXi, you can not run a virtual machine with 2gbsparse disk format on ESXi and that is why a conversion maybe required when moving from a hosted product to ESXi. So by disabling unnecessary VMkernel modules that were not used made sense to help reduce amount of resources needed to load up. This is especially important with stateless deployments, where you want your ESXi host to load up as fast as possible.

Once you have enabled this VMkernel module, the 2gbsparse format will function again with vmkfstools. I also found that this was mentioned in the vSphere 5.1 release notes (yes, you should read the release notes)

To load the multiextent VMkernel module, run the following ESXCLI command:

esxcli system module load -m multiextent

To check whether the multiextent VMkernel module has loaded, run the following ESXCLI command:

esxcli system module list | grep multiextent

If you wish to persist this configuration after a system reboot, I found that you need to add the following command in a start-up script /etc/rc.local.d/local.sh as just setting the "enabled" flag is not sufficient for this particular VMkernel module.

localcli system module load -m multiextent

Note: We are using localcli because hostd may not be completely ready and you can either add a sleep/timer or just use localcli.

Quickly Configuring the New vCenter Orchestrator 5.1 Appliance

Have you checked out the latest release of vCenter Orchestrator 5.1? If not, you should really consider taking a look as there are number of significant enhancements in this release. Deploying the vCO 5.1 virtual appliance is a quick and easy way of getting started and that is what I used to test drive the new vCO 5.1. One thing I noticed after deploying the appliance is that vCO now requires a few more steps for configuration and that means more clicking and longer wait before I can get started!

As you probably may have guessed, I had to look for a better way of setting up vCO so that I can get started faster! As far as I knew, there were no configuration utilities or APIs that I could leverage and all configuration changes had to be done through the vCO configuration web interface. After clicking around for a bit, I realized I could "ghetto" it by just sending in the raw HTTP requests the UI was performing. I ended up writing a very simple shell script and using Firebug, a very handy tool that I used from time to time to help me figure out what each request looked like.

Disclaimer: This is for educational purposes only, this is not officially supported by VMware. Please test this in a development environment before using it on actual systems.

You can download the script here called configureVCO51.sh

To use the script, you just need a machine with curl, you do not need to copy the script to your vCO server. You will need to edit the following variables in the script before executing:

VCO_IP_ADDRESS=172.30.0.203
VCO_DEFAULT_USERNAME=vmware
VCO_DEFAULT_PASSWORD=vmware
VCO_NEW_PASSWORD=vmware123

VCENTER_IP_ADDRESS=172.30.0.181
SSO_IP_ADDRESS=172.30.0.181
VCENTER_USERNAME=root
VCENTER_PASSWORD=vmware

The variables should be pretty self explanatory, you will need to provide the IP Address of your vCO Server and the default username/password for the vCO 5.1 appliance is vmware/vmware. You will need to specify the new password that you wish to set and then provide information about your vCenter Server as well as your vCenter SSO Server. In this example, I used the VCSA.

Note: The script currently assumes you will be using System-Domain\__Administrators__ as the admin group, if you decided to user another group, you will need to edit the confirmSSOServer section of the script. I also used the VCSA

Here is an example execution of the script:

The numbers on the left hand side of each task is the HTTP return code and as you can see, the script not only setups the vCO appliance but also enables the vCenter Server vCO plugin as well as adding in your vCenter Server 5.1 and then automatically restarting the vCO service for all changes to take effect. I timed my script versus a manual configuration (assuming you know exactly what to do), I was still able to get everything configured in less than 1.5minutes where as the manual clicking and waiting took almost 5minutes (remember, I knew exactly what to click on which means it could take even longer!).

At the end of the script, you should be presented with two URLs: one that takes you straight to the vCO configuration web interface in case you wanted to check or verify the settings and the other is the vSphere Web Client URL for your vCenter Server.

Here is a screenshot of the vCO configuration interface, we can see that everything is showing green for it's status:

Here is a screenshot of the vSphere Web Client and we can see that our vCO Server has been configured and is now accessible to us within vCenter Server.

This script can easily be modified to perform other operations within vCO (e.g. enabling other plugins/configurations), this was just a sample of what you could do. To add or modify other functionality, you just need to use Firebug and watch for the specific HTTP request that is made while you are manually performing the task in the vCO configuration interface. For more details on setting up the new vCO 5.1 appliance, I highly recommend you check out Christophe Decanni's vCenter Server & vCenter Orchestrator 5.1 integration tips doc.

Additional Resources: 

• Unattended Deployment of vCenter Orchestrator Virtual Appliance

Configuring Additional VCSA 5.1 as vSphere Web Client Servers

In my previous article, I showed you how to configure additional vSphere Web Client 5.1 Servers for a Windows environment. For those interested in using the VCSA (vCenter Server Appliance) instead to configure additional vSphere Web Client Servers, here is the process.

Disclaimer: This is for educational purposes only, this is not officially supported by VMware. Please test this in a development environment before using it on actual systems.

Before getting started, it is very important to ensure you have proper DNS resolution (forward / reverse working) on all your VCSA's.

Step 1 - You will need to know the IP Address or hostname of your vCenter SSO Server (If you are using the VCSA, then it is the same address as your vCenter Server). Next, deploy a brand new VCSA and go ahead and configure the IP Address, hostname, etc. in preparation for the next step.

Step 2 - Create a script called configureVCSAvSphereWebClientStandalone.sh (this script reside on the new VCSA itself or on a remote host) which contains the following
You will need to change the VCENTER_SSO_IPADDRESS variable to the address of your vCenter SSO Server.

Step 3 - Set the script to executable by running the following command:

chmod +x configureVCSAvSphereWebClientStandalone.sh

Step 4 - You can either execute the script locally on the VCSA or my preferred method is just to execute it remotely by running the following command:

ssh root@172.30.0.233 < configureVCSAvSphereWebClientStandalone.sh

Note: Make sure to replace the IP Address with address of your VCSA

As part of the configuration, the vSphere Web Client service will automatically start up at the very end. Once the service has successfully started, you will be able to connect to the vSphere Web Client URL by going to https://[webclient-hostname-ip]:9443/vsphere-client and you now should be able to see all the vCenter Servers that have registered with the same vCenter SSO server.

Here is another view using the new VIN 1.2 (vSphere Infrastructure Navigator) to show that I have two vSphere Web Client servers (webclient1 and webclient2) registered to my primary vCenter SSO Server (vcenter51-1) as well as two additional vCenter Server (vcenter51-2 and vcenter51-3) that are also registered with the same SSO Server.

As you can see, if you need to add additional vSphere Web Client Servers, the VCSA can easily be configured to enable only this feature to help you scale out and provide additional capacity as well as redundancy. Stay tuned for some more details on configuring a Load Balancer to load balance multiple vSphere Web Client Servers.

Additional Resources: 

• Automating VCSA 5.1 (vCenter Server Appliance) Configurations 
• Automatically Join Multiple VCSA 5.1 using New vCenter SSO (Single Sign-On) 
• Configuring Additional Windows vSphere Web Client 5.1 Servers

Default Password for vCenter SSO Admin Account on VCSA

I thought I share this quick tidbit about the VCSA (vCenter Server Appliance) default password for the vCenter SSO Administrator account as I was just asked about it today and this was something I had research just earlier in the week. In the Windows version of vCenter SSO installation, users are prompted during the install to select a password for this account, you might have seen it show up as admin@System-Domain. For the VCSA, vCenter SSO is already installed and you might be wondering what the default password is?

Well, the answer is ... there is no default password. During the installation process, there is a random password that is generated and once the installation is complete, the password is then immediately removed. This is a good thing from a security perspective, by not having a default password set. This account is not only a vCenter SSO Administrator but it also the only account that has access to the internal RSA IMS system. You should definitely go in and set a password for this account after setting up your VCSA which can only be done through the vSphere Web Client.

Here are the steps:

1. Click on the Administration tab on the left hand side of the vSphere Web Client navigation bar.

2. Next click on "SSO Users ad Groups" and you should see the admin user account.

3. Lastly, you just need to right click and edit the user or select the pencil icon and set a password for the admin user account. Be sure to use a strong password, as there is a password validation before the system accepts the change.

Big thanks goes out to Michael Haines for helping me track down this answer about the default (or not so default) password for the admin account on the VCSA.

Configuring Additional Windows vSphere Web Client 5.1 Servers

Did you know in vSphere 5.1, you can now configure multiple vSphere Web Client servers to multiple vCenter 5.1 Servers and get a single view of your entire infrastructure as well as provide redundancy to the vSphere Web Client? This is all made possible with the new vCenter SSO (Single Sign-On) and Lookup Service feature.

When you first install vCenter Server, you can install the vSphere Web Client on the same machine or you can install it on a separate machine. If you decide to install the vSphere Web Client on a separate system or add additional vSphere Web Client servers, you simply just need to point them to your main vCenter SSO instance.

Note: If you wish to re-point or re-register other vCenter Server components, there is nice VMware KB that goes over all the steps. 

Before getting started, it is very important to ensure you have proper DNS resolution (forward / reverse working) on all your Windows servers. 

If you are running Windows, when you install the vSphere Web Client, you will be brought to a screen to specify your SSO Administrator credentials (admin@System-Domain for windows) or (root for VCSA) as well as the Lookup Service URL which will be the same system running your vCenter SSO service.

Note: In my lab, I am using the VCSA for my vCenter Server as well as my SSO server, but this can also be a Windows vCenter Server and SSO Server.

You also have the ability to re-point your vSphere Web Client to another vCenter SSO server and you can do so by using the following script: C:\Program Files\VMware\Infrastructure\vSphereWebClient\scripts\client-repoint.bat To use the script, you just need to specify the new Lookup Service URL along with the SSO Administrator credentials.

C:\Program Files\VMware\Infrastructure\vSphereWebClient\scripts\client-repoint.bat https://172.30.0.181:7444/lookupservice/sdk root vmware

Once the script has finished registering with the new vCenter SSO server, you can now connect to the vSphere Web Client URL by going to https://[webclient-hostname-ip]:9443/vsphere-client and you now should be able to see all the vCenter Servers that have registered with the same vCenter SSO server.

Here is another view using the new VIN 1.2 (vSphere Infrastructure Navigator) to show that I have two vSphere Web Client servers (webclient 3 and webclient4) registered to my primary vCenter SSO Server (vcenter51-1) as well as two additional vCenter Server (vcenter51-2 and vcenter51-3) that are also registered with the same SSO Server.

As you can see this provides me with single view of all my vCenter Servers and I can now connect to either vSphere Web Client servers which can be used for both load balancing as well as redundancy. You probably might have guessed, the next logical step is to put an actual load balancer in front of multiple vSphere Web Clients and simply expose a single entry point for your end users .... stay tune :)

FYI - If you are interested in using the VCSA and only enabling the vSphere Web Client feature, the process is slightly different and I will share the procedure in a separate post.

A Pretty Cool Method of Upgrading to ESXi 5.1

I recently came across an interesting article by Andreas Peetz which shows you how to patch an ESXi host using an image profile that is directly available on VMware's online depot within the ESXi shell. I knew that VMware had online depots for use with VUM and Auto Deploy but I was not aware of this particular method, especially directly from the host.

Disclaimer: This method assumes you can install the default ESXi Image Profile with no additional drivers or packages, else you may have connectivity issue after the upgrade. If you still need to customize the ESXi Image Profile before installation, you will still need to use something like Image Builder and then upload that to your online depot.

Note: There are many ways that you can patch/upgrade your ESXi hosts, here is another article that provides more details for command-line only methods.

Before you get started, you will need to make sure that your ESXi host has the httpClient firewall rule enabled, else you will not be able to connect to VMware's online depot. To enable this, run the following ESXCLI command:

esxcli network firewall ruleset set -e true -r httpClient

Also make sure that your ESXi host can reach the following URL (you can specify a proxy if needed):

https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml

To view the available ESXi Image Profiles, run the following ESXCLI command (use the --proxy if you need to specify a proxy to reach VMware's online depot):

esxcli software sources profile list -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml

If you are able to successfully connect to the online depot, you see a list of all the ESXi Image Profiles that are available to you. You will see two ESXi 5.1 Image Profiles (these were recently published), one with VMware Tools and one without VMware Tools.

Note: Before you begin, make sure you do not have any running VMs and put your host into maintenance mode.

Let's go ahead and upgrade our ESXi 5.0 Update 1 host to latest ESXi 5.1. To install the new Image Profile, run the following command:

esxcli software profile update -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml -p ESXi-5.1.0-799733-standard

This can take a few minutes to complete depending on how fast you can pull down the Image Profile. Once it is done, you will see all the new VIBs that have been updated and you will be asked to reboot for the changes to go into effect and then you will be running ESXi 5.1! Pretty cool IMO!

Automating ESXi 5.1 Kickstart Tips & Tricks

There is not a whole lot of changes for kickstart configurations between ESXi 5.1 and ESXi 5.0, majority of the tips and tricks noted in the ESXi 5.0 kickstart guide are still relevant for ESXi 5.1. Below are a few new tips and tricks (some old) as well as a complete working ESXi 5.1 kickstart example that can be used as a reference.

Tip #1

There are 82 new ESXCLI commands, number of which are new as well as enhancements to existing commands and operations. The kickstart sample below converts many of the legacy esxcfg-* and vim-cmd/vsish commands over to ESXCLI such as, here are just a few:

• esxcli network ip route [ipv4|ipv6] (VMkernel routes)
• esxcli system snmp (SNMP)
• esxcli system maintenanceMode (maintenance mode)
• esxcli network ip interface tag (tag VMkernel traffic types)

Please refer to the vCLI/ESXCLI release notes for all new ESXCLI commands.

Tip #2

In previous releases of ESXi, you could add custom commands in /etc/rc.local which will automatically execute after all startup scripts have finished. With the latest release of ESXi 5.1, this functionality has been moved to /etc/rc.local.d/local.sh. If you try to edit the old file, you will find that it does not allow you to write any changes. This will be important as you migrate to ESXi 5.1 kickstart if you make use of this file for any custom startup commands.

Tip #3

To run nested ESXi and other hypervisors in ESXi 5.1, you need to to specify new vhv.enable parameter, please take a look at this article for more details. 

Tip #4

There is a new ESXi Advanced Setting in ESXi 5.1 that allows you to control when an interactive ESXi Shell session will automatically logout based on configured idle time (in seconds). You can find more details in this blog article by Kyle Gleed.

esxcli system settings advanced set -o /UserVars/ESXiShellInteractiveTimeOut -i 3600

Tip #5

By default, an ESXi host will automatically grant root permission to the "ESX Admins" group for use when a host is joined to an Active Directory domain. You can alter the default group name if you already have an AD group defined by using the following command:

vim-cmd hostsvc/advopt/update Config.HostAgent.plugins.hostsvc.esxAdminsGroup string "Ghetto ESXi Admins"

Tip #6

A really neat feature in ESXi 5.1 is the ability to control which local users have full admin privileges to the DCUI, this is really useful for troubleshooting and you want to provide DCUI console access but not administrative permissions on the ESXi host itself. You can specify a list of local users by using the following command:

vim-cmd hostsvc/advopt/update DCUI.Access string root,william,tuan

Tip #7

If you wish to prevent VMs from sending out BPDU (Bridge Protocol Data Unit) packets, there is a new global configuration on an ESXi 5.1 host which you can set. By default, this setting is disabled and you will need to configure this on every ESXi host if you wish to block VM guests from sending out BPDU packets.

esxcli system settings advanced set -o /Net/BlockGuestBPDU -i 1

Tip #8

Here's an article about enabling/disabling IPv6 using ESXCLI

Tip #9

Here's an article about creating custom VIB for ESXi 5.1



Here is a complete working example of an ESXi 5.1 kickstart that can help you convert your existing ESX(i) 4.x/5.x to ESXi 5.1:

Disabling IPv6 via Command-Line For ESXi 5.1 (Without Automatic Host Reboot)

IPv6 for the VMkernel interface is now automatically enabled by default for the latest release of ESXi 5.1 and you may have also noticed the additional IP Address in DCUI after the host boots up.

IPv6 support has been around for awhile now and you can enable IPv6 by using the old vSphere C# Client or the new vSphere Web Client. If you enable or disable IPv6, you will need to perform a system reboot for the changes to go into effect. You also have the ability to enable/disable it via the DCUI, which also has been around for awhile as well.

There is one very important thing to note if you do enable/disable IPv6 on the DCUI, after you made your changes and you wish to apply, there is a very important confirmation box that is displayed.

Carefully read the last sentence which is underline in red "In case IPv6 has been enabled or disabled this will restart your host". If you are not careful in reading the confirmation screen, you may hit yes and your host will issue a reboot. If you are going to use the DCUI to enable or disable IPv6, make sure you do not have any running VMs on your host and you should already have your host maintenance mode when making configuration changes to your host.

In addition to the two methods listed above (vSphere Web Client/C# CLient and DCUI) you can easily enable/disable IPv6 using ESXCLI (my preferred method) and restart the ESXi host when you get a chance.

To view whether IPv6 is currently enabled, run the following ESXCLI command:

esxcli system module parameters list -m tcpip3

As you can see from the screenshot above, ipv6 property is set to 1 which means it is enabled.

To disable IPv6, you just need to set the property to 0, run the following ESXCLI command:

esxcli system module parameters set -m tcpip3 -p ipv6=0

We can now reconfirm by re-running our list operation to ensure the changes were made successfully. All that is left is to perform a system reboot, you can either type in "reboot" or use the new ESXCLI 5.1 command:  

esxcli system shutdown reboot -d 60 -r "making IPv6 config changes"

Note: You can run the ESXCLI command locally on the ESXi Shell or you can run the same command remotely by specifying additional connection options & proxy through vCenter Server if you wish. Take a look here for additional connection options for ESXCLI. 

vCloud Director 5.1 Introduces "Open in vSphere Web Client" & Task Correlation ... Awesomeness!

Have you ever had to troubleshoot something in vCloud Director such as looking into a particular vApp and then realizing you needed to jump over to the vSphere environment to continue troubleshooting? This meant you had to perform a separate login to your vSphere environment and most likely using a different set of credentials and then find the set of Virtual Machines that made up that vCloud Director vApp ... painful and annoying right? Wish you could just right click and jump straight to that vApp construct in vSphere?

Well, now you can with the latest release of vCloud Director 5.1 which introduces a new option called "Open in vSphere Web Client".

In the screenshot above, I have a vCloud Director vApp which contains 3 Virtual Machines and I want to jump straight to that object in the vSphere Web Client. To do so, simply right click on the object and select "Open in vSphere Web Client".

This will open up a new browser to the vSphere Web Client and locate the object in the appropriate vCenter Server and automatically log you in using the new vCenter SSO. Pretty cool huh!? This new option is not just limited to vCloud Director vApps and VMs, but applies to any vSphere object found within vCloud Director.

Note: You will need to be a System Administrator in vCloud Director to perform this operation as you will need access to the vSphere infrastructure.

Another challenge that you might have faced while troubleshooting in vCloud Director is to be able to correlate the tasks generated from vCloud Director and map those back to vSphere tasks in vCenter Server. This is not a trivial thing to do and required you to jump between vCloud Director and vCenter Server manually correlating the tasks. In vCloud Director 5.1, this has been enhanced by providing users with a single view for a given vCloud Director task and all the associated vSphere tasks that were created.

When you open a given vCloud Director task, you will be defaulted to the vCloud Director task details as you would expect. In the same view, there is also a new tab called vSphere Tasks which contains all the vCenter Server tasks generated from that given vCloud Director task.

As you can see from these two enhancements, you will be able to quickly and easily view all tasks that have taken place between vCloud Director and vCenter Server, all from a single location. If you need to drill down further into the vSphere layer, it is simply a right click and you will be taken directly to the vSphere object in the vSphere Web Client without having to provide additional credentials.

ESXi 5.1 Officially Certified on Apple Mac Pro

If you are still trying to catch up with all the awesomeness that has been shared, blogged, tweeted, etc. about in the last 24hrs with the announcement of the new vCloud Suite 5.1 release, here is one additional tidbit to add to that list which you might not have heard about it. With the release of ESXi 5.1, the Apple Mac Pro is now officially supported!

I came across this little tidbit from some folks internally who have been working hard to get the Mac Pro platform certified and supported on ESXi. After the vCloud Suite went live, I did a quick search on the VMware HCL and sure enough, the Apple Mac Pro was listed with support for ESXi 5.1.

If you recall back when vSphere 5.0 was released, VMware introduced support for ESXi to run on Apple's XServe ... unfortunately, Apple had decided to announce it's EOL (End of Life) several months prior which made it difficult to purchase and rely on the XServe as a platform. The only way you could get your hands on an XServe is if you were one of the few who purchased them before the EOL date (I personally went through this) or you were willing to look on Ebay. Luckily, now with the latest release of ESXi 5.1, if you need to virtualize Mac OSX Server Virtual Machines and other guestOSes, you can do so using the Apple Mac Pro.

Nested ESXi 5.1 Supports VMXNET3 Network Adapter Type

I noticed something interesting while extracting the contents of ESXi 5.1 ISO for some kickstart configurations ....

Do you see it? It's a VMXNET3 driver for the VMkernel! I also confirmed by running the following ESXCLI command querying for the VMkernel module "vmxnet3":

~ # esxcli system module get -m vmxnet3
   Module: vmxnet3
   Module File: /usr/lib/vmware/vmkmod/vmxnet3
   License: GPL
   Version: Version 1.1.32.0, Build: 799733, Interface: 9.2 Built on: Aug  1 2012
   Signed Status:
   Signature Issuer:
   Signature Digest:
   Signature FingerPrint:
   Provided Namespaces:
   Required Namespaces: com.vmware.driverAPI@9.2.1.0, com.vmware.vmkapi@v2_1_0_0

***Disclaimer***: This is for educational purposes only, this is not officially supported by VMware. Use at your own risk. There is also a mention of this in the vSphere 5.1 release notes that VMs running on nested ESXi hosts using VMXNET3 driver could potentially crash. Again, not supported user at your own risk.

Next I decided to create a Nested ESXi 5.1 VM, but instead of selecting the e1000 driver which was the only network adapter type that would function for running a nested ESXi host, I choose the VMXNET3 adapter and to my surprise ESXi's networking stack was fully functional.

You can see from the above screenshot, I have a two VMXNET3 network adapters for my nested ESXi 5.1 VM. Here are two additional screenshot of the physical adapters as seen by nested ESXi 5.1 host and you can see that it shows up as VMware Inc. VMXNET3

I have not tried any performance tests, so not sure if there are going to be any significant benefits but pretty cool nonetheless!

Configuring New vSphere Web Client Session Timeout

Just like you could in the old vSphere C# Client, users can also configure a session timeout for the new vSphere Web Client in the latest release of vSphere 5.1. This not only ensures that idle sessions automatically disconnect after a certain period of time but also helps reduce the resources consumed on the vCenter Server as each session allocates a certain amount of resources.

To configure the session timeout, you will need to login to the server running the vSphere Web Client service (which is usually your vCenter Server) and find the webclient.properties file and change the default timeout and then restart the vSphere Web Client service. For the VCSA, the default timeout value is 120 minutes and I assume it is the same default for the Windows vCenter Server.

Step 1 - Locate the webclient.properties file:

    VCSA 

/var/lib/vmware/vsphere-client/webclient.properties

    Windows vCenter Server

%ALLUSERPROFILE%\VMware\vSphere Web Client\webclient.properties

Step 2 - Un-comment and change session.timeout value to desired value:

session.timeout = 120

Step 3 - Restart the vSphere Web Client Service:

 /etc/init.d/vsphere-client restart

You will need to restart the vSphere Web Client service for the changes to go into effect. For the Windows vCenter Server, just restart the vSphere Web Client service and for the VCSA, run the above command.

In my lab, I configured the time out to be 1 minute, once the session has gone idle for the configured period, you will automatically be logged out and brought back to login page with the following message:

How To Initiate a Wipe & Shrink Operation On an SE Sparse Based Disk

In my previous two articles, I showed you how to create your own SE Sparse disks as well as creating new virtual machine Linked Clones leveraging the new SE Sparse disk format. If you recall earlier, one of the features of the SE Sparse disk format is to provide the ability to reclaim unused blocks within the guestOS which is a two step process: wipe and shrink.

Here is a screenshot that describes the process which was taken from the What's New In vSphere 5.1 Storage Whitepaper by my colleague Cormac Hogan. I highly recommend you check out the whitepaper which includes more details about this feature and other storage improvements in vSphere 5.1

The process of kicking off this wipe and shrink operation will be done through an integration with VMware View (a future release from my understanding). Now, it's important to understand that it's not just simply calling these two operations but it is also when they are called. The wipe operation is more CPU intensive as it scans for unused space within the guestOS filesystem and the shrink operation is more I/O intensive as it issues the SCSI unmaps commands. I can only assume that these operations will be scheduled based on the utilization of the guestOS to help reduce the impact to the VM workload.

Now having said that, since the SE Sparse disk format is a feature of the vSphere 5.1 platform, so are both the wipe and shrink operations. Though they are not exposed in the public vSphere API like the SE Sparse disk format, you can still access the private APIs if you know where to look ;)

****** Disclaimer ******: This is for educational purposes only, this is not officially supported by VMware. Use at your own risk.

With some help from my good friend the vSphere MOB and some digging, I have located the two vSphere API methods for wipe and shrink operation. Before getting started, ensure you have a VM with at least one SE Sparse disk, else these commands will not be very useful.

Note: In this experiment, I tested the wipe and shrink operation with Windows XP image, this may or may not work on other OSes.

First you will need to search for the VM in question and retrieve it's vSphere MOB URL which is in the format of https://[vcenter-server]/mob/?moid=vm-X where X is the MoRef ID for your VM. You can either navigate through the vSphere MOB or use my MoRef finder script.

Wipe Operation

To issue the wipe API, enter the following URL into your web browser (remember to replace the MoRef ID with the one of your VM)

https://[vcenter-server]/mob/?moid=vm-X&method=wipeDisk

Here is a screenshot of what that looks like if you are able to successfully access the private API:

Go ahead and click on "Invoke Method" which will then execute the wipe operation. If you take a look at the vSphere Web Client, you should now see a new task for the wipe operation.

This can take a bit of time as it scans through the guestOS filesystem for unused space.

Shrink Operation

Once the wipe operation as completed, we then need to issue the shrink API. Enter the following URL into your web browser (remember to replace the MoRef ID with tone of your VM)

https://[vcenter-server]/mob/?moid=vm-X&method=shrinkDisk

Here is a screenshot of what that looks like if you are able to successfully access the private API:

Here you can specify particular disks (requires diskId) that you wish to perform the shrink operation on. If you leave it blank, it will try to shrink all disks associated with your VM. In our example, I will shrink all disks. Go ahead and click on the "Invoke Method" which will kick off the shrink operation. If you go back to the vSphere Web Client, you should now see a shrink task in progress.

Again, this operation can also take some time, but once it has finished, then you have successfully reclaimed any unused blocks within your guestOS.

vInception #NotSupported Slides Posted

I was pinged by a few folks asking if my #NotSupported session that I presented at VMworld US would be available online, so here is the slide deck to my vInception presentation.


I would also like to thank everyone that attended my session! I had a lot of fun and hopefully you did too!  

UPDATE: I just realized the livestream recording videos are online, but they are not very clear. Apologies for that. I heard the better records from the vBrownbag crew should be up shortly, so once those are up, I will replace them on the site.

Part 1:

Watch live streaming video from vmwarecommunitytv at livestream.com

Part 2:

vmwarecommunitytv on livestream.com. Broadcast Live Free

Creating SE Sparse Linked Clones Using the vSphere 5.1 API

In my previous article, I showed you how you can easily create your own SE Sparse (Space-Efficient) disks for you virtual machines running on vSphere 5.1. In addition to just creating new virtual disks using the new disk format, you can also create SE Sparse Linked Clones by leveraging the vSphere 5.1 API. I had previously shared a vSphere SDK for Perl sample script called vGhettoLinkedClone that allows you to create your own Linked Clones and I have updated the script to now support the new SE Sparse disk format.

The script now includes new parameter called --convert which allows you to specify whether or not the destination Linked Clone should follow the same source virtual disk format or the new SE Sparse disk format. Remember to upgrade your VMs to the latest ESXi 5.1 compatibility and later (virtual hardware version 9) if you wish to leverage the new SE Sparse disk format. The way this is accomplished is by specifying a new disk backing  VirtualDiskFlatVer2BackingInfo in the VirtualMachineRelocateSpec and passing in the deltaDiskFormat to be of type seSparseFormat.

Disclaimer: This is for educational purposes only, this is not officially supported by VMware. Please test this in a development environment before using it on actual systems.

Note: Linked Clones is a feature of vCenter Server, make sure you are connecting to a vCenter Server 5.1

If you have a VM that has a VMDK that is NOT an SE Sparse disk format but wish to create Linked Clones with the new SE Sparse disk format, then you can use the --convert sesparse option, here is an example command:

./vGhettoLinkedClone.pl --server vcenter51-1.primp-industries.com --username root --vmname Regular-VM --vmname_destination Linkedclone-SeSparse-Destination --snapname pristine --vmhost vesxi51-1.primp-industries.com --datastore datastore1 --convert sesparse

If you have a VM that has a VMDK that IS an SE Sparse disk format and you wish to create Linked Clones with the same disk format, then you can use the --convert source option, here is an example command:

./vGhettoLinkedClone.pl --server vcenter51-1.primp-industries.com --username root --vmname SeSparse-VM --vmname_destination Linkedclone-SeSparse-Destination --snapname pristine --vmhost vesxi51-1.primp-industries.com --datastore datastore1 --convert source

If we head over to our vSphere Web Client and take a look at our new Linked Clones, we can confirm their virtual disks is using the new SE Sparse disk format.

Specifying Default Domains for vSphere Web Client Login

If you add an additional identity source to vCenter SSO, such as Active Directory during vCenter Server setup, you might have noticed that you need to specify the full domain name and user id to be able to login to the vSphere Web Client.

It took me awhile to figure out where to set the default domain as I thought it would have been in the VCSA web management interface as I deployed my VCSA using an automated script. I thought I share this quick tidbit in case someone was running into a similar issue.

To specify a default domain for one of your identity sources, you will need to login to vSphere Web Client as "root" or an administrator who has access to the vCenter SSO Configurations. Click on Administration tab and then to Configuration under "Sign-On and Discovery" on the left hand side of the screen.

You should see your AD identify source at the top as well as the two default identify source provided by vCenter SSO (these can not be removed). If you want to add an identify source as a default domain, you will need to highlight your identity source and then select the CD icon with the arrow, which will add that to list of default domains at the bottom of the screen. 

Once your domain is listed at the bottom, you will need to perform one additional step which is to actually save the configuration by clicking on the little "disk" icon. I did not realize this until I logged out and nothing changed.

Note: By default, when you add to the default domain list, you will append to the very end of the list. You have the ability to arrange the order by highlighting the domain and using the up/down arrows.

Go ahead and log out and log back in and now you only have to specify the username as the default domain will automatically be used.