What's New in VMware Vsish for ESXi 5.1

While going through and cleaning up my ghettoSkunkworks projects folder before going on PTO, I came across this little post which I must have forgotten to publish. With the latest release of ESXi 5.1, there are now a total of 162 new vsish configuration parameters, with 63 of those being hidden parameters.

***As usual with any hidden configurations, please be careful and use at your own risk as the vsish interface is not officially supported by VMware***

Here is the complete list of new vsish configurations in ESXi 5.1 - https://s3.amazonaws.com/virtuallyghetto-download/complete_vsish_config_510.html

 Additional Resources:

• What Is VMware Vsish
  
• What's New in VMware Vsish for ESXi 5

Nested Virtualization APIs For vSphere & vCloud Director 5.1

Did you know with the release of vSphere 5.1 and vCloud Director 5.1, there are now APIs that allow you to enable/disable Nested Virtualization aka (Virtual Hardware Virtualization) on a Virtual Machine?

Disclaimer: Nested Virtualization is not officially supported by VMware.

In the vSphere 5.1 API, there are two new properties related to Nested Virtualization:

• nestedHVSupported is a new capability property which indicates whether your physical ESXi 5.1 host supports Nested Virtualization. This property is only true IF your physical CPU supports BOTH Intel-VT/EPT OR AMD-V/RVI. For more details, please refer to this article.
• nestedHVEnabled is new property on a Virtual Machine which allows you to enable or disable Nested Virtualization. You will need to ensure you are running a Virtual Machine with ESXi 5.1 Compatibility (e.g. virtual hardware 9) and your physical ESXi 5.1 host supports Nested Virtualization. 

Here is a screenshot of performing the same operation manually using the new vSphere Web Client:

In the vCloud Director 5.1 REST API, there two new operations related to Nested Virtualization:

• /action/enableNestedHypervisor is a new operation on a Virtual Machine that enables Nested Virtualization. Here is an example using the curl utility enable VHV:

  curl -i -k -H "Accept:application/*+xml;version=5.1" -H "x-vcloud-authorization: xX4IrWWi+Ofq77zOqPJaMEYHHJt4jxrwP+ntkO2tecQ=" -X POST https://vcd.primp-industries.com/api/vApp/vm-d9870545-a29a-4175-bff4-ae075f1c1bc0/action/enableNestedHypervisor

•  /action/disableNestedHypervisor is a new operation on a Virtual Machine that disables Nested Virtualization. Here is an example using the curl utility to disable VHV:

  curl -i -k -H "Accept:application/*+xml;version=5.1" -H "x-vcloud-authorization: xX4IrWWi+Ofq77zOqPJaMEYHHJt4jxrwP+ntkO2tecQ=" -X POST https://vcd.primp-industries.com/api/vApp/vm-d9870545-a29a-4175-bff4-ae075f1c1bc0/action/disableNestedHypervisor

Here is a screenshot of performing the same operation manually using the vCloud Director UI:

If you plan to leverage Nested Virtualization in your environment, you now have simple way of automating this feature for Virtual Machines that you wish to support Nested Virtualization.

vGhetto Lab #NotSupported Slides Posted

As promised, here are slides to my #NotSupported session at VMworld Europe which I continued the theme of home labs with my vGhetto Lab #NotSupported presentation.

The idea behind the vGhetto Lab is to easily setup a vSphere home lab without too much effort and most importantly, leveraging as little resources as possible. This is all accomplished with the following:

• Physical host running ESXi 5.x
• ESXi 5.x offline depot image
• VCSA 5.x (vCenter Server Appliance)

In addition to the above, you will also need to download the vGhetto Lab scripts which are shown in the video.

Here are some additional details on how to quickly get setup with your own vGhetto Lab.

Step 1 - After installing ESXi 5.x on your physical host, you will need to deploy the VCSA. Make sure you add a second network interface to VCSA as shown in the presentation. In my example, I created another vSwitch with no uplinks and portgroup for Auto Deploy network

Step 2 - Once the VCSA is powered on, go ahead and SCP the scripts to virtual machine. The first script that we will need to execute is the setupNetwork.sh and you will need to edit the following variables:

Note: To ensure that you do not accidentally run the script without changing out the variables, there is another variable called ACTUALLY_READ_SCRIPT that needs to be changed from "no" to "yes" else the script will not execute.

Step 3 - Next we need to configure the vCenter Server, we will need to execute the configureVCSA51.sh which will configure the embedded SSO Database as well as the database of the vCenter Server. You do not need to edit any variables in this script

Step 4 - Finally, we need to configure our DHCP, TFTP, Auto Deploy services as well extracting the ESXi offline depot image and preparing it for use with Auto Deploy. You will need to edit the following variables before running the setupvGhettoLab.sh script.

Note: For the Image Profile and Auto Deploy rule creation, if you wish for the script to execute them automatically versus echoing to the screen, remove the "echo" statement as well as the double quotes from the following so the last three lines look like this:

Step 5 - You are now ready to create your nested ESXi virtual machines. You can use RVC as shown in the presentation (there is a slide at the very end which lists the commands) or you can connect to vSphere Web Client and create the ESXi virtual machines the traditional way via the GUI.

After updating the DHCP configurations with the new MAC Addresses from your nested ESXi virtual machines, you should then see Auto Deploy automatically provision your ESXi hosts and join them to the VCSA you deployed earlier. 

Additional Links:

vInception #NotSupported Slides Posted

Using the New vCloud Networking and Security to Load Balance Multiple vSphere Web Client Servers

If you followed my previous two articles Configuring Additional Windows vSphere Web Client 5.1 Servers & Configuring Additional VCSA 5.1 as vSphere Web Client Servers I demonstrated how easy it is to add additional vSphere Web Client Servers to provide additional connection load balancing as well as redundancy in accessing your vSphere 5.1 environment. The next logical step is to actually place a load balancer in front of all the vSphere Client Servers to provide a single entry point for users to connect to.

I did a quick search for some load balancer solutions for testing, but found that they were quite complex and non-intuitive to setup. Since this was for my home lab, I decided to try out the new vCloud Networking & Security 5.1 (previously known as vShield). This being my first time setting up a load balancer (usually a task given to Network Engineers), I was pleasantly surprised and shocked at how easy it was to deploy and configure a load balancer using the new vCloud Networking and Security. Instead of going through this step by step, I thought I record a quick video.

The video below shows how you can leverage the new vCloud Networking & Security 5.1 to deploy and configure a Load Balancer to load balance multiple vSphere Web Client Servers.


If you have not tried out the new vCloud Networking and Security, I highly recommend you give it a try, especially with all the new network and security features in the latest 5.1 release.

Adding Non-Supported ESXi Builds to VIN (vSphere Infrastructure Navigator)

I was recently asked whether it was possible to add a non-supported ESXi host into VIN (vSphere Infrastructure Navigator)? The reason for this request is if you are running an ESX(i) build which is not on the list of supported builds, VIN would fail to discover virtual machines on that host. You may see the following error "Access failed. Unsupported ESX version"

The latest release of VIN 1.2 supports the following ESX(i) builds:

• ESX/ESXi 3.5 (builds 425420, 408533, and 409724)

• ESX/ESXi 4.0 (builds 398348, 403553, and 403554)

• ESX/ESXi 4.1 (builds 433742, 433803, and 433804)

• and all builds of ESXi 5.x.

If you happen to be running an ESX(i) build which is not listed but the build number is greater than the ones shown above, then there is a workaround. You can add the non-supported ESX(i) build into VIN's whitelist which would allow VIN to discover the virtual machines. In the example below I will be adding an ESXi 4.1 Update 3 Build 800380 which is not listed as a supported build.

Disclaimer: ESX(i) hosts with build numbers that are smaller than the ones listed may still be added, but this will most likely not be supported as the list of default build numbers are the minimum requirements. Please thoroughly test this in a lab environment before applying to your production environment.

You will need SSH access to your VIN appliance and before we get started, we will quickly verify the list of supported ESX(i) build versions by querying the VIN database (I was able to find the details in /var/log/vadm/dbconfig.log). Run the following command which will connect to VIN Postgres DB:

psql -h 127.0.0.1 -U vadm -d inception

The password for the database is vadm

Next, we will run the following SQL query to display the list of supported ESX(i) build numbers which should match the release notes. Run the following command:

select * from valid_host_build;

As you can see from the screenshot, the ESX(i) build numbers matches those listed in the VIN 1.2 release notes and we can also see that a wildcard is also a valid input value for ESXi 5.x which denotes any build of 5.x supported as noted in the release notes.

To add a non-supported ESXi build number into VIN's DB, we will be using the following script /opt/vadm-engine/set_valid_host_versions.sh which is located in the VIN appliance.

Note: You do not need to stop or restart the VIN service to run the command.

The script accepts a very simple XML file that contains the list of supported ESX(i) build numbers and it is very IMPORTANT to note that this will override the original defaults. This is part of the reason we performed a query to the VIN DB to ensure we have a copy of the original build numbers as a reference.

To add our additional ESXi build number, we will need to construct an XML file containing both the original build numbers as well as our non-supported. In this example, I created a file called myHostVersion.xml (based on the original DB data) which contains following:
Next we will pass in our XML input file to the set_valid_host_versions.sh script. Here is a screenshot of what that looks like after the operation has been successfully completed:

If we log back into the VIN DB and query the list of supported ESX(i) builds, we should be able to see our new build number that we inserted into the DB along with the original defaults:

During the next automatic discovery cycle, VIN should now be able to discover virtual machines running on the ESXi host that was not supported earlier.

Changing VCSA Failed Login Attempt & Lock Out Period

I was going through my twitter-feed this morning and came across an interesting article by @herseyc Locked out of the vCenter Server Virtual Appliance. I recommend you give Hersey's article a read as it contains some very useful information about failed login attempts to the VCSA (vCenter Server Appliance).

There was a question posed at the end of the article on how to increase the number of login attempts before an account would be locked out. The answer is to simply modify one of the PAM modules on the VCSA, specifically /etc/pam.d/common-auth

The system default for login attempts before locking out an account is 3 and you can modify this by changing the following line, where X is the number of attempts:

auth    requisite       pam_tally.so deny=X

You also have the option of specifying an "unlock" time which will lock the account for the specified period of time after reaching the max failed login attempts. This can useful if you do not want to manually reset a user account due to a user fat fingering a password. For more details about these parameters, you can search on Google or refer to this article.

Note: The login attempts here is specific to the OS system login on the VCSA (5.0 & 5.1) and not vCenter Server. If you successfully login before hitting the maximum attempts, the tally will automatically reset back to 0.

In vSphere 5.1, and with the use of vCenter SSO, you now have an easy way of controlling password and lock out policy using the new vSphere Web Client. Here is a screenshot of where the configurations are located at:

Note: These policies only pertain to identity sources connected to vCenter SSO and not OS system logins.

Nested Virtualization Resources

Here is a consolidated page on all the articles that I have written about the Nested Virtualizatoin (nested ESXi, Hyper-V, etc) and all the goodies that are "Not Supported".

vSphere / vCloud 5.1

• Having Difficulties Enabling Nested ESXi in vSphere 5.1?
• How to Enable Nested ESXi & Other Hypervisors in vSphere 5.1 
• How to Enable Nested ESXi & Other Hypervisors in vCloud Director 5.1

vSphere / vCloud 5.0

• How to Enable Support for Nested 64bit & Hyper-V VMs in vSphere 5
• The Missing Piece In Creating Your Own Ghetto vSEL Cloud

Additional Tips/Tricks

• Nested ESXi 5.1 Supports VMXNET3 Network Adapter Type
• How to Configure Nested ESXi 5 to Support EVC Clusters
• How to Enable Nested vFT (virtual Fault Tolerance) in vSphere 5
• How to Install VMware VSA in Nested ESXi 5 Host Using the GUI
• Cool Undocumented Features in vCloud Director 1.5
• The Missing Piece In Creating Your Own Ghetto vSEL Cloud
• Nested Virtualization APIs For vSphere & vCloud Director 5.1 
• How To Enable Nested ESXi Using VXLAN In vSphere & vCloud Director

VCSA (vCenter Server Appliance) Resources

Here is a consolidated page on all the articles I have written about the VCSA (vCenter Server Appliance). Hopefully this will be useful when looking for anything related to VCSA.

VCSA 5.1

• Automating VCSA 5.1 (vCenter Server Appliance) Configurations
• How to Register a vCenter Server 5.0 with Admin Tool on VCSA 5.1 Using SSH Port Forwarding
• Automatically Join Multiple VCSA 5.1 using New vCenter SSO (Single Sign-On)
• Configuring Additional VCSA 5.1 as vSphere Web Client Servers
• Configuring New vSphere Web Client Session Timeout
• Specifying Default Domains for vSphere Web Client Login
• Default Password for vCenter SSO Admin Account on VCSA
• How to Add/Remove vCenter SSO Identity Sources Using the Command-Line for Windows vCenter Server & VCSA 
• VCSA (vCenter Server Appliance) 5.1 VCDB & SSODB Password  
• Seperating Out the vCenter SSO, vSphere Web Client and vCenter Server Services Using the VCSA 
• vCenter Server Simulator

VCSA 5.0

• Automating vCenter Server Appliance 5.0 (VCSA) Configurations

Additional Tips/Tricks

• Forwarding vCenter Server Logs to a Syslog Server
• How to Send vCenter Alarm Notification to Growl 
• Changing VCSA Failed Login Attempt & Lock Out Period  
• Getting Rid of the Inventory Tree in the New vSphere Web Client
• vCloud Director Simulator 
• Automating VCSA Network Configurations For Greenfield Deployments  
• Automating SSL Certificate Regeneration in VCSA (vCenter Server Appliance)

How to Add/Remove vCenter SSO Identity Sources Using the Command-Line for Windows vCenter Server & VCSA

The new vCenter SSO (Single Sign-On) in vSphere 5.1 can support multiple Identity Sources (Active Directory, OpenLDAP and Local OS) and these configurations can all be managed by using the vSphere Web Client. Here are some additional documentation as well as a video on how to managing Identity Sources using the vSphere Web Client.


However, if you wish to manage the Identity Sources through the command-line for automated deployments, you probably do not want to use the GUI. Luckily, there is a vCenter SSO CLI that you can use on both a Windows deployment as well as on the VCSA (vCenter Server Appliance).

In the examples below, I will show you how to add an Active Directory Identity Source to both a Windows deployment as well as a VCSA deployment using the command-line.

Adding Active Directory Identity Source to Windows vCenter SSO Server

Before getting started, you will need to know the vCenter SSO Administrator password. This is the password that you had configure during the installation of vCenter SSO for the admin@System-Domain account.

Step 1: Change into the C:\Program Files\VMware\Infrastructure\SSOServer\utils directory and you will be using the rsautil command-line tool. To get a list of help options for managing Identity Sources, run the following command:

rsautil manage-identity-sources

Step 2:  Run the following command (substitute the values for your own enviornment) :

rsautil manage-identity-sources -a create -u admin -p !VMware123! -r ldap://fullerene.primp-industries.com --ldap-port 3268 -d primp-industries.com -l PRIMP-IND --principal-base-dn DC=primp-industries,DC=com --group-base-dn DC=primp-industries,DC=com -f "" -L administrator@primp-industries.com

Note: When you login into the vSphere Web Client, the username for the vCenter SSO account is "admin@system-domain" but for the CLI, it is just "admin". If you do not have a failover LDAP Server, specify "" for -f flag.

Here is a screenshot of adding the Active Directory Identity Source:

Step 3: You can also confirm the configurations by performing the "list" operation by running the following command:

rsautil manage-identity-sources -a list -u admin -p !VMware123!

Adding Active Directory Identity Source to VCSA

For the VCSA, there are few additional steps as the vCenter SSO Administrator account "admin" does not contain a default password as noted in this article. You can either follow the instructions in that article to set a password for the "admin" user and then jump to Step 3 OR you can perform all required steps using the command-line. 

Step 1: We first need to set the vCenter SSO Master Password as it is required to change the "admin" user account. Since the vCenter SSO Master Password is auto-generated during installation, we will recover the account which will then allow us to set a password of our own choosing. Run the following two commands and in the second command, you will specify the vCenter SSO Master Password of your choice.

source /etc/vmware-sso/keys/recovery.cfg
/usr/lib/vmware-sso/utils/ssowrench manage-secrets -a change -u "$SSO_RECOVERY_USERNAME" -p "$SSO_RECOVERY_PASSWORD" -N vmware123

Note: Do not forget the vCenter SSO Master Password that you have selected, as there is no way to recover the account after this.

Step 2: Now that we have the vCenter SSO Master Password, we can now set a password for the "admin" user. Run the following command and specify the Master Password (-m flag) as well as a new password (-p flag) for the admin (-u flag) user.

/usr/lib/vmware-sso/utils/ssowrench reset-admin-password -u admin -p '!VMware123!' -m vmware123

Step 3: Once we have a password set for the "admin" user, we will be using the same vCenter SSO CLI utility as we did in the Windows vCenter Server deployment but in the VCSA it is called ssowrench and it is located under /usr/lib/vmware-sso/utils

Run the following command which accepts the same set of options as the Windows version (substitute the values for your own enviornment).

/usr/lib/vmware-sso/utils/ssowrench manage-identity-sources -a create -u admin -p '!VMware123!' -r "ldap://fullerene.primp-industries.com" --ldap-port 3268 -d primp-industries.com -l PRIMP-IND --principal-base-dn "DC=primp-industries,DC=com" --group-base-dn "DC=primp-industries,DC=com" -f "" -L administrator@primp-industries.com

Here is a screenshot of adding the Active Directory Identity Source:

Step 4: To view all Identity Sources, you can run the following command (as seen in the screenshot above):

/usr/lib/vmware-sso/utils/ssowrench manage-identity-sources -a list -u admin -p '!VMware123!'

Note: If you do not wish to specify any of the credentials on the command-line, the utility in both Windows and VCSA will automatically prompt for input. 

Changing GuestOS Type Using a Custom vCO Workflow in the vSphere Web Client

Something you might not have noticed, is the fact that you can not change or modify the guestOS type after a virtual machine has been created in the new vSphere Web Client, this option is just grayed out.

Though this is a change in behavior compared to the old vSphere C# Client, I actually took this as an opportunity to try out one of the most interesting and unrealized feature in the vSphere 5.1 release. This feature being a tighter integration between vCenter Server and vCenter Orchestrator. This means that you can now take any of your existing vCO workflows or create new workflows and make them directly available to any of the vSphere objects within the new vSphere Web Client as a custom action. 

Note: A feature request/bug has already been filed with VMware to have the ability to change the Guest OS and Guest OS Version for a virtual machine after creation in the vSphere Web Client.

Here is an example of a custom workflow that I created called Change Guest OS Type and as you can see that it only shows up under the context of a virtual machine object in the vSphere Web Client. 

From my perspective, the use cases are endless as you can create ANY custom workflow to perform any action or series of operations that can span across VMware products as well as 3rd party systems and directly present them to your end users in the new vSphere Web Client. Not only that, users can specify which workflows they see by default on a given vSphere object and this can differ from user to user based on their daily set of tasks.

So going back to our scenario, here is a way to change the Guest OS and Guest OS Version using a custom vCO workflow.

Step 1 - Download Change Guest OS Type vCO workflow to local desktop.

Step 2 - Open up the vCO Workflow Client, you can do this by pointing your browser to your vCO Server and click on "Start Orchestrator Client" link.

Step 3 - Import the Change Guest OS Type vCO workflow from your desktop to your vCO Server

Step 4 - Next, we need to go to the vSphere Web Client to make this vCO workflow available on a particular vSphere object, in our case it is a virtual machine. On the home page of the vSphere Web Client, click on "vCenter Orchestrator" icon in the center pane or select it from the navigation pane on the left. Once you are in the vCenter Orchestrator configuration page, select the "Manage" tab and click on the "plus" icon.

In this view, you can specify which default vCO workflows are made available across the various vSphere objects. These can be modified or removed based on the frequency of workflow usage.

Step 5 - Locate the Change Guest OS Type vCO workflow on the left hand side and then click on the Add button. Finally, select type to be virtual machine as this workflow is only applicable to a VM and OK to save the settings.

If we take a look at the vCenter Orchstrator configuration page, we will see our new workflow is now listed as one of the defaults for a virtual machine object. You can edit and modify any of these based on the workflows you wish to see by default. I highly recommend you add workflows that you use frequently so you do not have to search through the entire list each time.

Finally, it is time to test drive our new workflow! Locate a virtual machine and right click on the object, in a second you should see a sub-menu for All vCenter Orchestrator Actions and then select our vCO workflow Change Guest OS Type which will start off a very familiar wizard.

The first screen is the object selected, which in our case is our virtual machine. You can of course change this, but we will leave it as it's context was automatically picked up.

The next screen is to select the Guest OS Family (Windows, Linux & Other) that you wish to modify your virtual machine to.

The last part is just to select the Guest OS Version which is provided as a list of the guest OSes based on your previous selection.

To apply the Guest OS change, just click finish and watch the vCO workflow execute.

Though the functionality of changing the Guest OS is not available in the new vSphere Web Client, you can still provide the same functionality to your end users through a custom vCO workflow which are now tightly integrated into the vSphere Web Client. Hopefully this sparks some ideas on other vCO workflows you can create or expose through the vSphere Web Client in your own environment. I know I have a few in mind :)

A big thanks goes out to Christophe Decanini for helping me with a few questions while creating this workflow.