ovftool

Heads Up: OVF/OVA always deployed as Thick on VSAN when using vSphere Web Client

06/03/2016 by William Lam 25 Comments

Just wanted to give folks a heads up on an issue that a colleague of mines recently identified when provisioning Virtual Appliances (OVF/OVA) onto a VSAN datastore when using the vSphere Web Client. He found that regardless of the VSAN Storage Policy that was selected, whether it is the default VSAN Storage Policy or a custom one, the Virtual Appliance will always be Thick provisioned.

This behavior only occurs when using the vSphere Web Client and is not observed when using either the vSphere C# Client or the ovftool CLI. My understanding of the issue is that there are two ways in which a VM can get provisioned as Thin. The "old" method which was to explicitly specify the disk allocation type (Thin vs Thick) and the "new" method which uses VM Storage Policies. To ensure that we maintain backwards compatibility for older clients, if a client specifies Thick provisioned, it would actually override the VM Storage Policy even if the Object Space Reservation capability was set to 0 (Thin provisioned). Since you can no longer specify the disk allocation type in the vSphere Web Client, the default behavior is to not Thin provision and hence the current Thick provisioning result even though the default VSAN Storage Policy has OSR set to 0.

Note: When referring to Thick provisioned in VSAN (proportionalCapacity = 100), It is defined as provisioned Thin with a reservation so there is a guarantee that space is available for the object. It is not accurate to compare this to Zeroed Thick or Eager Zeroed Thick in the VMFS/NFS world as VSAN is an Object Store.

Engineering has already been engaged and is currently investigating the issue. We have also asked for a VMware KB to be published, so hopefully once that goes up, folks can subscribe to that for more details and updates.

In the meantime, since it is actually pretty difficult to see if you have been affected by issue, I have created a simple PowerCLI script called Get-VSANPolicy.ps1 which will allow you to quickly scan through your VM(s) to identify whether you have any VMs that have been Thick provision residing on a VSAN Datastore. You can either get all VMs by piping Get-VM * or a specific set of VMs into the script.

The following example retrieves all VMs that start with "Photon-Deployed-From-*" and extracts their current VSAN VM Storage Policy for both VM Home and individual VMDKs. Here, we can see that both VMs are using the default VSAN VM Storage Policy.

Get-VM "Photon-Deployed-From-*" | Get-VSANPolicy -datastore "vsanDatastore"

Lets now only search for VMs that have been Thick provisioned by using the -thick option and setting that to true. Here we can see that the OVF we provisioned through the vSphere Web Client is the only VM listed.

Get-VM "Photon-Deployed-From-*" | Get-VSANPolicy -datastore "vsanDatastore" -thick $true

If we want to get more details on the underlying VM Storage Policy that was applied, we can also specify the -details option to true. Here we can clearly see that the 2nd VM has proportionalCapacity=100 which means Thick provision.

Get-VM "Photon-Deployed-From-*" | Get-VSANPolicy -datastore "vsanDatastore" -thick $true -details $true

Luckily, the fix is quite easy thanks to Paudie O'Riordan who found out that it was as simple as just re-applying the VSAN VM Storage Policy! (Policy Based Management FTW!) This means there is no need to perform unnecessary Storage vMotions to be able to convert the VM from Thick to Thin, it is literally a couple of clicks in the UI.

UPDATE (07/15/16) - Thanks to reader Jose, it looks like using the vSphere Web Client to re-apply the VSAN VM Storage Policy will correctly apply the policy to the VM/VMDKs, but does not reclaim the underlying storage. It is recommended that you use the PowerCLI script below to re-apply the policy which will then properly reclaim the underlying storage and will properly reflect the storage utilization.

As with anything, I still prefer Automation and with that, I have created a secondary script to help with the remediation. This is also a PowerCLI script called Set-VSANPolicy.ps1 which accepts a list of VMs and the name of the VSAN VM Storage Policy that you wish to re-apply.

Here is an example of running the script and remediating two VMs that contains multiple VMDKs:

Set-VSANPolicy -listofvms $listofvms -policy $vsanpolicy

If you now re-run the first script, you should see that you no longer have VMs that are provisioned Thick anymore (this may take some time depending on the size of your VMs).

Caveat when deploying Photon Controller Installer (v0.8) OVA to vCenter Server

04/11/2016 by William Lam Leave a Comment

I have recently been spending some time exploring the latest release of Photon Controller (v0.8). One of the new features in this release is the ability to deploy Photon Controller using a new UI installer provided by a virtual appliance. Since I already have a vCenter Server running in my lab environment, I decided to deploy the Photon Controller Installer OVA using the vSphere Web Client. There are several OVF properties that you can configure as part of deploying the appliance, just like you would with any VMware/3rd Party based virtual appliance. Below is a screenshot of the available OVF properties when deploying the Photon Controller Installer OVA.

I came to find out that the OVF properties that were specified were not actually being obeyed after the VM was powered on. I was scratching my head for a bit and after a big of digging, I found out that the OVA was actually missing a very important configuration which prevented the OVF properties from properly being passed down into the guestOS for customization. The fix was actually quite simple which I had also reported to the Engineering team. Below are the steps to take to properly deploy Photon Controller Installer OVA to vCenter Server.

Step 1 - Deploy the Photon Controller Installer OVA using either vSphere Web/C# Client and fill out the OVF Properties, but do NOT power on the VM yet.

Step 2 - Right click and Edit the Photon Controller VM that you had just deployed. Click on "vApp Options" tab and scroll to the bottom of that and check the "VMware Tools" box. This will ensure that the OVF properties (ovfEnv) can be passed down to the guestOS for customization.

Here is a screenshot for those using the vSphere C# Client to connect to their vCenter Server.

Step 3 - Power on the VM and it should be configured with the settings you had specified in the OVF properties.

Note: The password that you specify in the OVF properties is NOT the password to the esxcloud account but rather the password required to switch over to root account via su. I know the description might be confusing but the default password for esxcloud account is using default vmware.

For those of you wondering, the following tweak needs to be made to the OVA for this to work out of the box for vCenter Server. You just need to convert the OVA to an OVF and then adjust the following line from:

If you do not have a vCenter Server and wish to deploy the Photon Controller OVA directly to an ESXi host, you can do so by injecting the OVF properties by using either ovftool or govc CLI as ESXi does not have support for OVF properties.

Here is a quick snippet that you can use to deploy using ovftool:

PHOTON_CONTROLLER_OVA="/Volumes/Storage/Images/installer-vm.ova"
ESXI_HOST="192.168.1.50"
ESXI_USERNAME="root"
ESXI_PASSWORD="vmware123"
PC_PASSWORD="VMware1!"
PC_VM_NAME="Photon-Controller-Deployer"
PC_IP_ADDRESS="192.168.1.25"
PC_NETMASK="255.255.255.0"
PC_GATEWAY="192.168.1.1"
PC_NTP_SERVER="pool.ntp.org"
PC_DNS_SERVER="192.168.1.1"
PC_VM_NETWORK_NAME="VM Network"
PC_DATASTORE="vsanDatastore"
'/Applications/VMware OVF Tool/ovftool' --acceptAllEulas --allowAllExtraConfig --X:injectOvfEnv --powerOn --diskMode=thin "--name=${PC_VM_NAME}" "--datastore=${PC_DATASTORE}" "--net:NAT=${PC_VM_NETWORK_NAME}" "--prop:ip0=${PC_IP_ADDRESS}" "--prop:netmask0=${PC_NETMASK}" "--prop:gateway=${PC_GATEWAY}" "--prop:DNS=${PC_DNS_SERVER}" "--prop:ntp_servers=${PC_NTP_SERVER}" "--prop:admin_password=${PHOTON_CONTROLLER_PASSWORD}" "${PHOTON_CONTROLLER_OVA}" "vi://${ESXI_USERNAME}:${ESXI_PASSWORD}@${ESXI_HOST}"

What is the VMware Client Integration Plugin (CIP)?

12/10/2015 by William Lam 3 Comments

If you are a consumer of the vSphere Web Client, you might have seen something called the VMware Client Integration Plugin (CIP) and you may have even downloaded it from bottom of the vSphere Web Client page and installed it on your desktop.

However, have you ever wondered what CIP is actually used for? I know I personally have even though I have a general idea of what CIP provides, I have always been curious myself about the technical details. Recently there have been a few inquiries internally, so I figure I might as well do some research to see what I can find out.

The VMware CIP is actually a collection of different tools that are bundled together into a single installer that is available for either Microsoft Windows or Apple Mac OS X (Linux is being worked on). These tools provide a set of capabilities that are enabled when using the vSphere Web Client and below is a diagram of the different components included in CIP.

ovftool - Standalone CLI utility used to manage import/export of OVF and OVA images
Windows Authentication - Allows the use of SSPI when logging in from the vSphere Web Client
Remote Devices - Connecting client side devices such as a CD-ROM, Floppy, USB, etc. to VM
File Upload/Download - Datastore file transfer
Content Library - Operations related to the Content Library feature such as import and export
Client Side Logging/Config - Allows for writing non-flash logs + vSphere Web Client flash and logging settings

In addition to capabilities shown above, CIP is also used to assist with basic input validation when deploying the vCenter Server Appliance deployment using the new guided UI installer.

Internally, CIP is referred to as the Client Support Daemon or CSD for short. Prior to vSphere 6.0 Update 1, CIP ran as a browser plugin relying on the Netscape Plugin Application Programming Interface (NPAPI). In case you had not heard, Google Chrome and other popular browsers have all recently removed support for NPAPI based plugins in favor of better security and increased speed improvements. Due to this change, CIP had to be re-written to no longer rely on this interface and starting with vSphere 5.5 Update 3a and vSphere 6.0 Update 1, the version of CIP that is installed uses this new implementation. CIP is launched today via a protocol handler which is a fancy term for a capability web browsers that allows you to run a specific program when a link is open.

One observation that some customers have made including myself when installing the CIP is that an SSL Certificate is generated during the installation process. To provide the CIP services to the vSphere Web Client, a secure connection must be made to vSphere Web Client pages. To satisfy this requirement, a self-signed SSL Certificate is used and instead of pre-packaging an already generated certificate, one is dynamically created to ensure that no 3rd Party would have access to the private key and be able to access it from the outside.

The longer term plan is to try to move as much of the CIP functionality onto the server side as possible, although not everything will be able to move to the server side. For things like remote devices, it has already been moved to the Standalone VMRC which already provides access to the VM Console and being able to connect to client side devices makes the most sense. Hopefully this gives you a better understanding of what CIP provides and hint of where it is going in the future.

Here are some additional info that you might find useful when installing and troubleshooting CIP:

CIP Installer Logs:

Windows -
%ALLUSERSPROFILE%\VMware\CIP\csd\logs
Mac OS X -
/Applications/VMware Client Integration Plug-in.app/Contents/Library/data/logs

CIP Application Logs:

Windows -
%USERPROFILE%\AppData\Local\VMware\CIP\csd\logs
Mac OS X -
$HOME/VMware/CIP/csd/logs

vSphere Web Client / CSD Session Logs:

Windows -
%USERSPROFILE%\VMware\CIP\ui\sessions
Mac OS X -
$HOME/VMware/CIP/ui/sessions

CIP SSL Certificate Location:

Windows -
%ALLUSERSPROFILE%\VMware\CIP\csd\ssl
Mac OS X -
/Applications/VMware Client Integration Plug-in.app/Contents/Library/data/ssl

How to deploy and run the VSAN 6.1 Witness Virtual Appliance on VMware Fusion & Workstation?

09/11/2015 by William Lam 18 Comments

One of the most exciting new feature in VSAN 6.1 is the new Stretched Clustering capability which also provides support for a 2-Node ROBO deployment. If you are interested in learning more about the new VSAN 6.1 capabilities, be sure to check out Duncan's blog post here as well as a video on how to configure the new VSAN Stretched Clustering here. Like many of you, I am sure you are looking forward to giving both vSphere 6.0 Update 1 as well as the new VSAN 6.1 capabilities a spin in your home lab or development environment. By now, you probably know how easy it is to run Nested ESXi on top of your existing vSphere environment. However, not everyone has access to a vSphere environment. The next best thing is using VMware Fusion and Workstation which also supports Nested ESXi and for many of our customers and field, it is a great solution as it allows you to easily play with all the VMware goodies while you are on the go, especially useful if you travel frequently.

I was interested in setting up a 2-Node VSAN configuration and as part of the setup, you will also need to deploy the new VSAN Witness Virtual Appliance. I wanted to see what it would take to deploy the VSAN Witness Appliance onto VMware Fusion and Workstation and after a bit of exploration, hair pulling and OVF hackery, I was able to finally work out the process as shown in the steps below.

Disclaimer: This is not officially supported by VMware, please use this only for evaluational and testing purposes.

Note: If you plan on deploying the VSAN Witness Appliance directly to an ESXi host, you can use the injectOvfEnv method shown here.

Step 1 - Download the VSAN 6.1 Witness Virtual Appliance OVA from here

Step 2 - We need to make a few minor adjustments to the OVF file before we can import it into VMware Fusion/Workstation. Before we do so, we need to first convert the OVA to an OVF using ovftool.

Here is an example of running this on Mac OS X system:

/Applications/VMware\ OVF\ Tool/ovftool VMware-VirtualSAN-Witness-6.0.0.update01-3029758.ova VMware-VirtualSAN-Witness-6.0.0.update01-3029758.ovf

Once the conversion has completed, you should see a total of 8 files (6 VMDK files, 1 manifest & 1 OVF file). Before moving onto the next step, you will need to delete the manifest file (extension ending in .mf). We need to do this because the checksum will no longer be valid after we edit the OVF and the upload will fail.

Step 3 - The first edit that we need to make in the OVF is to change the required OVF parameter for specifying the password to the VSAN Witness Appliance from "true" to "false" since both Fusion/Workstation do not support OVF properties.

Change the following from

Step 4 - The final edit that we need to make in the OVF is to specify the deployment size (tiny, medium or large) for the Witness VM. By default, it will use the medium option. Below is a quick table of the deployment size and for evaluation/testing purposes, I suspect most you will want to stick with the "tiny" configuration.

Deployment Size	vCPU	vMEM	vDISKs	Components
tiny	2	8GB	1x8GB 1x15GB 1x10GB	750
medium	2	16GB	1x8GB 1x350GB 1x10GB	22K
large	2	32GB	1x8GB 1x350GB 1x10GB	45K

The way this is specified in the OVF is by adding ovf:default="true" to the specific deployment size. As mentioned, by default this is set to the "medium" deployment size which looks like the following:

If you wish to change this to some other deployment size, you will need to move the ovf:default="true" entry to the deployment size you wish to use. In our case, we will move it to "tiny" size by adding the following

Step 5 - Now that we are done with the OVF surgery, we can now import our VSAN Witness OVF into VMware Fusion or Workstation using the "Import" option. Ensure you select "Customize" option after the import has completed to prevent the VM from automatically powering on. This is very important because we still need to add one final configuration to the VMX file for the appliance to be properly configured.

Step 6 - The last and final step is to add a VMX entry which will configure the root password for VSAN Witness Appliance. This is not necessary when deploying to vCenter Server which has OVF support, but is a problem when deploying VMware Fusion, Workstation and even ESXi. You will need to add the following entry to the VMX file and be sure to replace the password of your choice which is highlighted below in blue.

guestinfo.ovfEnv = "<?xml version='1.0' encoding='UTF-8'?><Environment xmlns='http://schemas.dmtf.org/ovf/environment/1' xmlns:oe='http://schemas.dmtf.org/ovf/environment/1'><PropertySection><Property oe:key='vsan.witness.root.passwd' oe:value='vmware123'/></PropertySection></Environment>"

Step 7 - You are now ready to power on your VSAN Witness and if everything was configured correctly, you should be able to login to DCUI of the VSAN Witness which as you probably have guessed by now is running Nested ESXi 🙂

Here is a quick screenshot of configuring my very first 2-Node / Stretched VSAN Cluster which is comprised of 2 Nested ESXi VMs running on my Mac Mini which is running vSphere 6.0 Update 1 and my VSAN Witness Appliance running on my iMac using VMware Fusion. If you need additional instructions on configuring either a 2-Node / Stretched VSAN Cluster, be sure to check out the how-to video here. I plan to share this feedback internally with Engineering and hopefully deploying VSAN Witness in the future can be much more easier when it comes to running it on VMware Fusion and Workstation.

Quick Tip – Using ovftool to connect to vCloud Director behind a 2-Factor Authentication portal

06/08/2015 by William Lam 2 Comments

A couple of months back I ran into an issue while trying to upload a couple of files to VMware's internal OneCloud environment which uses vCloud Director. The issue that I encountered was that our OneCloud environment no longer supported basic username/password authentication, which I hoping to automate using ovftool. Instead, it is now front-ended with VMware Application Manager which requires 2-Factor Authentication (2FA) and once authenticated, a SAML token is then passed to vCloud Director which then automatically logs you in.

The problem with this is that ovftool can not be used to directly login to vCloud Director as it does not have support for 2FA, which makes automating operations against our OneCloud environment pretty difficult. After spending a few days looking for an alternative and not having any luck, my last hope was to reach out to the ovftool developers to see if they had seen this before.

After a few email exchanges, although ovftool does not support 2FA, you can however get this to work using session ticket authetnication which it does support using either the --I:sourceSessionTicket or --l:targetSessionTicket option. To get more details on these options, you can run the following ovftool command:

ovftool --help integration

In order to use this session ticket mechanism to authenticate into a 2FA environment, you must first login manually using a web browser. Once you have successfully logged in, you will need to either use browser developer tool or something like Firebug to record the authenticated vCloud Director cookie which will then be passed to ovftool.

In this example, I am using Chrome and you can find the Developer Tools by going to Options->More Tools->Developer Tools. Next, refresh the webpage so you are able to see the web requests between your browser and vCloud Director. Now navigate to Network->Cookies option and select any one of the requests to the left of the screen such as "amf".

What you will be looking for is the value to the cookie named "vcloud_session_id" which is is the authenticated session that we will use to provide to ovftool. Once you have that value, you can then specify the connection to ovftool using the following:

ovftool --I:targetSessionTicket=[VCLOUD_SESSION_ID_VALUE] vcloud://...

I had known that ovftool supported session based tickets, however I did not realize it could be used to authenticate behind a 2FA solution like VMware Application Manager. For folks interested in using ovftool and session based tickets directly with vSphere, check out this awesome post by my good friend Jake Robinson who demonstrates this using PowerCLI, ovftool and the AcquireCloneTicket() vSphere API method.

Quick Tip – How to upload files to vCloud Air On-Demand using ovftool?

05/04/2015 by William Lam 4 Comments

I have recently spent some time playing around with the new vCloud Air On-Demand offering and with the free $300 credit promotion which I am currently using, it is an easy and painless way of evaluating vCloud Air. A common operation that many of you will most likely be performing when you first get setup is uploading content to be used within vCloud Air. You can do this by either using the UI in vCloud Director OR you can use one of my favorite command-line tools called ovftool.

To assist with using ovftool, I have created a simple shell script called upload-to-vca.sh which wraps ovftool and you simply just need to fill out a couple of required variables and the script will automatically handle the rest.

Note: Though the script is is specifically for a *nix/Mac OS X environment, the ovftool syntax is exactly the same for a Windows based environment. You just need to change the variable type and can then easily get this working on a Windows system using a simple batch or PowerShell script.

There are four pieces of information that you will need to pull from your vCloud Air On-Demand environment which will map to the following variables:

VCA_URL - This is the vCloud Director URL found when logging into your vCloud Air On-Demand instance
VCA_ORG_NAME - This is the name of your Organization for your vCloud Air On-Demand, it will be in a UUID type format
VCA_ORG_VDC_NAME - This is the name of your Virtual Datacenter that you would need to create when initially logging int
VCA_CATALOG_NAME - This is the vCloud Director Catalog name, by default it is called "default-catalog" unless you go into vCloud Director to create a custom one

To find the VCA_URL you just need to look at the base of the URL when logged into your vCloud Air On-Demand instance, in my example it is us-california-1-3.vchs.vmware.com and to find the VCA_ORG_VDC_NAME you will look under the "Virtual Data Centers" section as seen in the screenshot below. In my example, it is called vGhetto-VDC

To find the next two variables, you will need to open up the vCloud Director UI. To do so, just right click on the name of your VDC and select "Manage Catalogs in vCloud Director". Once logged in, you can find the VCA_ORG_VDC_NAME by looking at the URL after the /org/ entry (which you can copy/paste) but it is also displayed at the upper left corner when you first log in. In my example, it is the b51b26... Finally, to find the VCA_CATALOG_NAME you just need to click on the Catalogs tab and by default vCloud Air will use default-catalog as the name of your private Catalog. If you decide to create a new one, then you will need to use that name instead.

Once we have all four of the variables filled out based on our vCloud Air On-Demand instance, we also need to specify within the script whether we want to upload an ISO/floppy image or an OVA/OVF. There are two examples in the script and just un-comment the one you wish to run. Once you have completed editing the script, you can go ahead and run it and it will prompt you for your vCloud Air credentials which is what you used to initially sign into vchs.vmware.com.

Here is an example of uploading an OVF to my vCloud Air On-Demand VDC:

In my environment, I have uploaded both a Nested ESXi OVF template which should be stored under the vApp Tempalates tab as well as an ESXi 6.0 ISO which will be stored under the Media & Other section as shown in the screenshots below.

At this point, I can now consume both my OVF and ISO either directly in the vCloud Director UI or I can jump back to my vCloud Air UI and consume it that way as shown in the screenshot below.

If you have a lot content to upload or just prefer using the CLI (like I do), then this is a great way to easily upload content so that it is accessible in vCloud Air.