ova

Default hashing algorithm changed in OVFTool 4.2 preventing OVF/OVA import using vSphere C# Client

11/18/2016 by William Lam 9 Comments

After upgrading my home lab recently to vSphere 6.5, I also updated some of the related utilities such as the various SDKs and CLIs. One of the CLIs that I had updated was the latest version of OVFTool which is now at 4.2. I use OVFTool extensively to automate various Virtual Machine deployments (import/export). While testing out a new OVA that I had been working on, I needed to verify that it also worked with previous release of vSphere like vSphere 6.0 Update 2. I happen to have the vSphere C# Client open and connected to a vCenter Server and when I tried to import the newly created OVA, but it failed with the following error message:

The following manifest file entry(line1) is invalid: SHA256

I was pretty surprised by this since I went through this exact same workflow a couple of days ago without any problems. The only change that had happened was OVFTool and error seems to indicate an issue with the hashing algorithm. I ran OVFTool again using just the --help option to check what the default SHA hashing algorithm was, it was SHA256. I then compared that to an older version of OVFTool and it looks like the default had changed from SHA1 to SHA256.

From a security standpoint, this is a positive change as SHA1 is no longer considered a secure hashing algorithm and a stronger version should be used. It also turns out that the vSphere C# Client can only support SHA1 which is why I received the error after upgrading to the new version of OVFTool. Luckily, this is NOT a problem when using the vSphere Web Client or the vSphere HTML5 Client and only affects the vSphere C# Client. If you do need to use the vSphere C# Client for importing OVF/OVAs exported from the latest version of OVFTool, the workaround is quite simple, just override the default hashing algorithim when exporting by adding the additional CLI option:

--shaAlgorithm=sha1

Heads Up: OVF/OVA always deployed as Thick on VSAN when using vSphere Web Client

06/03/2016 by William Lam 16 Comments

Just wanted to give folks a heads up on an issue that a colleague of mines recently identified when provisioning Virtual Appliances (OVF/OVA) onto a VSAN datastore when using the vSphere Web Client. He found that regardless of the VSAN Storage Policy that was selected, whether it is the default VSAN Storage Policy or a custom one, the Virtual Appliance will always be Thick provisioned.

This behavior only occurs when using the vSphere Web Client and is not observed when using either the vSphere C# Client or the ovftool CLI. My understanding of the issue is that there are two ways in which a VM can get provisioned as Thin. The "old" method which was to explicitly specify the disk allocation type (Thin vs Thick) and the "new" method which uses VM Storage Policies. To ensure that we maintain backwards compatibility for older clients, if a client specifies Thick provisioned, it would actually override the VM Storage Policy even if the Object Space Reservation capability was set to 0 (Thin provisioned). Since you can no longer specify the disk allocation type in the vSphere Web Client, the default behavior is to not Thin provision and hence the current Thick provisioning result even though the default VSAN Storage Policy has OSR set to 0.

Note: When referring to Thick provisioned in VSAN (proportionalCapacity = 100), It is defined as provisioned Thin with a reservation so there is a guarantee that space is available for the object. It is not accurate to compare this to Zeroed Thick or Eager Zeroed Thick in the VMFS/NFS world as VSAN is an Object Store.

Engineering has already been engaged and is currently investigating the issue. We have also asked for a VMware KB to be published, so hopefully once that goes up, folks can subscribe to that for more details and updates.

In the meantime, since it is actually pretty difficult to see if you have been affected by issue, I have created a simple PowerCLI script called Get-VSANPolicy.ps1 which will allow you to quickly scan through your VM(s) to identify whether you have any VMs that have been Thick provision residing on a VSAN Datastore. You can either get all VMs by piping Get-VM * or a specific set of VMs into the script.

The following example retrieves all VMs that start with "Photon-Deployed-From-*" and extracts their current VSAN VM Storage Policy for both VM Home and individual VMDKs. Here, we can see that both VMs are using the default VSAN VM Storage Policy.

Get-VM "Photon-Deployed-From-*" | Get-VSANPolicy -datastore "vsanDatastore"

Lets now only search for VMs that have been Thick provisioned by using the -thick option and setting that to true. Here we can see that the OVF we provisioned through the vSphere Web Client is the only VM listed.

Get-VM "Photon-Deployed-From-*" | Get-VSANPolicy -datastore "vsanDatastore" -thick $true

If we want to get more details on the underlying VM Storage Policy that was applied, we can also specify the -details option to true. Here we can clearly see that the 2nd VM has proportionalCapacity=100 which means Thick provision.

Get-VM "Photon-Deployed-From-*" | Get-VSANPolicy -datastore "vsanDatastore" -thick $true -details $true

Luckily, the fix is quite easy thanks to Paudie O'Riordan who found out that it was as simple as just re-applying the VSAN VM Storage Policy! (Policy Based Management FTW!) This means there is no need to perform unnecessary Storage vMotions to be able to convert the VM from Thick to Thin, it is literally a couple of clicks in the UI.

UPDATE (07/15/16) - Thanks to reader Jose, it looks like using the vSphere Web Client to re-apply the VSAN VM Storage Policy will correctly apply the policy to the VM/VMDKs, but does not reclaim the underlying storage. It is recommended that you use the PowerCLI script below to re-apply the policy which will then properly reclaim the underlying storage and will properly reflect the storage utilization.

As with anything, I still prefer Automation and with that, I have created a secondary script to help with the remediation. This is also a PowerCLI script called Set-VSANPolicy.ps1 which accepts a list of VMs and the name of the VSAN VM Storage Policy that you wish to re-apply.

Here is an example of running the script and remediating two VMs that contains multiple VMDKs:

Set-VSANPolicy -listofvms $listofvms -policy $vsanpolicy

If you now re-run the first script, you should see that you no longer have VMs that are provisioned Thick anymore (this may take some time depending on the size of your VMs).

Caveat when deploying Photon Controller Installer (v0.8) OVA to vCenter Server

04/11/2016 by William Lam Leave a Comment

I have recently been spending some time exploring the latest release of Photon Controller (v0.8). One of the new features in this release is the ability to deploy Photon Controller using a new UI installer provided by a virtual appliance. Since I already have a vCenter Server running in my lab environment, I decided to deploy the Photon Controller Installer OVA using the vSphere Web Client. There are several OVF properties that you can configure as part of deploying the appliance, just like you would with any VMware/3rd Party based virtual appliance. Below is a screenshot of the available OVF properties when deploying the Photon Controller Installer OVA.

I came to find out that the OVF properties that were specified were not actually being obeyed after the VM was powered on. I was scratching my head for a bit and after a big of digging, I found out that the OVA was actually missing a very important configuration which prevented the OVF properties from properly being passed down into the guestOS for customization. The fix was actually quite simple which I had also reported to the Engineering team. Below are the steps to take to properly deploy Photon Controller Installer OVA to vCenter Server.

Step 1 - Deploy the Photon Controller Installer OVA using either vSphere Web/C# Client and fill out the OVF Properties, but do NOT power on the VM yet.

Step 2 - Right click and Edit the Photon Controller VM that you had just deployed. Click on "vApp Options" tab and scroll to the bottom of that and check the "VMware Tools" box. This will ensure that the OVF properties (ovfEnv) can be passed down to the guestOS for customization.

Here is a screenshot for those using the vSphere C# Client to connect to their vCenter Server.

Step 3 - Power on the VM and it should be configured with the settings you had specified in the OVF properties.

Note: The password that you specify in the OVF properties is NOT the password to the esxcloud account but rather the password required to switch over to root account via su. I know the description might be confusing but the default password for esxcloud account is using default vmware.

For those of you wondering, the following tweak needs to be made to the OVA for this to work out of the box for vCenter Server. You just need to convert the OVA to an OVF and then adjust the following line from:

If you do not have a vCenter Server and wish to deploy the Photon Controller OVA directly to an ESXi host, you can do so by injecting the OVF properties by using either ovftool or govc CLI as ESXi does not have support for OVF properties.

Here is a quick snippet that you can use to deploy using ovftool:

PHOTON_CONTROLLER_OVA="/Volumes/Storage/Images/installer-vm.ova"
ESXI_HOST="192.168.1.50"
ESXI_USERNAME="root"
ESXI_PASSWORD="vmware123"
PC_PASSWORD="VMware1!"
PC_VM_NAME="Photon-Controller-Deployer"
PC_IP_ADDRESS="192.168.1.25"
PC_NETMASK="255.255.255.0"
PC_GATEWAY="192.168.1.1"
PC_NTP_SERVER="pool.ntp.org"
PC_DNS_SERVER="192.168.1.1"
PC_VM_NETWORK_NAME="VM Network"
PC_DATASTORE="vsanDatastore"
'/Applications/VMware OVF Tool/ovftool' --acceptAllEulas --allowAllExtraConfig --X:injectOvfEnv --powerOn --diskMode=thin "--name=${PC_VM_NAME}" "--datastore=${PC_DATASTORE}" "--net:NAT=${PC_VM_NETWORK_NAME}" "--prop:ip0=${PC_IP_ADDRESS}" "--prop:netmask0=${PC_NETMASK}" "--prop:gateway=${PC_GATEWAY}" "--prop:DNS=${PC_DNS_SERVER}" "--prop:ntp_servers=${PC_NTP_SERVER}" "--prop:admin_password=${PHOTON_CONTROLLER_PASSWORD}" "${PHOTON_CONTROLLER_OVA}" "vi://${ESXI_USERNAME}:${ESXI_PASSWORD}@${ESXI_HOST}"

Slick way of deploying OVF/OVA directly to ESXi & vCenter Server using govc CLI

04/08/2016 by William Lam 9 Comments

I have been meaning to write about this neat little feature that was added to the govc CLI late last year that allows you to easily deploy any OVF/OVA without the need of ovftool. You might ask, why not use ovftool? Well, if you just need to perform a very basic OVF/OVA deploy and prefer not to install anything on your desktop, this can be a nice alternative. govc is provided as a simple binary that is platform agnostic and supports Windows, Linux & Mac OS X. govc is built using govmomi which is also known as the vSphere SDK for Go and this also means you can consume this capablitity beyond just the CLI but also programmatically if you wish. Obviously, the CLI is the easiest method which I will demonstrate below.

Just to be clear, there is still a huge amount of value in using ovftool as it contains a large mount of functionality that is not found any where else. It is still the recommended tool for deploying OVF/OVA across all VMware based Hypervisors and is extensively used by other VMware's products for general OVF/OVA deployment.

Here are the steps on using govc to deploy an OVF/OVA:

Step 1 - Download the latest govc binary for your platform which can be found here. In this example, I will be running govc on my Mac OS X system and you can easily download it by running the following commands:

curl -L https://github.com/vmware/govmomi/releases/download/v0.5.0/govc_darwin_amd64.gz | gunzip -d > /usr/local/bin/govc
chmod +x /usr/local/bin/govc

Step 2 - To verify that govc is working, go ahead and run the following command to retrieve the version which should display v0.5.0:

govc version

Note: If you are new to govc and want a quick primer on how it works, please head over to this article here for more details.

Step 3 - Next, we need to setup a couple of GOVC environmental variables which will define the vSphere environment (ESXi or VC) you plan to deploy to. You also have the option to specify this in the command-line using the --help option for more details. In this example, I will just set them as environmental variables. Below is an example of my environment which is going to be deploying directly to an ESXi host, you will want to update these variables based on your environment.

export GOVC_INSECURE=1
export GOVC_URL=nuc.primp-industries.com
export GOVC_USERNAME=root
export GOVC_PASSWORD=vmware123
export GOVC_DATASTORE=vsanDatastore
export GOVC_NETWORK="VM Network"
export GOVC_RESOURCE_POOL='*/Resources'

Note: If you are new to govc and want a quick primer on how it works, please head over to this article here for more details.

Step 4 - To ensure that govc can properly communicate with your vSphere enviornment, you can run the following command which should return details about the system you are connected to:

govc about

Step 5 - To deploy an OVF/OVA, we first need to create a spec file by inspecting the actual OVF/OVA using the import.spec command which will output in a JSON format. To make it easier to read, we will format the output using python. To do so, run the following command (replace with the OVF/OVA you wish to deploy):

govc import.spec /Volumes/Storage/Images/Current/VMware-VCSA-all-6.0.0-3634788/vcsa/vmware-vcsa | python -m json.tool

As you can see from the screenshot above, we are inspecting the OVA for the latest vCenter Server Appliance (VCSA) 6.0 and it has automatically generated all the OVF properties that can be specified for this OVA. We then just need to re-direct the output to a file in which we can then edit the properties and pass that configuration when we deploy. We can do so by running the following command:

govc import.spec /Volumes/Storage/Images/Current/VMware-VCSA-all-6.0.0-3634788/vcsa/vmware-vcsa | python -m json.tool > vcsa.json

Step 6 - Now, we just need to open up the JSON file we saved and edit the values before we can use it in our deployment. You may have also noticed that there is an InjectOvfEnv param which means you can use govc to also inject OVF properties when deploying directly to ESXi host 🙂

Here is an example of the vcsa.json for deploying an Embedded VCSA:

Note: There is a known bug currently in which the "Deployment" pararm is always generated. You will need to delete this if you have an OVA that does not use this parameter. I have already been reported here.

Step 7 - Finally, we just need to use the import.ova command and specify the path to our JSON configuration file as well as the path to our OVA. To do so, run the following command:

govc import.ova -options=vcsa.json /Volumes/Storage/Images/Current/VMware-VCSA-all-6.0.0-3634788/vcsa/vmware-vcsa

Note
Note: There is known bug in which you will see a warning about Invalid value for the network which you can ignore. I have already reported the issue here.

Once the VM powers on, the OVF properties will automatically be injected into the VM and in a few minutes, you will have a fully configured VM. Pretty cool right!? What I really love about this solution beyond its simplicity is that the tool allows you to easily generate the OVF property spec file. This is much easier than having to manually inspect an OVF/OVA and then extract the params by hand which can be very error prone. Lastly, I want to give a big shoutout to Pieter Noordhuis who was the Engineer who had implemented this awesome feature and is also one of the Engineers who created govmomi. Thanks Peter!

Hidden OVF 2.0 capablity found in the vSphere Content Library

01/12/2016 by William Lam Leave a Comment

There are a number of new and useful capabilities that have been introduced in the OVF 2.0 specification. One such capability which I thought was really interesting and that could easily benefit VMware-based solutions is the ScaleOutSection feature. This feature allows you specify the number of instances of a given Virtual Appliance to instantiate at deployment time by making use of pre-defined OVF Deployment Options which can also be overriden by a user.

Lets use an example to see how this actually works. Say you have a single Virtual Appliance (VA) and the application within the appliance can scale to N, where N is any number greater or equal to 1. If you wanted to deploy 3 instances of this VA, you would have to deploy it 3 separate times by either by running through an OVF upload or deploying it from a template. In either case, you are performing N-instantiations. Would it not be cool if you could still start with a single VA image and specify at deployment time the number of instances you want to deploy and only need to upload the VA just once? Well, that is exactly what the OVF ScaleOutSection feature provides.

Below is a diagram to help illustrate this feature further. We start out with our single VA, which contains several pre-defined Deployment Options which can contain any text you wish for the logical grouping. In this example, I am using the terms "Single", "Minimal" and "Typical" to map to number of VA's to deploy which are 1, 3, and 4 respectively. If we choose the "Minimal" Deployment option, we would then get 3 instantiated VA's. If we decide that the defaults are not sufficient, we could also override the default by specifying a different number which the VA supports.

A really cool use case that I had thought about when I first came across the ScaleOutSection feature was to make use of it with my Nested ESXi Virtual Appliance. This capability would make it even easier to standup a vSphere or VSAN Cluster of any size for development or testing purposes. Today, vSphere and many of the other VMware products only support OVF 1.x specification and as far as I know, OVF 2.0 was not something that was being looked at.

Right before holiday break, I was chatting with one of the Engineers in the Content Library team and one of the topics that I had discussed in passing was OVF 2.0 support. It turns out that, although vSphere itself does not support OVF 2.0, the vSphere Content Library feature actually contains a very basic implementation of OVF 2.0 and though not complete, it does have some support for the ScaleOutSection feature.

This of course got me thinking and with the help of the Engineer, I was able to build a prototype version of my Nested ESXi Virtual Appliance supporting the ScaleOutSection feature. Below is a quick video that demonstrates how this feature would work using a current release of vSphere 6.0. Pretty cool if you ask me!? 🙂

Demo of Prototype Nested ESXi Virtual Appliance using OVF 2.0 ScaleOut from lamw on Vimeo.

Now, before you get too excited. There were a couple of caveats that I found while going through the deployment workflow. During the deployment, the VMDKs were not properly being processed and when you power on the VMs, it was as if they were empty disks. This was a known issue and I have been told this has already been resolved in a future update. The other bigger issue is how OVF properties are handled with multiple instances of the VA. Since this is not a supported workflow, the OVF wizard is only brought up once regardless of the number of instances being deployed. This means that all VAs will inherit the same OVF values since are you are only prompted once. The workaround was to deploy the VAs, then go into each individual VA and update their OVF properties before powering on the VMs. Since OVF 2.0 and the ScaleOutSection feature is not an officially supported feature, the user experience is not as ideal as one would expect.

I personally think there are some pretty interesting use cases that could be enabled by OVF 2.0 and ScaleOutSection feature. A few VMware specific solutions that I can think of off the top of my head that could potentially leverage this capability are vRealize Log Insight, vRealize Operations Manager and vRealize Automation Center to just name a few. I am sure there are others including 3rd party and custom Virtual Appliances that have been developed and I am curious to hear if this is something that might be of interest to you? If you have any feedback, feel free to leave a comment and I can share this with the Content Library PM.

How to bootstrap the VCSA using the ESXi Embedded Host Client?

12/22/2015 by William Lam 5 Comments

In the past, I have written about various ways of "bootstrapping" vCenter Server (here and here) which can be useful for setting up greenfield vSphere deployments. This topic has always been of interest to me because it can be the most challenging to solve, especially when you only start out with a single ESXi host. Historically, these "bootstrapping" options have mostly been driven from a CLI standpoint which is not a bad thing when you think about it from an Automation standpoint and needing to replicate this a few dozen times. However, from a user experience point of view, it may not be as ideal, especially if this is an infrequent task. One of the other cool features of the ESXi Embedded Host Client (EHC) which recently had its v4 release is that it can be used to deploy Virtual Appliances stored in the OVF format.

OVF deployment had been around since v2 of EHC if I recall correctly, but it did not support all the different types of OVF capabilities such as Deployment Option as one example until most recently. One of the difficulties with OVF support on ESXi is not just supporting the ability to import/export but it is also supporting the full OVF specification which ESXi does not currently support today. This means, to provide full OVF support through EHC, it would have to implement a similar capability to what ovftool does today with the --injectOvfEnv option. Luckily, this was something that was added very early on based on my feedback which in my opinion is critical when it comes to greenfield deployments and one of the core use cases that I see for EHC.

Without further ado, below are the instructions on how to bootstrap the vCenter Server Appliance (VCSA) using the Embedded Host Client.

Step 1 - Download the VCSA OVA and then convert it to an OVF using either 7zip or ovftool. The reason you need to convert it to an OVF is that there is currently a known issue when trying to extract larger OVAs within the EHC. By doing this, you also speed up the time it takes to perform the upload and not have to wait for UI to extract it into the actual consumable format which is the OVF and VMDKs.

Step 2 - Click on the "Create/Register VM" option and then choose "Deploy a virtual machine from an OVF or OVA file" option. You will need to specify the name for your VCSA along with the OVF and the 3 VMDKs that is included if you are using vSphere 6.0/6.0 Update 1.

Step 3 - Next you will configure the VM Network and the disk provisioning option. You will also be asked to select the "Deployment Type" which is an option in vSphere 6.0/6.0 Update 1 that allows you to specify whether you are deploying an Embedded VCSA, External vCenter Server or External Platform Services Controller (PSC). You may notice the drop down includes duplicate entries and the reason for this is how the VCSA OVA was built which re-uses the same description in each of the labels but they actually have different meanings. Below is a quick table of the correct mappings to the current ordering parsed by EHC to the different VCSA Deployment Types:

Label	Actual Deployment Type
Tiny (up to 10 hosts 100 VMs)	Embedded VCSA Node
Small (up to 100 hosts 1K VMs)	Embedded VCSA Node
Medium (up to 400 hosts 4K VMs)	Embedded VCSA Node
Large (up to 1K hosts 10K VMs)	Embedded VCSA Node
Tiny (up to 10 hosts 100 VMs)	External vCenter Server Node
Small (up to 100 hosts 1K VMs)	External vCenter Server Node
Medium (up to 400 hosts 4K VMs)	External vCenter Server Node
Large (up to 1K hosts 10K VMs)	External vCenter Server Node

Step 4 - You will need to fill out the OVF properties which are required to properly configure the VCSA. The following 3 sections are the ONLY ones you need to modify for install:

Networking Configuration
SSO Configuration
System Configuration

Step 4a - The Networking Configuration section will require you to specify the following:

Host Network IP Address Family - ipv4 or ipv6
Host Network Mode - static or dhcp
Host Network IP Address - IP Address of the VCSA
Host Network Prefix - This is the CIDR notation of the network you plan to place the VCSA on. Example would be /24 (255.255.255.0) which you need to specify as just 24
Host Network Default Gateway - Gateway to use
Host Network DNS Servers - DNS Server to use
Host Network Identity (optional) - This is the FQDN of the VCSA. If you are in a DHCP enabled environment, you can leave this blank which it will automatically default to localhost.localdomain

Step 4b - The SSO Configuration section will require you to specify the following:

Directory Password - SSO Admin password
Directory Password confirm - SSO Admin password
Directory Domain Name - SSO Domain (select vsphere.local if you want the default which I would recommend)
Site Name - SSO Site Name
New Identity Domain - This is required if it is a new setup and you are not setting up SSO replication with an existing PSC

Step 4c - The System Configuration section will require you to specify the following:

Root Password - The root password for the OS
Root Password confirm - The root password for the OS
SSH Enabled (option) - If you wish for SSH to be enabled after deployment
Tools-based Time Sync Enabled - If you do not have an NTP server, you should select this option
NTP Servers (optional) - Specify a valid NTP server that you wish to use

Step 5 - You will have the ability to review your configurations before starting the deployment. You should double check to ensure that all OVF properties are correct, else you may get a failed deployment.

Step 6 - Once you are ready, go ahead and click the "Finish" button. This will start the OVF import which you can monitor using the Recent Tasks pane. As part of the OVF/OVA workflow, once the import has completed, it will automatically power on the VM for you. Please do NOT interrupt this process as EHC will be injecting the OVF properties you had specified earlier to the VM to ensure the VCSA will be properly configured.

Once the VM has been powered on, you can then click into the VM Console to view the status of the deployment and hopefully in a couple of minutes, you will have a fully configured and functional VCSA ready for use!

Although EHC today has implemented a pretty generic OVF/OVA interface in the UI to support almost any OVF/OVA, you can see how this could be further improved specifically for the VCSA deployments from a user experience perspective. Who knows, this might get even easier in the future 🙂