virtuallyGhetto

unmap

Configure new automatic Space Reclamation (VMFS UNMAP) using vSphere 6.5 APIs

by 5 Comments

Since its first introduction in vSphere 5.5, VMFS UNMAP also know as Space Reclamation for a VMFS based datastore has been a pretty popular Storage capability in vSphere. A commonly asked question from customers is when will the "automatic" capability return? Well, it looks like it is now back in the upcoming vSphere 6.5 release as blogged about here by Duncan Epping. Below is a screenshot of where you can find the setting. VMFS UNMAP is now enabled by default and you will need to have a VMFS 6 datastore to take advantage of this new feature.

vmfs-unmap-vsphere-65-api-0
For customers who wish to automate the configuration of the VMFS UNAMP capability whether that is to check the current settings or to enable/disable it, there are some new vSphere 6.5 APIs that have been introduced which differ from the previous implementations. To change the VMFS UNMAP setting, there is a new vSphere API called UpdateVmfsUnmapPriority() which accepts the UUID of a VMFS 6 datastore as well as an unmapPriority property which can either be "low" which means it is enabled or "none" which means it is disabled. To view the current VMFS UNMAP settings, there is a new property under the Datastore->Info->Vmfs object called UnmapPriority.

To demonstrate this new vSphere API, I have created two small PowerCLI functions called Get-VMFSUnmap and Set-VMFSUnmap which can be downloaded from here.

Here is an example of retrieving the current VMFS UNMAP settings:

Get-Datastore "mini-local-datastore-hdd" | Get-VMFSUnmap

vmfs-unmap-vsphere-65-api-1
Here is an example of enabling automatic VMFS UNMAP setting:

Get-Datastore "mini-local-datastore-hdd" | Set-VMFSUnmap -Enabled $true

vmfs-unmap-vsphere-65-api-2

Using latest PowerActions 1.5.0 to issue VMFS UNMAP API in vSphere 6.0 Web Client

by 4 Comments

Last week, the popular PowerActions Fling was updated to version 1.5.0 which now finally adds support for vSphere 6.0. PowerActions is a vSphere Web Client Plugin that allows administrators to easily execute PowerCLI scripts against inventory objects within the vSphere Web Client. This is a very powerful capability that PowerActions is providing and allows users to easily extend new and custom operations that may not be available within the vSphere Web Client. One such example is being able to easily issue a VMFS UNAMP which in vSphere 5.5 was only available through the use of ESXCLI, I actually demonstrated how easy it is to provide this capability using PowerActions which you can read more about here.

With the release of vSphere 6.0, we now have the ability to issue a VMFS UNMAP using the vSphere API which I have blogged about here among other new vSphere 6.0 APIs. Given that PowerActions now supports vSphere 6.0, I figured this would be a good opportunity to take advantage of the new vSphere 6.0 API using the updated version of PowerActions. I have created a new PowerCLI script called Issue UNMAP 2.0 on Datastore.ps1 which now uses the new UnmapVmfsVolumeEx_Task vSphere API to perform the VMFS UNMAP. I have also submitted a new pull request for this example to Alan Renouf's PowerActions Github repository.

Here is a screenshot of my running the new VMFS UNMAP PowerActions operation against one of my vSphere Datastores and you can that it successfully completed in the Recent Tasks bar.

poweractions-vmfs-unmap
In addition to the new VMFS UNAMP operation that can be added as a PowerAction, here are just a few other examples of new vSphere 6.0 capabilities that you can easily extend into a PowerAction:

I am personally excited for the future and potential of PowerActions and I hope to see the PowerActions framework extend beyond just PowerCLI and support other scripting languages. I think this will be a very powerful capability that the vSphere Web Client can offer to our administrators, operators and developers.

Want to issue a VAAI UNMAP operation using the vSphere Web Client?

by 3 Comments

Recently, I have seen several requests from both customers and partners wanting to be able to run the VAAI UNMAP operation from within the vSphere Web Client. For those of you not familiar with the VAAI UNMAP operation, I recommend you check out this blog post by my colleague Cormac Hogan. Today, the only way to issue the UNMAP operation is by using ESXCLI either remotely or in the ESXi Shell. There is currently not a vSphere API for this operation and therefore it would be difficult to build a native vSphere Web Client Plugin to provide this functionality.

Having said that, one way to provide this capability is through the use of a vCenter Orchestrator (vCO) Workflow which can remotely execute an ESXCLI command whether that is going through ESXCLI using a Linux jump box or through PowerCLI using a Windows jump box. Starting with vSphere 5.5, you can now extend the vSphere Web Client and attach a vCO Workflow to a vSphere Object and be able to execute the workflow right from the vSphere Web Client. This is great if you are already using vCO, but for those that are not, it can be somewhat complex to setup along with a steep learning curve depending on your experience.

Today, there was an exciting announcement from my Automation buddy, Alan Renouf for a new VMware Fling called PowerActions for the vSphere Web Client. This new Fling allows you to easily extend the vSphere Web Client in the following ways:

  • Access a PowerCLI console directly in the vSphere Web Client
  • Ability to run a context aware PowerCLI script directly from the vSphere Web Client

The prerequisite for setting up PowerActions is no different than vCO calling a PowerCLI script, you just need a Windows "jump-box" that has PowerCLI installed along with PowerActions. The added benefit, is that you do need to setup another piece of infrastructure like vCO if you are not already using it. This made setting up PowerActions extremely easy to setup and even I was able get it up and running in under 5minutes (minus a quick RTFM moment :)).

Given the number of inquiries regarding VAAI UNMAP operation via the vSphere Web Client, I thought that would be a great use case for my first PowerActions script! Below are the instructions on creating the VAAI UNMAP script for PowerActions:

Step 1 - Click on the PowerCLI Scripts option on the left hand side of the Object Navigator and then click on the "New Script" Icon. Select Datastore as the context aware object for the script.

unmap-command-in-vsphere-web-client-0
Step 2 - Provide a name and description for the script. Also make sure to select "Action".

unmap-command-in-vsphere-web-client-1
Step 3 - Copy and paste the following script from https://github.com/lamw/vghetto-scripts/blob/master/powershell/unmap-poweraction.ps1 inside the script window and then save the script. What the script does is takes the Datastore object and retrieves a list of ESXi hosts that has access to the Datastore and then randomly selects one of the host. This is required because ESXCLI operations on a per host level and we use that information to pass into Get-EsxCli cmdlet to issue the VAAI UNMAP operation.

Step 4 - To test the script, you just need to right click on a VMFS Datastore and click on PowerCLI->Execute a Script

unmap-command-in-vsphere-web-client-2
Note: Please be aware of the impact when running a UNMAP operation, you may want to run this on a non-production datastore for testing purposes or during off hours when your workload may not be as busy.

Step 5 - Select the VAAI UNMAP script you just created and once selected and you will be prompted to specify the number of VMFS blocks to unmap per iteration which is exactly the same input when manually ESXCLI.

Screen Shot 2014-09-17 at 10.30.09 PM
At this point, if everything was successful the VAAI UNMAP operation should begin and you can tail /var/log/hostd.log to see the UNAMP operation. Once completed, you should see the prompt return true.

As you can see, it was extremely easy to create my own PowerAction script that expose new functionality and making it available within the vSphere Web Client. I think this is going to be a pretty popular Fling and remember if this is something you would like to see officially in the product, be sure to leave a comment on the PowerAction for vSphere Web Client Fling page, product managers are listening! The only feedback I have is that I would love to see this get extended beyond just PowerCLI and into a generic script extension framework, just imagine the possibilities!

How To Initiate a Wipe & Shrink Operation On an SE Sparse Based Disk

by 6 Comments

In my previous two articles, I showed you how to create your own SE Sparse disks as well as creating new virtual machine Linked Clones leveraging the new SE Sparse disk format. If you recall earlier, one of the features of the SE Sparse disk format is to provide the ability to reclaim unused blocks within the guestOS which is a two step process: wipe and shrink.

Here is a screenshot that describes the process which was taken from the What's New In vSphere 5.1 Storage Whitepaper by my colleague Cormac Hogan. I highly recommend you check out the whitepaper which includes more details about this feature and other storage improvements in vSphere 5.1

The process of kicking off this wipe and shrink operation will be done through an integration with VMware View (a future release from my understanding). Now, it's important to understand that it's not just simply calling these two operations but it is also when they are called. The wipe operation is more CPU intensive as it scans for unused space within the guestOS filesystem and the shrink operation is more I/O intensive as it issues the SCSI unmaps commands. I can only assume that these operations will be scheduled based on the utilization of the guestOS to help reduce the impact to the VM workload.

Now having said that, since the SE Sparse disk format is a feature of the vSphere 5.1 platform, so are both the wipe and shrink operations. Though they are not exposed in the public vSphere API like the SE Sparse disk format, you can still access the private APIs if you know where to look 😉

Disclaimer: This is for educational purposes only, this is not officially supported by VMware. Use at your own risk.

With some help from my good friend the vSphere MOB and some digging, I have located the two vSphere API methods for wipe and shrink operation. Before getting started, ensure you have a VM with at least one SE Sparse disk, else these commands will not be very useful.

Note: In this experiment, I tested the wipe and shrink operation with Windows XP image, this may or may not work on other OSes.

First you will need to search for the VM in question and retrieve it's vSphere MOB URL which is in the format of https://[vcenter-server]/mob/?moid=vm-X where X is the MoRef ID for your VM. You can either navigate through the vSphere MOB or use my MoRef finder script.

Wipe Operation

To issue the wipe API, enter the following URL into your web browser (remember to replace the MoRef ID with the one of your VM)

https://[vcenter-server]/mob/?moid=vm-X&method=wipeDisk

Here is a screenshot of what that looks like if you are able to successfully access the private API:

Go ahead and click on "Invoke Method" which will then execute the wipe operation. If you take a look at the vSphere Web Client, you should now see a new task for the wipe operation.

This can take a bit of time as it scans through the guestOS filesystem for unused space.

Shrink Operation

Once the wipe operation as completed, we then need to issue the shrink API. Enter the following URL into your web browser (remember to replace the MoRef ID with tone of your VM)

https://[vcenter-server]/mob/?moid=vm-X&method=shrinkDisk

Here is a screenshot of what that looks like if you are able to successfully access the private API:

Here you can specify particular disks (requires diskId) that you wish to perform the shrink operation on. If you leave it blank, it will try to shrink all disks associated with your VM. In our example, I will shrink all disks. Go ahead and click on the "Invoke Method" which will kick off the shrink operation. If you go back to the vSphere Web Client, you should now see a shrink task in progress.

Again, this operation can also take some time, but once it has finished, then you have successfully reclaimed any unused blocks within your guestOS.

Automating Dead Space Reclamation in ESXi 5.0u1

by 4 Comments

VMware released vSphere 5.0 Update 1 last week, which mainly included bug fixes but it also brought back one very cool feature that was initially introduced with the release of vSphere 5.0 called Thin Provisioning UNMAP primitive for an ESXi host. You can read more about the details in this article by my colleague Cormac Hogan.

As you can see from From Cormac's article, the process of reclaiming of dead space on a thin provisioned LUN is currently a manual process, but does it have to be? The answer is, No of course, we can can definitely automate this!

Disclaimer: This script is not officially supported by VMware, please test this in a development environment before using on production system. This script is provided as an example on you can automate this manual process.

Before you proceed, please understand that the UNMAP operation can potentially take a few minutes up to a few hours depending on the size of your datastore and how your array handles this operation. You should consider performing this operation during a maintenance window or during off peak hours else you could impact VMs residing on the datastore. You should also ensure you have a VAAI-capable storage array before performing running this script.

I wrote a simple shell script called reclaimMyDeadsSace.sh which needs to be executed on the ESXi Shell via SSH. The script will also perform some validation such as ensuring you are running ESXi 5.0 Update 1 and that your host is in maintenance mode as a per-caution to ensure no running VMs are on the host during this process.

You will only need to run the script on one of the hosts connected to all the datastores you wish to reclaim dead space on. You may use scp or WinSCP to transfer the script to your ESXi host and ensure you set the execute permission on the script (chmod +x reclaimMyDeadSpace.sh)

The script can be executed in two ways:

  1. Identify ALL VMFS3 and VMFS5 volumes and perform the reclaim based on the percentage entered by the user
  2. Reclaim on specific datastores specified by the user as well as the percentage to be reclaimed (this is recommended, that way script does not choose all datastores including local ones)

Here is an example of selecting ALL VMFS3 and VMFS5 datastores to reclaim 60% of free space:

Here is an example of selecting just 4 datastores specified in a file and we will be reclaiming 60% of free space:

In this example, we created a file called "datastore_list.txt" (you may name the file anything you want) which contains the following:
iSCSI-1
iSCSI-2
iSCSI-3
iSCSI-4

So if you are using thin provisioned LUNs and would like to reclaim some of that dead space back and have a VAAI-capable storage array, be sure to check out the UNMAP functionality in ESXi 5.0u1.

How to Automate the disabling of the VAAI UNMAP primitve in ESXi 5

by 5 Comments

I just saw an interesting article on Jason Boche's blog about VMware's recall of the new VAAI UNMAP primitive for vSphere 5. VMware released a new KB article KB2007427 documenting the details along with a recommendation to disable the VAAI UNMAP primitive for now (a patch will be released in the future to automatically disable this until the issue is resolved).

One thing that caught my in the VMware KB article is to disable the VAAI UNMAP primitive, you need to manually login to ESXi 5 Tech Support Mode (SSH) to run the local version of esxcli command. This is trivial if you have several hosts, but it can be time consuming if you have several hundred hosts to manage. Even though the "VMFS3.EnableBlockDelete" is a hidden parameter that can not be seen using any of the supported utilities, you can enable and disable the property using the remote version of esxcli which is part of vMA 5 or vCLI 5.

Here is an example of the command when connecting directly to an ESXi 5 host:

esxcli --server himalaya.primp-industries.com --username root system settings advanced set --int-value 1 --option /VMFS3/EnableBlockDelete

Here is an example of command when connecting to vCenter Server:

esxcli --server reflex.primp-industries.com --vihost himalaya.primp-industries.com --username administrator system settings advanced set --int-value 1 --option /VMFS3/EnableBlockDelete

As you can see, you can easily wrap this in a simple for-loop to disable the VAAI UNMAP primitive. So here is a script to help with exactly that called vaaiUNMAP.sh

The script accepts 4 parameters:

  1. A list of ESXi 5 hosts to enable or disable VAAI UNMAP primitive
  2. Name of the vCenter Server managing the ESXi 5 hosts
  3. vCenter auth file which contains the username/password
  4. 0 to disable or 1 to enable VAAI UNMAP primitive

The auth file is just a file that contains the following:

VI_USERNAME=administrator
VI_PASSWORD=y0mysuperdupersecurepassword

Here is an example of disabling VAAI UNMAP primitive on 3 ESXi hosts being managed by a vCenter Server:

To help extract all ESXi 5 hosts from your vCenter Server, you can use the following vSphere SDK for Perl getESXi5Hosts.pl

Here is an example of running the script and just save the output to a file:

One of the unfortunate thing about the VMFS3.EnableBlockDelete is that it is a hidden parameter along with others, so it will not automatically display when using local or remote ESXCLI, but thanks to Craig Risinger, you can still get the information using the remote ESXCLI by specifying the --option parameter which is great because you do not need to login to the ESXi Shell to retrieve the information.

esxcli --server himalaya.primp-industries.com --username root system settings advanced list --option /VMFS3/EnableBlockDelete

I would also recommend adding an entry into your ESXi 5 kickstart to automatically disable VAAI UNMAP by default until a fix is released.