virtuallyGhetto

api

How to Create an SE Sparse (Space-Efficient) Disk in vSphere 5.1

by 8 Comments

You probably may have heard, that with the upcoming release of vSphere 5.1, a new virtual machine disk format will be introduced called called SE Sparse (Space-Efficient). One of it's features is to provide the ability to reclaim unused blocks from within the guestOS. I would highly recommend you check out a recent blog post vSphere 5.1 Storage Enhancements – Part 2: SE Sparse Disks by Cormac Hogan for more details about the new SE Sparse disk format as well as other storage improvements in vSphere 5.1.

As Cormac points out, this new disk format will initially be leveraged by VMware View (in a future release from my understanding), as there are additional integrations required to use this feature than just using the new SE Sparse disk format. Having said that, the SE Sparse disk format is a feature of the vSphere 5.1 platform and with that, you do have the ability to create an SE Sparse disk.

Disclaimer: This is for educational purposes only, this is not officially supported by VMware. Please test this in a development environment before using it on actual systems.

There are two methods in which you can create an SE Sparse disk, directly on the ESXi Shell of an ESXi 5.1 host or remotely connecting to an ESXi 5.1 host.

Option 1 - Using vmkfstools on ESXi Shell 

Though it may not be documented, you can easily create a new VMDK with the new SE Sparse disk format by running the following command (10GB disk in this example):

vmkfstools -c 10g -d sesparse WindowsXP.vmdk

Here is a screenshot of new SE Sparse disk descriptor file to prove we have successfully created a new VMDK using the new format:

Option 2 - Using vSphere 5.1 API w/modified remote version of vmkfstools

As mentioned, the SE Sparse disk format is a feature of the vSphere 5.1 platform and as so, you can also leverage the vSphere 5.1 API to create a new VMDK using the virtualDiskManager and specifying the new SeSparseVirtualDiskSpec.

Note: Even though the vSphere API reference mentions the ability to set grain size via grainSizeKb property, I have found that it is not possible and just leaving it blank will automatically default to 1024K (1MB) which might be a system default for now.

You can download the modified version of the remote vmkfstools called vmkfstools-lamw which requires the the installation of vCLI 5.1 or vMA 5.1.

Here is an example of creating the same 10GB VMDK using the new SE Sparse disk format:

./vmkfstools-lamw --server 172.30.0.187 --username root -c 10G -d sesparse "[datastore1] WindowsXP.vmdk"

After you have created your new SE Sparse disk, the next logical step is assign it to a virtual machine. Since this is a new feature in vSphere 5.1, you will need to use the new vSphere Web Client to perform the operation as the legacy vSphere C# Client is not aware of this new disk type. You will also need to ensure that the virtual machine is running the latest ESXi 5.1 compatibility and later (virtual hardware version 9).

Once you have added our newly created disk from the datastore, it should now show up in the vSphere Web Client as Flex-SE for the disk type.

Additional Resources:

 

Enabling/Disabling EVC using the vSphere MOB

by 2 Comments

There were some discussions this morning on twitter regarding the configuration of EVC for a vSphere Cluster using one of the vSphere CLI's such as PowerCLI or directly leveraging the vSphere API. Unfortunately, this is not possible today as the operations pertaining to EVC are not currently exposed in the vSphere API. This means you will not be able to use the vCLI, PowerCLI, vCO or the vSphere API to configure and manage EVC configurations, you will need to use the vSphere Client to do so.

Having said that, one could still "potentially" automate EVC configurations using the vSphere MOB interface using the private vSphere API, but it may not be ideal and will require some "creativity" and custom coding to integrate with your existing automation solution. This particular limitation of the vSphere API is one that I have personally faced and have filed a bug with VMware awhile back. I am hoping this will eventually be added to the public vSphere API, so that users can fully automate all aspects and configurations of a vSphere Cluster.

Disclaimer: This is not officially supported by VMware, use at your own risk and discretion.

Step 1 - Connect to your vCenter MOB and traverse to the vSphere Cluster of interest (note the MOID will be different in your specific cluster).

Step 2 -  Now replace the URL with the following while substituting the cluster MOID that you see in your browser:

https://reflex.primp-industries.com/mob/?moid=domain-c1550&method=transitionalEVCManager

and hit enter and you'll be brought to TransitionalEVCManager() method, you'll then want to click on the "Invoke Method". Once you do so, you should be returned with a task object and you'll have a link to something like evcdomain-cXXXX. Click on this and you'll be brought to ClusterTransitionalEVCManager.

Step 3 - From here you'll have have some basic evcState information which you can click on to see what the current EVC configuration is set to, guaranteedCPUFeatures and valid EVC Modes (the last part will be important for reconfiguring EVC)

Step 4 - Now let's say the cluster currently has EVC Mode set to intel-merom and you would like to change it to Nehalem, you would need to retrieve the key from the previous page, in our example it's intel-nehalem. Now, you need to click on the method link called ConfigureEVC_Task which is pretty straight forward, it just accepts the EVC Mode Key, enter the string and click on "Invoke Method" and now your cluster will be reconfigured if you go back to the evcState or look at your vCenter task. You can also disable EVC by using DisableEVC_Task

 
Note: If EVC is already configured in your vSphere Cluster, you can use the vSphere API to view it's current configuration by looking at the ClusterComputeResource's summary property. You just will not be able to make any changes or disabling EVC using the vSphere API.

VM Provisioning on Datastore Clusters in vSphere 5

by 23 Comments

Last year I wrote an article called Automating Storage DRS & Datastore Cluster Management in vSphere 5 and I provided a pretty comprehensive vSphere SDK for Perl script to help administrators automate Storage DRS configurations. These past few months I have noticed an increase in interest on the VMTN developer forums relating to Storage DRS. Majority of the questions has been related to which vSphere API methods to use and how to use these methods for cloning VMs to datastore clusters.

If you have cloned a VM before, the underlying vSphere API method being used is the CloneVM_Task(), but when cloning a VM to a datastore cluster, a different API must be used.

In vSphere 5, VMware introduced several new API methods in the StorageResourceManager and the two specific ones relating to provisioning VMs are RecommendDatastores() and ApplyStorageDrsRecommendation_Task(). The process to clone a VM to a datastore cluster is a two step process:

  1. Call RecommendDatastores() which accepts very similar input as the CloneVM_Task(). In addition, you need to specify the datastore cluster also known as a Storage Pod in the vSphere API as well as the "type" (create, clone, relocate, reconfigure), in this example we will be performing a clone operation. This method will then generate a recommendation on where to place the VM which is based on the SDRS placement engine. No provisioning will occur at this point, just a placement recommendation.
  2. To perform the actual provision of the VM, you will need to call ApplyStorageDrsRecommendation_Task() which accepts a recommendation ID that was generated from the first step. Once the recommendation is applied, the provisioning of the VM will start just like it did when you called the CloneVM_Task().

Note: The RecommendDatastores() will return multiple recommendations, the best one will be first entry in the array. This is the same algorithm used when performing this same operation in the vSphere Client, it also selects the first recommendation.

Now that we understand how the APIs work, let's take a look at how we can leverage this in a script for some automation! Here is a simple vSphere SDK for Perl script called datastoreClusterVMProvisioning.pl which allows you to clone an existing VM onto a datastore cluster. You will need a system that has vCLI 5.0 installed or you can use VMware vMA 5 to run the script. You will also need to connect to a vCenter Server 5 for all SDRS operations.

The script requires 4 input parameters:

  • vmname - Name of the VM you wish to clone from
  • clonename - Name of the cloned VM
  • vmfolder - Name of a vCenter folder
  • datastorecluster - Name of a datastore cluster

Here is a screenshot of cloning an existing VM onto a datastore cluster:

The script is pretty straight forward and it can easily be adapted to include other configurations as required in your own environment.

Hopefully this gives you a better idea on how to leverage the new provisioning APIs for Storage DRS and start automating your VM deployments onto datastore clusters and get the benefits Storage DRS in your vSphere environment.

How to Access vCloud Director Remote Console using vCloud & VMRC API

by 17 Comments

If you are familiar with the vCloud Director UI and have used the virtual console for a virtual machine, you may have noticed it is serviced by the VMRC (Virtual Machine Remote Console) vCD browser plugin which is only accessible in the vCD UI. If you are building a custom provisioning portal, you may want to provide similar functionality in your own portal for your users to access their VMs.

You can do so by leveraging the vCloud API and the new VMRC API that was made available with the release of vCloud Director 1.5.

In addition to the VMRC API documentation, there is an sample VMRC API Reference Implementation that you can download and experiment with. When you extract the contents of the zip file, you will find an index.html and console.html web page and you can use this locally on your system or host it on a web server.

The index.html is the landing page that you will use to provide the vCloud VM's screen ticket using the vCloud API's AcquireTicket REST API method. The console.html will then load the VMRC for the requested VM assuming you provided a valid screen ticket.

Here are the four simple steps to access a vCloud VM's remote console using the vCloud REST API & VMRC API using curl:

Step 1 - Login

You will need a system that has curl installed and you will need an account in vCD that has access to some powered on VMs. In the example below, I will be logging into a regular organization and not the System organization. There are a few parameters you need to specify to login to vCD and obtain an authorization token. You will need to specify the following parameters and the URL to your vCloud Director instance which should be in the form of https://vcd-fqdn/api/sessions:

  • -i = Include headers
  • -k = Performs an "insecure" SSL connection
  • -H = Setting the header for the version of vCloud Director (1.5 in this example)
  • -u = User credentials in the format of [[email protected]:password]
  • -X = Request type

Note: For more details on the cURL flags, please refer to the cURL documentation.

curl -i -k -H "Accept:application/*+xml;version=1.5" -u [email protected]:vmware -X POST https://vcd.primp-industries.com/api/sessions

If you have successfully logged in, you should get an HTTP 200 response and get similar output as above. You will need to make note of your authorization token which is located on the third line that starts with "x-vcloud-authorization". This will be needed throughout the remainder of the session

Step 2 - Query All Powered On VMs

Using the new Query Service API in vCloud 1.5, we will locate all VMs within the vCD instance with the vm type. If you are logged in as a System Administrator and using the System Org, use the adminVM type instead. You can see the command is very similar to the one above but instead of specifying the credentials, we are using the authorization token from step 1. We are also including specific fields in the XML output for readability (VM name + vApp name), if you remove '&fields=name,containerName', you will see the other fields in the XML response. We also want to filter for only powered-on VMs, we can do this by adding '&filter=status==POWERED_ON'.

curl -i -k -H "Accept:application/*+xml;version=1.5" -H "x-vcloud-authorization: n/+ZglmIwJ2SbWlyz04XC5sHUDJqefU1mCMa9TFu9lk=" -X GET "https://vcd.primp-industries.com/api/query?type=vm&filter=status==POWERED_ON&fields=name,containerName"

If the operation was successful, you should see a HTTP 200 response and list of VMs in your vCD instance. You will need to make a note of the VM's href property that you wish to obtain the remote console screen ticket for which is needed in the next step.

Step 3 - Obtain Screen Ticket for VM

Next we need to obtain the screen ticket using the AcquireTicket vCloud REST API method. You will need to perform a POST operation and using the href property from the previous step, append "/screen/action/acquireTicket". This link is only available IF the VM is powered-on and must be used within 30seconds or it will be considered invalid.

curl -i -k -H "Accept:application/*+xml;version=1.5" -H "x-vcloud-authorization: n/+ZglmIwJ2SbWlyz04XC5sHUDJqefU1mCMa9TFu9lk=" -X POST https://vcd.primp-industries.com/api/vApp/vm-25a4228e-1202-45a7-8f85-cafaeb1c3bc8/screen/action/acquireTicket

Now we just need to take the screen ticket which starts with mks://..... and copy that over to our VMRC sample web page and hit the submit button. If everything was successful, you should see the a remote console popup up for the VM you had just requested the screen ticket from. You will also notice the IP Address in the screen ticket is not the same address as your vCloud Director Cell, it is actually the console proxy address if you were wondering why the IP is different.

So this is great, but you still need some interaction with the browser to actually submit the screen ticket request. What if we just wanted to provide a single URL that would automatically launch the remote console for a requested VM? You can definitely do so, but you will need to perform a URL encoding on the mks://... ticket and construct the proper URL.

Here is a simple shell script called vcloud-vAppRemoteConsole.sh that uses the same vCloud REST API calls as above and allows a user to select the vCloud VM to to retrieve a screen ticket and returns back a single URL to access the remote console of the VM.

The script requires three parameters: ([email protected], password and vcloud-host)

Here is an sample execution:

Once you have successfully logged in, you should see a list of VMs that are currently powered on (highlighted in blue). You will need to select a VM and provide the "href" property as input (highlighted in green). Then the AcquireTicket method will be called and the screen ticket is then encoded using a quick Perl one-liner. The encoded entry is then appended to the hosted URL of the VMRC sample https://air.primp-industries.com/vmrc/console.html? (you can replace the URL to fit your environment). The output of the script (highlighted in orange) is the URL you would then paste into a supported web browser and the remote console will automatically launch without any additional user input.

With the above knowledge, you can easily translate this using any of the vCloud SDKs (Java, .NET, PHP), vCO or the new vCloud Director cmdlets and provide console access to your vCloud VMs.

Note: For vSphere virtual machine remote console access instructions, please take a look at this article here.

How to Create a vCenter Alarm to Monitor for root Logins

by 6 Comments

Another interesting question on the VMTN forums this week, a user was looking for a way to trigger a vCenter alarm when a someone would login to an ESX(i) host using the root account. By default there are several dozen pre-defined vCenter alarms that you can adjust or modify to your needs, but it does not cover every single condition/event that can be triggered via an alarm. This is where the power of the vSphere API comes in. If you browse through the available event types, you will find one that corresponds to sessions called sessionEvent and within that category of events, you will see a UserLoginSessionEvent.

Now that we have identified the particular event we are interested in, we simply just create a new custom alarm that monitors for this event and ensure that "userName" property matches "root" as the user we are trying to alarm on. I wrote a vSphere SDK for Perl script called monitorUserLoginAlarm.pl that can be used to create an alarm on any particular user login.

The script requires only two parameters: alarmname (name of the vCenter alarm) and user (username to alarm on). Here is a sample output for monitoring root user logins on an ESX(i) host:

The alarm will be created at the vCenter Server level and you should see the new alarm after executing the script.

Note: The alarm action is currently to alert within vCenter, if you would like it to perform other operations such as sending an email or an SNMP trap, you can edit the alarm after it has been created by the script.

Next it is time to test out the new alarm, if you click on the "Alarms" tab under "Triggered Alarms" and login to one of the managed ESX(i) host using a vSphere Client with the root account, you should see the new alarm trigger immediately.

If we view the "Tasks/Events" tab for more details, we can confirm the login event and that it was from someone using the root account.

As you can see even though this particular event was not available as a default selection, using the vSphere API, you can still create a custom alarm to monitor for this particular event.

I do not know what the original intent of monitoring for monitoring root logins, but if there is a fear of the root  account being used, the easiest way to prevent this is to enable vCenter Lockdown Mode for your ESXi host.

How to Query VM Disk Format in vSphere 5

by 5 Comments

Prior to vSphere 5, it was not trivial to identify the particular disk format for a given virtual machine's disk. Using the vSphere Client, you would see a virtual machine's disk be displayed as either thin or thick. The problem with this is that the "thick" format can be either:

  • zeroedthick - A thick disk has all space allocated at creation time and the space is zeroed on demand as the space is used
  • eagerzeroedthick - An eager zeroed thick disk has all space allocated and wiped clean of any previous contents on the physical media at creation time. Such disks may take longer time during creation compared to other disk formats.

Users would not be able to distinguish the exact type using the vSphere Client or the vSphere 4 APIs. With the release of vSphere 4, VMware did introduce a new property in the vSphere 4 API called eagerlyScrub which was supposed to help identify whether a virtual disk was allocated as an eagerzeroedthick disk. Unfortunately there may have been a bug with the property as it never gets modified whether a disk is created as zeroedthick or eagerzeroedthick.

The only method that I was aware of to truly figuring out the disk format would be to manually parse the virtual machine's vmware.log file to identify the disk type which I wrote a script for in 2009.

During the vSphere 5 beta, I had noticed the vSphere Client UI now properly displays all three virtual machine disk format: zeroedthick (displayed as flat), thin and eagerzeroedthick (displayed as thick).

Seeing that VMware now displays the three different formats, I wanted to see if it was possible to extract this using the vSphere 5 APIs and not have to rely on the hack of reading the vmware.log files. It turns out that the eagerlyScrub property is now functioning properly when a VMDK is provisioned or has been inflated/converted to the eagerzeroedthick format. I wrote a simple vSphere SDK for Perl script called getVMDiskFormat.pl which allows you to extract the disk formats of all virtual machines connecting to either vCenter or directly to an ESX(i) host.

The script allows for two types of output: console (directly on the console) or csv (creates .csv file)

If you select csv output, by default it will be stored in a file called "vmDiskFormat.csv". You also have the option of specifying the filename by using the --filename flag and providing a name of your choosing.

You can then load the csv file into excel and easily sort through the various disk format types.

All this is already included in the latest version of the VMware vSphere Health Check Report 5.0 if you want a centralize report that includes virtual machine disk format.