virtuallyGhetto

python

New vSAN Management 6.6 API / SDKs / CLIs

by 1 Comment

With all the new awesome capabilities that have been introduced in vSAN 6.6, there is just as much Automation goodness that will be available for our customers to consume to help them easily mange and operate at scale.

vSAN Management 6.6 API

Below are all the new Managed Objects that have been introduced in the new vSAN Management 6.6 API. This does not even cover all the new methods or object types. For the complete list of vSAN 6.6 APIs, be sure to check out the vSAN Management 6.6 API Reference Guide here.

  • VsanVcsaDeployerSystem – Virtual Center Service Appliance deployment APIs onto vSAN datastore, operating at both vCenter Server and ESXi Host sides
  • VsanVdsSystem – vSAN system optimized VDS related operations, especially migrations from VSS to VDS
  • VsanUpdateManager – VIB installation engine operating at vSAN cluster level (optimized for vSAN clusters)
  • VsanCapabilitySystem – APIs to query vSAN capability, available on both vCenter and ESXi
  • VsanMassCollector – vSAN system management query API's to access data and managed object properties, operating at a vSAN Cluster level in vCenter Server only
  • VsanPhoneHomeSystem – vSAN online health related query API, operating at a vSAN Cluster level in vCenter Server only

[Read more...] about New vSAN Management 6.6 API / SDKs / CLIs

Using the vSphere API to remotely generate ESXi performance support bundles

by 2 Comments

This is a follow-up post from my previous article on how to run a script using a vCenter Alarm action in the vCenter Server Appliance (VCSA). What I had demonstrated was using a pyvmomi (vSphere SDK for Python) script to be triggered automatically to generate a VMware Support Bundle for all ESXi hosts for a given vSphere Cluster. The one caveat that I had mentioned in the blog post was that the solution slightly differed from the original request which was to create an ESXi performance support bundle. However, the vSphere API only supports the creation of generic VMware Support Bundle which may not be as useful if you are only interested in collecting more granular performance stats for troubleshooting purposes.

After publishing the article, I had thought about the problem a bit more and realized there is still a way to solve the original request. Before going into the solution, I wanted to quickly cover how you can generate an ESXi Performance Support Bundle which can be done either directly in the ESXi Shell using something like the following:

vm-support -p -d 60 -i 5 -w /vmfs/volumes/datastore1

or you can actually use a neat little trick which I had blogged about here back in 2011 where you can simply open a web browser and run the following:

https://esxi-1.primp-industries.com/cgi-bin/vm-support.cgi?performance=true&interval=5&duration=60

Obviously, the first option is not ideal as you would need to SSH (generally disabled for good security practices) to each and every ESXi host and then manually run the command and copy the support bundle off of each system. The second option still requires going to each and every ESXi host, however it does not require ESXi Shell or SSH access. This is still not ideal from an Automation standpoint, especially if these ESXi hosts are already being managed by vCenter Server.

However, the second option is what gave me the lightbulb idea! I had recalled a couple of years back that I had blogged about a way to efficiently transfer files to a vSphere Datastore using the vSphere API. The solution leveraged a neat little vSphere API method called AcquireGenericServiceTicket() which is part of the vCenter Server sessionManager. Using this method, we can request a ticket for a specific file with a one time HTTP request to connect directly to an ESXi host. This means, I can connect to vCenter Server using the vSphere API, retrieve all ESXi hosts from a given vSphere Cluster and request a one time ticket to remotely generate an ESXi performance support bundle and then download it locally to the VCSA (or any other place that you can run the pyvmomi sample).

Download the pyvmomi script: generate_esxi_perf_bundle_from_vsphere_cluster.py

Here is the sample log output when triggering this script from vCenter Alarm in the VCSA:

2016-06-08 19:29:42;INFO;Cluster passed from VC Alarm: Non-VSAN-Cluster
2016-06-08 19:29:42;INFO;Creating directory /storage/log/esxi-support-logs to store support bundle
2016-06-08 19:29:42;INFO;Requesting Session Ticket for 192.168.1.190
2016-06-08 19:29:42;INFO;Waiting for Performance support bundle to be generated on 192.168.1.190 to /storage/log/esxi-support-logs/vmsupport-192.168.1.190.tgz
2016-06-08 19:33:19;INFO;Requesting Session Ticket for 192.168.1.191
2016-06-08 19:33:19;INFO;Waiting for Performance support bundle to be generated on 192.168.1.191 to /storage/log/esxi-support-logs/vmsupport-192.168.1.191.tgz

I have also created a couple more scripts exercising some additional use cases that I think customers may also find useful. Stay tuned for those additional articles later this week.

UPDATE (06/16/16) - There was another question internally asking whether other types of ESXi Support Bundles could also be generated using this method and the answer is yes. You simply just need to specify the types of manifests you would like to collect, such as HungVM for example.

To list the available Manifests and their respective IDs, you can manually perform this operation once by opening browser specifying the following URL:

https://192.168.1.149/cgi-bin/vm-support.cgi?listmanifests=true

Screen Shot 2016-06-16 at 5.38.08 AM
To list the available Groups and their respective IDs, you can specify the following URL:

https://192.168.1.149/cgi-bin/vm-support.cgi?listgroups=true

Screen Shot 2016-06-16 at 5.45.56 AM
Here is an example URL constructed using some of these params:

https://192.168.1.149/cgi-bin/vm-support.cgi?manifests=HungVM:Coredump_VM%20HungVM:Suspend_VM&groups=Fault%20Hardware%20Logs%20Network%20Storage%20System%20Userworld%20Virtual&vm=FULL_PATH_TO_VM_VMX

The following VMware KB 2005715 may also be useful as it provides some additional examples on using these additional parameters.

How to run a script from a vCenter Alarm action in the VCSA?

by 1 Comment

As more and more customers transition from the vCenter Server for Windows to the vCenter Server Appliance (VCSA), a question that I get from time to time is how do I run a script using a vCenter Alarm from the VCSA? Traditionally, customers would create and run Windows specific scripts such as a simple batch script or a more powerful Powershell/PowerCLI script. However, since the VCSA is a Linux based operating system, these Windows specific scripts would no longer function, at least out of the box.

There are a couple of options:

  1. If you have quite a few scripts and you prefer not to re-write them to run on the VCSA, then you can instead have them executed on a remote system.
  2. Re-write or create new scripts that would run in the VCSA. You can leverage pyvmomi (vSphere SDK for Python) which is already included in the VCSA. This will allow you to access the vSphere API just like you would with PowerCLI on a vCenter Server for Windows if you had that installed.

Given there are already multiple ways of achieving #1 and I am sure there are other options if you just did a quick Google search, I am going to focus on #2 which will demonstrate how to create a pyvmomi script that can be triggered by a vCenter Alarm action running on the VCSA.

Disclaimer: It is generally recommended that you do not install or run additional tools or scripts on the vCenter Server (Windows or VCSA) itself as it may potentially impact the system. If it is possible, a remote system using programmatic APIs to provide centralize management and configuration is preferred and it also limits the number of users that may have access directly to the vCenter Server.

The following example is based off of a real use case from one of our customers. The requirement was to automatically generate and collect an ESXi performance bundle (part of the support bundle) for all hosts in a given vSphere Cluster when a specific vCenter Alarm is triggered. In this example, we will create a simple vCenter Alarm on our vSphere Cluster that triggers our script when a particular VM is modified as this is probably one of the easiest way to test the alarm itself.

UPDATE (06/14/16) - For generating an ESXI performance support bundle, please have a look at this article which can then be applied to this article in terms of running the script in the VCSA.

When a vCenter Alarm is triggered, there are a bunch of metadata that is passed from the vCenter Alarm itself and stored in a set of VMware specific environmental variables which you can find some more details in the documentation here. These environmental variables can then be accessed and used directly from within the script itself. Here is an example of what the environmental variables looks like for my vCenter Alarm which triggers off of the VM reconfigured event:

The variable that we care about in this particular example is the VMWARE_ALARM_EVENT_COMPUTERESOURCE as this provides us with the name of the vSphere Cluster that we will need to iterate all the ESXi hosts to then generate the ESXi support bundles. I have created the following pyvmomi script called generate_esxi_support_bundle_from_vsphere_cluster.py which you will need to download and upload to your VCSA. The script has been modified so that the credentials are stored in (line 44 and 50) within the script versus prompting for it on the command-line (you can easily tweak it for testing purposes). By default, the script will store the logs under /storage/log but if you wish to place it else where, you can modify line 55.

Note: If you wish to manually test this script before implementing the vCenter Alarm, you will probably want to comment line 105 and manually specify the vSphere Cluster in line 106.

Below are the instructions on configuring the vCenter Alarm and the python script to run:

Step 1 - Upload the generate_esxi_support_bundle_from_vsphere_cluster.py script to your VCSA (I stored it in /root) and set it to an executable by running the following command:

chmod +x generate_esxi_support_bundle_from_vsphere_cluster.py

Step 2 - Create a new vCenter Alarm under a vSphere Cluster which has some ESXi hosts attached. The trigger event will be VM reconfigured and the alarm action will be the path to the script such as /root/generate_esxi_support_bundle_from_vsphere_cluster.py as shown in the screenshot below.

run-script-from-vcenter-alarm-vcsa-0
Step 3 - Go ahead and modify the VM to trigger the alarm. I found the easiest way with the least amount of clicks is to just update the VM's annotation 😉 If everything was configured successfully, you should see the vCenter Alarm activate as well as a task to start the log bundle collection. Since the script is being call non-interactively, if you want to see the progress of the script itself, you can login to the VCSA and all details from the script is logged to its own log file in /var/log/vcenter_alarms.log.

Here is a snippet of what you would see

2016-06-05 17:37:54;INFO;Cluster passed from VC Alarm: Non-VSAN-Cluster
2016-06-05 17:37:54;INFO;Generating support bundle
2016-06-05 17:40:55;INFO;Downloading https://192.168.1.190:443/downloads/vmsupport-527c584f-f4cf-cd6d-fc3e-963a7a375bc9.tgz to /storage/log/esxi-support-logs/vmsupport-192.168.1.190.tgz
2016-06-05 17:40:55;INFO;Downloading https://192.168.1.191:443/downloads/vmsupport-521774a8-4fba-7ce4-42a5-e2f0a244f237.tgz to /storage/log/esxi-support-logs/vmsupport-192.168.1.191.tgz

Hopefully this gives you an idea of how you might run custom scripts triggered from a vCenter Alarm from within the VCSA. With a bit of vSphere API knowledge and the fact that pyvmomi is installed on the VCSA by default, you can create some pretty powerful workflows triggered from various events using a vCenter Alarm in the VCSA!

Getting started with the new VSAN 6.2 Management API

by 3 Comments

As I have previously written, with the release of VSAN 6.2 (vSphere 6.0 Update 2), a new VSAN Management API has been introduced which allows developers, partners and administrators to automate all aspects of VSAN functionality including: complete lifecycle (install, upgrade, patch), monitoring (including RVC and VSAN Observer capabilities), configuration and troubleshooting. Simply put, anything that you can do from the vSphere Web Client UI or the RVC CLI from a VSAN standpoint, you will be able to completely automate using one of the four new VSAN Management SDKs: Python, Ruby, Java and C#.

In this article, I will show you how to quickly get started with the new VSAN Management API by exercising two of the VSAN Management SDKs: Python and Ruby. Another must bookmark is the VSAN Management API Reference Guide which provides more details on the individual APIs and how they work.

Step 1 - Download the VSAN Management SDK of your choice. You can find the VSAN Management SDK downloads in either of two locations:

In this example, I will be using the VSAN Management for Python and Ruby.

Step 2 - Extract the VSAN Management SDK zip file which should give you a directory that contains a README on how to setup the SDK and three folders as shown in the screenshot below:

Screen Shot 2016-03-17 at 6.27.58 AM
The bindings directory contains the language specific library to the VSAN Management API. The docs folder contains the offline copy of the VSAN Management API Reference Guide and lastly the sample directory contains a basic sample to connect to VSAN Cluster as well as an individual ESXi host contributing to a VSAN Cluster.

Step 3 - Each of the VSAN Management SDKs extends the existing vSphere Management SDKs. This means that you will need to have the appropriate vSphere Management SDK installed on your system before you can proceed further. In our example, Python requires pyvmomi (vSphere SDK for Python) and Ruby requires rbvmomi (vSphere SDK for Ruby). If you are on Mac OS X, it is pretty easy to install these packages. Make sure you are running the latest version of these SDKs.

Installing pyvmomi:

pip install pyvmomi

Upgrading pyvmomi: (if you already have it installed)

pip install --upgrade pyvmomi

Installing rbvmomi:

gem install rbvmomi

Step 4 - Copy the VSAN Management SDK library file over to the samples directory.

VSAN Mgmt SDK for Python:

cp bindings/vsanmgmtObjects.py samplecode/

VSAN Mgmt SDK for Ruby:

cp bindings/vsanmgmt.api.rb samplecode/

Step 5 - At this point, we can quickly verify that everything was setup correctly by going into the samplecode directory and then run one of the following commands below. If everything is working as expected, then you should see the usage information being printed out. If you do not, the issue is most likely with vSphere Management SDK either not being the latest version or not configured in the default library path for the sample to access.

VSAN Mgmt SDK for Python:

python vsanapisamples.py

Screen Shot 2016-03-17 at 6.43.32 AM
VSAN Mgmt SDK for Ruby:

ruby vsanapisamples.rb

Screen Shot 2016-03-17 at 6.43.56 AM
Step 6 - Now that we have verified our VSAN Management SDK installation was successful, we can now connect to a real VSAN Cluster. To do so, run the following command and specify your vCenter Server along with the credentials as well as the name of the VSAN Cluster. If successful, you should see the status for each of your VSAN hosts and its current state as seen in the screenshot below.

VSAN Mgmt SDK for Python:

python vsanapisamples.py -s 192.168.1.139 -u 'administrator@vghetto.local' -p 'VMware1!' --cluster VSAN-Cluster

Screen Shot 2016-03-17 at 6.54.38 AM
VSAN Mgmt SDK for Ruby:

ruby vsanapisamples.rb -o 192.168.1.139 -u 'administrator@vghetto.local' -k -p 'VMware1!' VSAN-Cluster

Screen Shot 2016-03-17 at 6.56.34 AM
Step 7 - Each individual ESXi hosts that participate in the VSAN Cluster also exposes an VSAN Management API endpoint. We can use this exact same sample to connect to one of the hosts to get some additional information. To do so, run the following command and specify your ESXi hosts along with the credentials.

VSAN Mgmt SDK for Python:

python vsanapisamples.py -s 192.168.1.190 -u root -p vmware123

Screen Shot 2016-03-17 at 7.00.28 AM
VSAN Mgmt SDK for Ruby:

ruby vsanapisamples.rb -o 192.168.1.190 -u root -p vmware123

Screen Shot 2016-03-17 at 6.59.46 AM
As you can see, it is pretty straight forward on getting the new VSAN Management SDK up and running. The provided sample only scratches the surface of what is possible and for a complete list of capabilities within the new VSAN Management API, be sure to check out the VSAN Management API Reference document for more information. I am really looking forward to seeing what solutions our customers and partners develop using this new API. If you would like to contribute code samples back to the community or just find new samples be sure to check out the VMware Developer Center Sample Exchange. 

pyvmomi (vSphere SDK for Python) 5.5.0-2014.1 released!

by 1 Comment

I just saw an awesome update from Shawn Hartsock, a fellow VMware colleague. For those of you who do not know him, Shawn works in our Ecosystems and Solutions Engineering (EASE) organization and is the primary maintainer of VMware's pyvmomi (vSphere SDK for Python) open-source project. The pyvmomi project was open sourced since last December which I had written about here, it has received over 3K+ downloads and has a very active community. Much of this success has been due to the hard from Shawn fostering an active community around pyvmomi.

The announcement today from Shawn is a new release of pyvmomi at version 5.5.0-2014.1:

As mentioned earlier, the pyvmomi project is a very active project and Shawn is constantly engaging with users looking for feedback, suggestions or requests for new samples to build. If you are interested in vSphere Automation and would like to leverage Python, be sure to check out the pyvmomi Github repository! Lastly, if you have written some cool scripts/applications or would like to request specific sample scripts, be sure to send a pull request to Shawn as we would love to see more contributions and collaborations from the community!

Having some fun with the Marvel Comics API & vSphere SDK for Python

by 2 Comments

As a child of the 80's, I still remember waking up super early on a Saturday morning to watch some of my favorite cartoons including Transformers, GI Joe, Ninja Turtles and many of the Marvel cartoon series. Last week I saw an interesting tweet from our former VMware CTO, Steve Herrod:

It looks like Marvel has just released a Marvel Comics REST API which allows developers to access the entire Marvel Comics Universe and build custom applications and websites with that data. So, you might be wondering what does this have to do with VMware and virtualization? Well, when I heard about this API, I knew I had to find a way to incorporate this into a cool VMware project.

I decided to create a sample vSphere SDK for Python (also known as pyvmomi) script that would allow a user to specify a number of random Marvel Characters to select from the Marvel database and then automatically create dummy VMs based on those names which could be used for testing. There were a variety of VMware SDK's (Perl, Ruby, PowerCLI, Java, etc) that I could have used, but I opted for the vSphere SDK for Python given my Python skills were not very strong. This also gave me a good reason to play around with our Python SDK.

There are a couple of things you will need to setup before you can use the sample script:

  1. Setup a Marvel API key
  2. Install vSphere SDK for Python, you can take a look here.
  3. Download the create_random_marvel_vms.py

Disclaimer: There is not much error handling built into the sample script, please use at your own risk.

You will need to edit the script and replace the following two variables: marvel_public_key & marvel_private_key with the API key after creating an account on Marvel's website. One thing I really like about the Marvel API and I know many other vendors also provide this which is an interactive API tester. This allows a user to explore and execute a particular API requests without writing a single line of code to get a better understanding of the results before one starts to develop their application.

To run the script, you will need to pass in the following parameters:

--host : Name of your vCenter Server (only tested with vCenter)
--user : User to login with
--password : Password for the user
--datastore : Name of the datastore in which VMs will be created in
--count : The number of Dummy VMs to create

Here is an example of running the script:

./create_random_marvel_vms.py --host vcenter --user root --password vmware --datastore mini-local-datastore-1 --count 10

As you can see from the screenshot above I have just created 10 random VMs where each name is a random Marvel character extracted using a GET request on /v1/public/characters. If we head over to our vSphere Web Client, we should now see our 10 newly created VMs which contain 1vCPU and 128MB of memory. To keep things simple, I did not add a vDisk but you can easily tweak the dummy VM configuration based on your requirements.

Disclaimer: Data provided by Marvel. © 2014 Marvel