virtuallyGhetto

William Lam

The vStorage API, do you really know what it is?

by 2 Comments

I’ve seen this question posed quite a few times both in the VMware developer and VMTN forum asking what exactly is the vStorage API?

So what is it?

If your answer was VMware marketing term, then you win 50 Schrute Bucks!

The VMware vStorage API is actually a blanket umbrella term that encompasses 4 separate individual APIs, all with different functionalities:

  1. vStorage API for Data Protection (VADP)
  2. vStorage API for Site Recovery Manager (VASRM)
  3. vStorage API for Multi-pathing (VAMP)
  4. vStorage API for Array Integration (VAAI)

I’m actually going to quote one of Chad Sakac’s reply on a blog about these 4 APIs as he has done a great job of explaining the differences:

1) vStorage API for Data Protection – a set of APIs focused on local backup/recovery use cases.

2) vStorage APIs for Site Recovery Manager – a set of APIs focused on array vendor remote replication such that they can be orchestrated by Site Recovery Manager.

3) vStorage APIs for Multipathing (otherwise known as the Pluggable Storage Architecture). A set of APIs for 3rd parties to extend vSphere’s core multipathing architectures.

4) vStorage APIs for Array Integration (VAAI). Technically not in vSphere 4, but have been discussed in VMworld events in the past. Will be available in future vSphere-generation releases. This allow array vendors to “offload” various tasks from the ESX host’s vmkernel stack – things like writing blocks that make up VMs, copying/snapshotting blocks, doing thin-provisioning out of space handling, and also a much more advanced global locking mechanism than VMFS uses today. These each will make common actions 5x-10x faster (clone, deploy from template, create a FT VM), and improve VMFS scaling by an order of magnitude. More on that here, for folks that are interested (note that when I wrote this, vStorage API for Data Protection was called the VCB Backup Framework)

Generally when the vStorage APIs are brought up, most people think about backups and the new Change Block Tracking feature in vSphere. That is because VMware and other 3rd party backup vendors has done a good job of marketing this "must have" feature. Leveraging Change Block Tracking helps decrease the duration of a backup by only copying the blocks that have changed and some users have seen up to 5-10x increase in speed.

This is great! But now you might ask the question, how do I use vStorage API for Data Protection and how does it tie into VMware's VCB product? Both the vStorage API for Data Protection and VCB are backup API frameworks, they are similar in functionality but different in features (for a break down of the two, take a look at this VMware blog post). VADP will be pretty much hidden from the end user's perspective, you just need to ensure that backup vendors are implementing it and utilizing Change Block Tracking to efficiently backup your VMs. VADP is only available in vSphere and not in VI 3.5.

If you want to write your own backup application, then you will need to know how to hook into VADP. As far as I understand today, and correct me if I'm wrong, the vStorage API for Data Protection is actually the combination of the vSphere 4.0 API + VMware Virtual Disk Development Kit (VDDK) which are both available to users to develop against. There's actually a guide within the VDDK page on Designing Backup Solutions for VMware vSphere which goes into great detail on creating a backup solution using the Change Block Tracking feature.

The three other APIs (VASRM, VAMP and VAAI) are targeted at third party hardware/software vendors and storage array providers to hook in their special sauces with the various VMware solutions. This includes hooking into Site Recovery Manager, PSA (Pluggable Storage Architecture) plugins and offloading VMware operations such as VM cloning and storage vMotions, etc. onto the actual storage array. These APIs are only exposed to partners who provide solutions to one of these features and are not available to the public for general use.

I personally think vStorage API is going to be a game changer and Change Block Tracking is just one of the many cool features to come!

Hopefully this all made sense and if you're interested to learn more about the vStorage API and some of the upcoming features, take a look at these additional resources:

http://www.ntpro.nl/blog/archives/1461-Storage-Protocol-Choices-Storage-Best-Practices-for-vSphere.html
http://virtualgeek.typepad.com/virtual_geek/2008/09/so-what-does-vs.html
http://www.vmware.com/products/vstorage-apis-for-data-protection/
http://www.yellow-bricks.com/2009/03/19/pluggable-storage-architecture-exploring-the-next-version-of-esxvcenter/

virtuallyGhetto says hello to the blogosphere!

by Leave a Comment

We finally decided to move our content over from our ghetto web page - http://engineering.ucsb.edu/~duonglt/vmware/ over to Google's free blogspot. This move not only helps us to organize our content and benefit from automatic RSS feeds, but it also makes the content searchable.

This site will still focus primarily on VMware automation using the various scripting APIs and SDKs. We hope you like the new layout and we look forward to sharing our scripts with the VMware community!

vSphere Tips and Tricks Part 1

by 3 Comments

Disclaimer: Ensure these changes have been thoroughly tested and confirmed with VMware support prior to rolling out into production. Most of these changes and tips may be undocumented by VMware, test at your own risk!

Map Service Console/COS VMDK to specific ESX host (useful if you boot from SAN):

grep -i "hostname" /vmfs/volumes/[datastore]/[esxconsole-datastore-uuid]/logs/sysboot-vmkernel-late.log
[[email protected] ~]# grep -i "hostname" /vmfs/volumes/everest-local-storage/esxconsole-49a095a4-204d-c7ba-9bb8-001f29c7b048/logs/sysboot-vmkernel-late.log
0:00:00:06.289 cpu1:4105)Config: 363: "HostName" = "everest.primp-industries.com", Old value: "localhost" (Status: 0x0)

Retrieve VmId from vmware-cmd similiar to that in vimsh/vmware-vim-cmd (only available on classic ESX, not sure why it's not in vCLI):

vmware-cmd /vmfs/volumes/[datastore]/[path_to_vm]/vm.vmx getid
[[email protected] ~]# vmware-cmd /vmfs/volumes/d32c5a97-e039bdbf/vCenter-ESX4.02172009/vCenter-ESX4.02172009.vmx getid
getid() = 32

VMware FT statistics and debugging utility
Available after some patch set, generally 4.0u1+

[[email protected] ~]# ft-stats
Usage: ft-stats [-hvf] [-c ] [-l] [-n ] [-d ] [-s ] [-w ]
-h, --help Show this message
-v, --verbose verbose output
-f, --force Continue even if version check fails
-c, --cache Use VSI cache file
-l, --list list FT VMs
-n, --iterations Print stats for n iterations
-d, --delay Print stats every n secs (default:5)
-s, --snapshot Collect stats for n secs
-w, --world Print stats for worldID

Note: This tool is intended to assist in troubleshooting and should only be used
with direction by VMware support. All other use of this tool is unsupported.

esxupdate - Query for installed/retired packages with undocumented "--vib-view" option

esxupdate --vib-view query
[[email protected] ~]# esxupdate --vib-view query
--------------------------------------VIB ID------------------------------------- Package State -----------Timestamp------------
rpm_[email protected]x86_64 installed 2010-01-06T12:24:38.100201-05:00
[email protected]_64 installed 2010-04-02T17:09:44.566014+00:00
[email protected]_64 installed 2010-04-02T17:09:44.171847+00:00
[email protected]_64 installed 2010-04-02T17:09:41.148547+00:00
[email protected]_64 installed 2010-04-02T17:09:36.911452+00:00

Change default COW (copy on write) disk growth value
Query (Default is 16MB):

[[email protected] ~]# esxcfg-advcfg -g /COW/COWDiskSizeIncrement
Value of COWDiskSizeIncrement is 32768

Set (e.g. 256MB):

[[email protected] ~]# esxcfg-advcfg -s 524288 /COW/COWDiskSizeIncrement
Value of COWDiskSizeIncrement is 524288

The input value for changing COWDiskSizeIncrement is in sectors

UNIT SECTORS COW GROWTH SIZE
32768 16MB
65536 32MB
131072 64MB
262144 128MB
524288 256MB
2097152 1024MB
.... ....

Easy formula is (SIZE_IN_MB)*(1024)*(2) = SECTOR VALUE

Getting started with vMA

by 4 Comments

Documentation

What is vMA?

  • Here is a video explaining vMA on VMware KBTV

Visit the vMA Home Page:

vMA Forum:

You will find latest Downloads, Reference Documentation and Community Discussions and links to resources.

Downloads

Tutorials

    Setting up vMA video by David Davis

    Setting up vMA to be a syslog server by Simon Long

    vi-fastpass

    Upgrade vMA:

    How to clone vMA:

    Backup & Restore vMA + vi-fastpass DB:

    Scripts and Tools

    vGhetto Script Repository:

    New tools and scripts to add/manage ESX(i) and vCenter targets on vMA 4.x:

    vMA Compatible Agent/Scripts(Third Party) Community List:

    Health check reports on your VMware environment:

    Taking screenshots of your VM using vMA 4.x:

    Host and VM shutdown with UPS integration using vMA:

    Backup VMs using vMA:

    Utilize vi-fastpass On vMA to run Perl script without clear text password:

    Tweaks

    Hot add memory to vMA or other supported Linux guestOSes using vSphere ESX(i) 4.x:

    How to increase/resize vMA Disks:

    Offline patching of vMA 4.x:

    How to install VMware VDDK on vMA:

    Reconfigure networking on vMA:

    vMA & Active Directory:

    Removing stale targets from vMA:

    Enable root account:

    Edit /etc/passwd using vi

    sudo vi /etc/passwd

    Change

    root:x:0:0:root:/root:/sbin/nologin

    to


    root:x:0:0:root:/root:/bin/bash

    Filed Under: Uncategorized Tagged With: ,

    Getting started with the vSphere SDK for Perl

    by 8 Comments

    Step 1. Visit the vSphere SDK for Perl Home Page

    You will find latest Downloads, Reference Documentation and Community Discussions and links to resources.

    Step 2. Become familiar with vSphere SDK for Perl Resources:

    Step 3. Explore new ways of managing your vSphere Platform using the remote CLI

    • Try some of the esxcfg-* commands bundled with the vCLI/RCLI

    Additional utilities bundled with vCLI/RCLI installation

    Step 4. Develop and create your own scripts and utlities using the vSphere SDK for Perl
    Become familiar vSphere 4.0 API Reference documentation

    Step 5. Learn how to get started with writing your own Perl script using vSphere SDK for Perl

    • This VMware document is a little out dated but it still all applies to writing your own Perl scripts using the vSphere SDK for Perl

    Step 6. For more scripts developed against the vSphere SDK for Perl, check out the vGhetto Script Repository for more examples!
    I think the easiest way to learn this is to start playing with some of the canned scripts and to make it even easier to have an environment that's all setup for you with both the vCLI and vSphere SDK for Perl, you can download VMware vMA.

    vSphere ESX 4.0 – Crash VM Bug?

    by Leave a Comment

    We recently discovered an anomaly while backing up one of our development VMs using ghettoVCB.sh. When attempting to back up this powered on VM, the backup was successful however oddly, we were left with a powered off VM immediately following the first VMDK clone operation. After some investigation, we found that the problematic VM contained virtual disks spread across two datastores with dissimilar blocksizes (1MB and 2MB).

    The VM configuration alongside its main OS disk was stored on the datastore with a 1MB blocksize while it’s data disk (>256 GB) resided on the other datastore which was initialized with a 2MB blocksize. We came to the conclusion that this might have had something to do with the VM configuration residing on a datastore with a blocksize that was smaller than what is needed for the larger VMDK (which was on a datastore with an ample blocksize). Manually snapshotting this VM apparently fails however different behavior was experienced when the commands are executed from a script.

    Believing that this was a corner case, we decided that it was best practice to keep all VMFS volume block sizes consistent. This was to be remediated at a later time.

    Today we noticed a blog post http://www.yellow-bricks.com/2009/08/24/vsphere-vm-snapshots-and-block-size/ from Duncan Epping regarding the snapshot issue. This may not be a corner case as we thought so we wanted to share this finding with everyone.

    If you have a similar configuration from above, it is guaranteed that the VM will crash if you run a script that tries to take a snapshot of the described VM and then subsequently exports the VMDK using vmkfstools.

    Here is a video displaying the symptoms decribed from above:
    http://engineering.ucsb.edu/~duonglt/vmware/crashVM

    Output from script execution on VM: Quentin:

    [[email protected] ~]# ./crashVM.sh Quentin /vmfs/volumes/dlgCore-NFS-bigboi.VM-Backups/Quentin-clone.vmdk
    thinking, give me a few ...
    The power state of VM: "Quentin" is On
    Extracted VmId and locating VM configured datastore (which should live on a smaller VMFS block size)
    Located VMDK: "/vmfs/volumes/himalaya-local-SAS.Savvio/Quentin/Quentin.vmdk"
    Trying to create snapshot ... (this should fail)
    Create Snapshot:
    Trying to vmkfstools copy "/vmfs/volumes/himalaya-local-SAS.Savvio/Quentin/Quentin.vmdk" (this should REALLY fail! right?)
    Destination disk format: VMFS zeroedthick
    Cloning disk '/vmfs/volumes/himalaya-local-SAS.Savvio/Quentin/Quentin.vmdk'...
    Clone: 9% done.

    Download crashVM.sh script