1200+ undocumented .vmx parameters

Recently while performing some skunkworks testing in my personal lab, I came across a slew of documented and undocumented virtual machine .vmx configuration parameters. Using one of my favorite UNIX/Linux utility strings, I was able to uncover some interesting things in the /usr/lib/vmware/bin/vmware-vmx binary which is used to load a virtual machines configuration file.

Here are some of the interesting observations I have made:

vSphere is hypervisor aware?

%s: %s detected by CPUID
%s: VMware detected
Microsoft Hyper-V
%s: Xen detected by hypercall
Xen detected but hypervisor unrecognized (Xen variant?)

I noticed the following strings around detecting certain guest hypervisors, is this a hint that VMware is going to support other virtual "hypervisors", specifically Microsoft and Xen?

vSphere to support Mac OSX?

Linux Host
Windows Host
Mac OS Host

There were some text that listed the various types of host, including Mac OSX.

Make sure that you have installed all available Mac OS X software updates.
@&!*@*@(msg.cdrom.darwindisconnect)Your Mac OS guest is using this CD-ROM device. The safest way to disconnect this virtual CD-ROM is by pressing %s, then ej
ecting the media from inside the guest%s. To continue anyway, press %s.%s
@&!*@*@(msg.Backdoor.OsNotMacOSXServer)The guest operating system is not Mac OS X Server.
@&!*@*@(msg.cpuid.darwinWithBTHV)Mac OS X is not supported with software virtualization. Change the execution mode to automatic.
@&!*@*@(msg.cpuid.darwinWithBT)Mac OS X is not supported with software virtualization. To run Mac OS X you need a host on which %s supports hardware virtuali
zation.
isolation.bios.IsGOS.Darwin

There were some text that listed various messages regarding Mac OSX

sbios
vbios
bios440
efi32
efi64
nvram
lsibios
nbios
nxbios
nx3bios
e1000bios
vmibios
vmmmods
sas1068bios
pvscsibios

As you can see, there is mention of EFI support which is required to boot Mac OSX. Does this mean future version of vSphere will support virtualizing Mac OSX?

New guestOS types?

darwin10
darwin10-64
darwin-64
mandrake-64
opensuse
opensuse-64
winServer2008Cluster-32
winServer2008Cluster-64
winServer2008Datacenter-32
winServer2008Datacenter-64
winServer2008DatacenterCore-32
winServer2008DatacenterCore-64
winServer2008Enterprise-32
winServer2008Enterprise-64
winServer2008EnterpriseCore-32
winServer2008EnterpriseCore-64
winServer2008SmallBusiness-32
winServer2008SmallBusiness-64
winServer2008SmallBusinessPremium-32
winServer2008SmallBusinessPremium-64
winServer2008Standard-32
winServer2008Standard-64
winServer2008StandardCore-32
winServer2008StandardCore-64
winServer2008Web-32
winServer2008Web-64
XenVMMXenVMM

There was a section that I came across which listed all supported guestOS types, here you can see there have been a few more that were added between vSphere 4.0 and 4.1. One interesting thing that I am not sure if a lot of people have noticed, is the VirtualMachineGuestOsIdentifier in the vSphere API. This basically provides the guestos identifier that is supported in each release of VI/vSphere. Interesting enough, a darwin guestos support has been documented as of vSphere 4.0:

Though we all know we can not run Mac OSX on ESX(i) ... at least not just yet from what the above is hinting at.

These were just a few of the interesting things I found while parsing through the strings output when looking at the ESX 4.1's vmware-vmx binary.

Here is a collection of over 1200+ documented and undocumented .vmx configuration parameters.

**** These are not documented by VMware, use at your own risk! ****

http://https://s3.amazonaws.com/virtuallyghetto-download/hidden_vmx_params.html

**** These are not documented by VMware, use at your own risk! ****

Some of these hidden .vmx entries have been shared by the VMware and the community, here are just a few:

• http://www.virtuallyghetto.com/2010/10/how-to-control-maximum-number-of-vmware.html
•  snapshot.maxSnapshots = Control the maximum number of VMware snapshots
• http://www.vcritical.com/2009/05/vmware-esx-4-can-even-virtualize-itself/
• monitor_control.restrict_backdoor = Run virtual ESX(i) hosts on top of ESX or ESXi
  
• http://vinf.net/2009/06/07/vsphere-cannot-enable-ft-for-a-nested-vm/ 
•  replay.allowBTOnly = Allow FT to be enabled on vVM running on vESX(i)
•  replay.allowBT = Allow FT to be enabled on vVM running on vESX(i)
• http://kb.vmware.com/kb/1010184
• cpuid.coresPerSocket = Specify the number of cores per physical socket
• http://www.sanbarrow.com/vmx/vmx-advanced.html 

How to control maximum number of VMware snapshots

There are currently no methods of controlling the number of VMware snapshots using vCenter or ESX(i) permissions today, you either provide the snapshot privilege or you deny it all together. I recently discovered an undocumented .vmx entry that allows you to control the maximum number of VMware snapshots for a given virtual machine. By default, a virtual machine can have a snapshot tree depth of 31, in the worse case scenario supporting up to a maximum of 496 snapshots.

Here is what a VM looks like with 496 snapshots (unexpanded):

Note: If you are interested in what this looks like fully expanded, take a look at the screenshot at the very bottom of this post.

If you like to prevent the above or at least control the maximum number of snapshots for a given virtual machine, you can add the following into a VM's .vmx configuration file.

snapshot.maxSnapshots = "n"

where n = max number of snapshots and n <= 496

Here is a screenshot of adding this .vmx parameter using the vSphere Client:

The virtual machine above already has one snapshot and per the configuration change, we should not be able to take any additional snapshots:

Next, we will try to take a second snapshot:

As you can see, an error is thrown that we have reached the maximum number of permitted snapshots. If you would like to disable snapshots all together, you can set the value to be 0 and this will prevent anyone from taking snapshots, including administrators. 

Here is a an screenshot of the expanded view of a VM with 496 snapshots:

Note: These snapshots were created with a VM running in an vESXi host and script to exhaust the maximum snapshot depth of 31. Each snapshot level was also exhausted with the maximum number of snapshots. Starting from level-1: it was the maximum depth minus 1, level-2: it was maximum depth minus 2, and so fourth. This was just a test to see what the system could handle, you should not try this a home or on a production VM ;) Use at your own risk

Where are the "Power" Perf Metrics in the vSphere API?

A recent question was posed on the VMTN developer forum on how to obtain the new power utilization metrics using the vSphere API. This new performance metric was introduced with the release of vSphere 4.x and can be seen using either esxtop or resxtop and specifying the "p" option for power if you are on an ESX or ESXi host.

You can also get these counters by using the vSphere Client and using the Advanced Charts:

This actually seemed like a simple enough question, pointing the user over to the vSphere API reference documentation under the perfManager. Though after taking a second look, it appears that no such metric exists in the documentation from VMware:

After a few minutes of digging around, I found that Power metrics actually do in fact exists but were not properly documented when they were first introduced. I wrote a quick vSphere SDK for Perl script called perfQuery.pl looking for metrics that were related to "power" and I identified the following:

As you can see these match up to those seen using the vSphere Client and I output the metrics using its rollup type, units, internal name and metric description. While writing this script, I also noticed there were two other performance metric types that existed and were not documented by VMware. Here is a mapping of the API performance metric keys to vSphere API perfManager, the last two including power metric types are undocumented by VMware:

vSphere Client Chart Option	vSphere API Perf Metric Key	Documented
Cluster Services	clusterServices	yes
CPU	cpu	yes
Management Agent	managementAgent	yes
Memory	mem	yes
Network	net	yes
Resource Scheduler	rescpu	yes
Storage Capacity	disk	yes
Datastore	datastore	yes
Disk	disk	yes
Virtual Disk	virtualDisk	yes
Storage Adapter	storageAdapter	yes
Storage Path	storagePath	yes
System	sys	yes
Virtual Machine Operations	vmop	yes
Power	power	no
vCenter Resources	vcResources	no
vCenter Debug Info	vcDebugInfo	no

Using the script and the performance metric key, you can actually query either all or a specific metric type that you are interested in. This is helpful, for those metrics that have not been publicly documented by VMware. However, the power metric should have been documented and I believe this to be a documentation bug that was missed by VMware.

Download: perfQuery.pl

If you are interested in learning more about the vSphere statistics and performance monitoring, I highly recommend checking out Luc Dekens three part series (Part1, Part2 and Part3) on vSphere performance monitoring. Even though his posts are specific to PowerCLI, all the concepts discussed apply to all the vSphere SDKs when dealing with performance monitoring using the vSphere APIs.

vGhetto Tech Exchange VMworld 2010 Video

Back in September, I presented at my very first VMworld: vGhetto - Communities Building Great Solutions (PPC-07) alongside Edward Haletky at Tech Exchange VMworld 2010. For those that missed the session, the famous Pablo Roesch of VMware has graciously uploaded our presentation to Vimeo :)

You can view the presentation here.


vGhetto - Communities Building Great Solutions from heyitspablo on Vimeo.

vMotion' With Style

I got this idea after catching an interesting tweet by Cody Bunch last Friday:

I had just recently finished a post on How to Ack & Reset vCenter Alarm implementing hidden API method and thought this might work by using a vCenter alarm. After few hours in our skunkwork's lab, I found "a" method to exactly what Cody wished for. If you can not wait, jump straight to the video at the bottom :)

The following tools were used:

• PsExec -Windows utility to perform remote operations
• SayStatic - Windows text to speech utility
• A random mp3 audio file found online

The environment consisted of two vESXi 4.1 hosts and Windows XP VM residing on shared storage that was vMotion functional:

You will need to upload some files to your vCenter server and to the desktop in which you want to implement this hack.

Local Desktop Server Setup

1) Download SayStatic to your local desktop ( in the example, it is located on the desktop)

2) Download a playable mp3 file (in the example, it is located on the desktop)

Note: If you decide to change the path to the files above, please make note of the path as it will be used later in the alarm script.

Here is a screenshot of my local desktop:

vCenter Server Setup

1) Download PsTools toolkit and extract the contents to your vCenter Server and make a note of the path (in the example, it is located on the desktop)

2) Create an alarm script and make a note of the path ( in the example, it is located in C:\alarm.cmd)

The alarm.cmd should contain the following:

You will need to edit the script and update at minimum the following variables:
REMOTE_SERVER = This is the hostname or IP address of your local desktop, make sure you keep the double slashes which is needed

REMOTE_USERNAME = This is the username to login to your local desktop and it should be the one you use to login else you will not see anything interesting. Make sure you include both the Domain and username, if your system is not part of a domain, use the local system name else the script will fail.

REMOTE_PASSWORD = The password to the account aove

Note: If you have changed the path of the files on either the local desktop or vCenter Server, you will need edit the remainder variables that specify where to look for the executable's.

Here is a screenshot of vCenter Server desktop:

vCenter Alarm:

Finally, we need to create an alarm in vCenter, you can create this at any level of the vCenter infrastructure.

1) Create an alarm and give it a name and ensure the type is of "Virtual Machines" and monitor type is "Monitor for specific events occurring on this object" (in the example, I call it VMOTIONIN_WITH_STYLE)

2) Now click on the Triggers tab and look for an event type called "VM emigrating", no this is not a typo. VMware apparently has both "VM migrating" and "VM emigrating", apparently the functional alarm is under the name emigrating ... don't ask me why they called it that ;). Make sure the status is set to "alert"

3) Next, click on the Actions tab and set the action to "Run a command" and specify the configuration to the path of the alarm.cmd. In the example, I stored the script in C:\alarm.cmd and then make sure it alarms when it hits "red" or error and then click okay for the alarm to be created.

Now you are ready to go. Instead of explaining and providing screenshots, I thought I show you what this would look like (explanation of what is going on is at the very bottom).

Without further ado, here is a recorded video of this in action:


vMotion' With Style from lamw on Vimeo.

What is actually going on:
When a vMotion is triggered, it will fire off the alarm.cmd script which basically uses SayStatic.exe to remotely execute on your local desktop to announce the virtual machine being vMotioned by capturing the VMware specific environmental variables and then it will also remotely playing the local audio file using Windows Media Player.One caveat that I was not able to solve, was clearing the alarm after the vMotion. You will notice the virtual machines that vMotion and fire off this alarm will stay alerted and will not play again until it has been resetted to green. I thought about creating another alarm to clear this initial alarm but it did not actually clear the alarm.

There you have it, vMotioin' with style ... though cool in concept, I doubt you will last very long with this kicking off on every single vMotion in your environment.

Big thanks to Cody Bunch for the idea! :)

resxtop bug in vCLI 4.1 not vMA 4.1

I recently noticed a thread in the vMA forums regarding an issue using resxtop on vMA 4.1 to view VM disk statistics on an ESXi 4.0 hosts. The thread was started on July 26th 2010 and as far as I could tell, no resolution was ever provided. A recent comment that was left on Oct 14th 2010 by another user experiencing the same behavior got my attention while browsing the VMTN forums. I decided to perform a small test to see if this was in fact an issue and it turns out it maybe a bug in resxtop running on vMA 4.1.

The test environment consists of the following:

• 1 x vESXi 4.0u2 running 1 VM
• 1 x vESXi 4.1 running 1 VM
• 1 x vMA 4.0
• 1 x vMA 4.1

Here is a screenshot of running esxtop locally within Tech Support Mode (Busybox console) on the ESXi 4.0 hosts and you can see, the VM disk statistics are visible and present:

Here is a screenshot of running esxtop locally within Tech Support Mode (Busybox console) on the ESXi 4.1 hosts and you can see, the VM disk statistics are visible and present:

Now here is a screenshot of running both vMA 4.1 (on top) and vMA 4.0 (on bottom) connecting to an ESXi 4.0 host running a single virtual machine called VM2. I use resxtop to connect to the ESXi 4.0 and select "v" option or VM disk statistics and as you can see from the screenshot, no statistics are being displayed when using vMA 4.1:

I perform the same exact test but now connecting to an ESXi 4.1 host using both vMA 4.1 (on top) and vMA 4.0 (on bottom) and what is actually surprising is, the VM disk statistics shows up for both vMA 4.0 and vMA 4.1:

It looks like something changed in the resxtop binary between vMA 4.0 and vMA 4.1 that causes the the VM disk statistics on ESX(i) hosts running on 4.0 not to be visible. I have not found any VMware KB articles documenting this issue nor found anything in vMA's release notes in which this configuration is not supported. This looks like a bug to me and I will try follow-up with the vMA's product manager to get an official word.

Note: I used ESXi since it was quicker to deploy for this test, but the issue affects both ESX and ESXi 4.0 when using resxtop from vMA 4.1

UPDATE:  After further investigation, I found out the issue is in fact with vCLI 4.1 installation and not with vMA 4.1. To confirm, I spun up a CentOS VM and installed and individually tested vCLI 4.0u2 and vCLI 4.1 and experience the same behavior as in vMA. I have already reported the issue to vMA product manager and hopefully we can get this resolved in either a patch or an updated released.

How to Ack & Reset vCenter Alarm implementing hidden API method

There was a recent question in the VMTN developers forum around automating the acknowledgment and resetting of a triggered vCenter alarm using vSphere SDK for Perl. This is actually a trivial task using the vSphere Client, you would first identify the alarm and acknowledge the alarm by right clicking on the alarm and selecting "Acknowledge Alarm". To reset the alarm, you would right click on the alarm and select "Reset Alarm to Green".

To automate this task using the vSphere SDK for Perl which uses the vSphere API, you would perform the same two API operations. Doing a search, you will find a method called AcknowledgeAarm which should does exactly that, but if you try to search for method to reset an alarm, you will notice no such method exists. This was something I was aware of since vSphere 3.5 API, but never understood why it was kept hidden.

Here is a screenshot of the alarmManager in the vSphere MOB and you will notice no methods pertains to resetting an alarm:

You might ask, how is the vSphere Client performing this operation and what API method is it using? Sadly, this operation is one of the many hidden API methods that VMware choose to hide and not public expose for various reasons.

However, there are several ways of identifying some of these hidden vSphere API methods and properties. When you install the vSphere Client on your workstation, there is a catalog directory (e.g. C:\Program Files\VMware\Infrastructure\Virtual Infrastructure Client\4.1\Catalogs\Default)  that is created based on the version of vSphere and within the vim directory, there are these *.vmsg files that contain information regarding the VIM API (Virtual Infrastructure Management API) and searching within these files, you will find some hidden goodies.
 

Searching the task.vmsg file and within the alarm section, you will see a method name called setAlarmStatus:

alarm.AlarmManager.setAlarmStatus.label = "Set alarm status"
alarm.AlarmManager.setAlarmStatus.summary = "Sets the status of an alarm for an entity"

As you can see, there are other methods with respect to the alarmManager that is documented, but the method above is not documented in the vSphere API reference. We can take this information and further validate by using the vSphere MOB to see what parameters are required for this method.

By generating the proper URL, you can see the method is exposed and the required parameters to this function:

We can perform one additional confirmation to ensure that the method above is actually the method the vSphere Client is performing when resetting an alarm. The tool that we will use here is Onyx, yep, it is not only useful for PowerCLI but for all vSphere developers and administrators. I created a dummy alarm that would trigger if a VM is powered on or powered off and by running Onyx to capture the API calls when acknowledging and resetting an alarm.

Here is the output from Onyx:

As you can see, the two operations uses the AcknowledgeAlarm and the hidden SetAlarmStatus API method. We now can definitively say that the above hidden API method is being used and now we need to figure out how we can incorporate this method into the existing vSphere SDK for Perl.

In this example, I will be working with the vSphere SDK for Perl found on vMA 4.1, but the same set of changes will apply to vCLI 4.x being installed on a Windows or Linux system. There are two client Perl stubs or Perl Modules that contains prototype and definition of a vSphere API method:

/usr/lib/perl5/5.8.8/VMware/VIM25Stub.pm (prototype definition)
/usr/lib/perl5/5.8.8/VMware/VIM25Runtime.pm (method definition)

We will first edit the VIM25Runtime.pm module and if you are on vMA, you will need to use 'sudo' to edit the file. We will add the SetAlarmStatus entry try similar the AcknowledgeAlarm method:

Next we will edit the VIM25Stub.pm and again it will be very similar to an existing method, but now we must populate the parameters based on what we discovered from the vSphere MOB:

Now we are ready to implement this method in a vSphere SDK for Perl Script. I quickly threw together a script that helps manages your vCenter alarms using the CLI by listing all active and triggered alarms in either a red or yellow state.


Download Script: alarmManagement.pl

Here is an example of listing all triggered alarms that are either in a red or yellow state:

Here is an example of acking and resetting the above alarm and utilizing the new hidden API method that was just implemented:

There you have it, a method that was once hidden is no longer =)

If you are interested in locating other hidden API goodies, take a look at this post on How to browse the internal vSphere APIs

Does SIOC actually require Enterprise Plus & vCenter Server?

After reading a recent blog post by Duncan Epping, SIOC, tying up some loose ends, I decided to explore whether or not VMware's Storage I/O Control feature actually requires an Enterprise Plus license and vCenter Server. To be completely honest, Duncan's article got me thinking but it was also my recent experience with VMware's vsish and the blog post I wrote What is VMware vsish? that made me think this might be a possibility. vsish is only available on ESXi 4.1 within the Tech Support Mode, but if you have access to debugging rpm from VMware, you can also obtain vsish for classic ESX.

Within vsish, there is a storage section and within that section there is a devices sub-section which provides information regarding your storage devices that includes paths, partitions, IO statistics, queue depth and new SIOC state information.

Here is an example of the various devices that I can view on an ESXi 4.1 host:

~ # vsish

/> ls /storage/scsifw/devices/
t10.F405E46494C45400A50555567414D2D443E6A7D276D6F6E4/
mpx.vmhba1:C0:T0:L0/
mpx.vmhba32:C0:T0:L0/

Here is an example of various properties accessible to a given storage device:

/> ls /storage/scsifw/devices/t10.F405E46494C45400A50555567414D2D443E6A7D276D6F6E4/
worlds/
handles/
filters/
paths/
partitions/
uids/
iormInfo
iormState
maxQueueDepth
injectError
statson
stats
inquiryVPD/
inquirySTD
info

In particular, we are interested in iormState and you can see the value by just using the cat command:

/> cat /storage/scsifw/devices/t10.F405E46494C45400A50555567414D2D443E6A7D276D6F6E4/iormState
1596

This value may not mean a whole lot and I have seen this as the default value when SIOC is disabled as well as 2000 from my limited set of tests. Now, since we can access these particular SIOC parameter, I wanted to see how this value was affected when SIOC is enabled and disabled. To test this, I used the following VMware KB1022091 to enabling additional SIOC logging which goes directly to /var/log/messages with the logger tag "storageRM", this allows you to easily filter out SIOC logs via simple grep. 

For testing purposes, you can just enable the logging to level 2 which is more than sufficient to get the necessary output. You will perform the following command to change the default SIOC logging level from 0 to 2 using Tech Support Mode:


~ # esxcfg-advcfg -s 2 /Misc/SIOControlLogLevel
Value of SIOControlLoglevel is 2

Now you will want to open a separate SSH session to your ESXi host and tail /var/log/messages to monitor the SIOC logs:

~ # tail -f /var/log/messages | grep storageRM

Oct 10 18:39:05 storageRM: Number of devices on host = 3
Oct 10 18:39:05 storageRM: Checked device t10.F405E46494C45400A50555567414D2D443E6A7D276D6F6E4 iormEnabled= 0 LatThreshold =30
Oct 10 18:39:05 storageRM: Checked device mpx.vmhba1:C0:T0:L0 iormEnabled= 0 LatThreshold =232
Oct 10 18:39:05 storageRM: Checked device mpx.vmhba32:C0:T0:L0 iormEnabled= 0 LatThreshold =30
Oct 10 18:39:05 storageRM: rateControl: Current log level: 2, new: 2
Oct 10 18:39:05 storageRM: rateControl: Alas - No device with IORM enabled!

You should see something similar to the above. In my lab, it is seeing an iSCSI volume, local storage and CD-ROM and you will notice there is an iormEnabled flag and all three has SIOC disabled and the default latency threshold specified by LatThreshold, which is 30ms by default.

Now we know what these values are when SIOC is disabled, let's see what happens when we enable SIOC from vCenter on this ESXi 4.1 host. I am using evaluation license for the host which supports the Storage I/O Control from a licensing perspective. After enabling SIOC and using the default 30ms on my iSCSI volume, I took a look at the SIOC logs and saw that there were some changes in the logs:

~ # tail -f /var/log/messages | grep storageRM

Oct 10 18:48:56 storageRM: Number of devices on host = 3
Oct 10 18:48:56 storageRM: Checked device t10.F405E46494C45400A50555567414D2D443E6A7D276D6F6E4 iormEnabled= 1 LatThreshold =30
Oct 10 18:48:56 storageRM: Found device t10.F405E46494C45400A50555567414D2D443E6A7D276D6F6E4 with datastore openfiler-iSCSI-1
Oct 10 18:48:56 storageRM: Adding device t10.F405E46494C45400A50555567414D2D443E6A7D276D6F6E4 with datastore openfiler-iSCSI-1

As you can see now, SIOC is enabled and the iormEnabled flag has changed from 0 to 1. This should not be a surprise, now let's take a look at vsish storage property and see if that has changed:

~ # vsish -e get /storage/scsifw/devices/t10.F405E46494C45400A50555567414D2D443E6A7D276D6F6E4/iormState
1597

If you recall from the previous command above, the default value was 1596 and after enabling SIOC, the value has incremented by one. I found this to be an interesting observation and I tried a few other configurations including enabling SIOC on local storage and found that this value was always incremented by 1 if SIOC was enabled and decrement or kept the same if SIOC is disabled.

As you may or may not know, SIOC does not use vCenter, it is only required when enabling the feature and from this simple test, this looks to be the case. It is also important to note, as pointed out by Duncan in his blog post that the latency statistics is stored in .iormstats.sf file which is stored within each of the VMFS datastores that has SIOC enabled. Putting all this together, I hypothesize that Storage I/O Control could actually be enabled without an Enterprise Plus license and without vCenter Server.

The test utilized the following configuration:

• 2 x virtual ESXi 4.1 hosts licensed as free ESXi (vSphere 4.1 Hypervisor)
• 1 x 50GB iSCSI volume exported from Openfiler VM
• 2 x CentOS VM installed with iozone to generate some IO

Here is a screenshot of the two ESXi 4.1 free licensed hosts displaying the licensed version and each running CentOS VM residing on this shared VMFS iSCSI volume:

I configured vm1 which resides on esxi4-1 and set the disk shares to Low with default value of 500 and configured vm2 which resides on esxi4-4 and set the disk shares to High with default value of 2000:

I then ran iozone a filesystem benchmark tool on both CentOSes VM which is generating some amount of IO on the single VMFS volume shared by both ESXi 4.1 hosts:

I then view the SIOC logs on both ESXi 4.1 in /var/log/vmkernel and tail the IO statistics for the VMFS iSCSI datastore using vsish:

Note: The gray bar tells you which host you are viewing the data for which is being displayed by using GNU Screen. The first two screen displays the current status of SIOC which is currently disabled and the second two screen displays the current device queue depth which in my lab environment defaulted to 128, by default it should be 32 as I remember.

Now I enable SIOC on both ESXi 4.1 hosts using vsish and perform the command:

~ # vsish -e set /storage/scsifw/devices/t10.F405E46494C45400A50555567414D2D443E6A7D276D6F6E4/iormState 1597

Note: Remember to run a "get" operation to check the default value and you will just need to increment by one to enable SIOC. From my testing, the default value will either be 1596 or 2000 and you will change it to 1597 or 2001 respectively.

You can now validate that SIOC is enabled by going back to your SSH session and verify the logs:

As you can see, the iormEnabled flag has now changed from 0 to 1, which means SIOC is now enabled.

If you have running virtual machines on the VMFS volume and SIOC is enabled, you should now see a  iormstats.sf latency file stored in the VMFS volume:

After awhile, you can view IO statistics via vsish to see what the device queue is currently configured to and slowly see the throttle based on latency. For this particular snapshot, you can see the vm1 was configured with "high" disk share and vm2 was configured with "low" disk share and there is a large queue depth for the very bottom ESXi host versus the other host which only has a smaller queue depth.

Note: During my test I did notice that the queue depth was dramatically decreased from 128 and even from 32 to single digits. I am pretty sure it was due to the limited amount of resources in my lab that some of these numbers were a little odd.

To summarize, it seems that you can actually enable Storage I/O Control without the use of vCenter Server and an Enterprise Plus license, however, this requires the use of vsish which is only found on ESXi 4.1 and not on classic ESX 4.1. I also found that if you enabled SIOC via this method and joined your host to vCenter Server, it is not aware of these changes and marks SIOC as disabled even though the host actually has SIOC enabled. If you want vCenter to see the update, you will need to enabled SIOC via vCenter.

I would also like to thank Raphael Schitz for helping me validate some of my initial findings.

Update: I also found hidden Storage I/O Control API method called ConfigureDatastoreIORMOnHost which allows you to enable SIOC directly on the host, which validates the claim from above that this can be done directly on an ESX or ESXi host.

How to run a VM on Dropbox Storage

In my previous blog post How to backup VMs in ESX onto Dropbox, I showed you how to backup your virtual machines on ESX to your Dropbox account, but who said it should stop there? I realized after playing with Dropbox on ESX, I had a crazy idea of trying to run a virtual machine that was stored in my Dropbox account. Since I am using the free Dropbox account, I have a maximum of 2GB of online storage and decided to spin up a tiny Linux virtual machine and upload it to my Dropbox account. Not surprised, I was able to register the virtual machine and fire it up and it worked like a charm.

Now, this got me thinking .... if I can run a virtual machine from Dropbox via ESX, could I get another ESX host to see this same Dropbox account? The answer is YES! One feature of Dropbox is the ability to access your files across multiple devices: Windows, Linux, Mac OSX desktop, iPhone, iPad, Andriod and of course ESX. For demonstration purposes, I created two virtual ESX 4.1 hosts called west and east and authorized both hosts to access my Dropbox account.

Here is a screenshot of the two ESX hosts via "My Computers" tab on Dropbox account:

Here is a screenshot of the small 1GB Linux Debian virtual machine I created and uploaded to my Dropbox account:

Here is a screenshot of both ESX hosts (east and west) accessing the directory of the small Linux virtual machine:

I registered the virtual machine on the west coast ESX host and here is a side by side screenshot of both vSphere Clients:

As you can see, the virtual machine is powered on and running. If we take a look at where the virtual machine configuration file and disks is stored, you will see it is currently on the Dropbox account accessed by "west" coast ESX host:

Here is a screenshot showing the registration of the small Linux VM on west and there is currently nothing registered on east:

Let's say we had a failure on the west coast ESX host and we need to access this virtual machine via east coast ESX host. I registered the virtual machine and powered it on and now the system is back up and running again:

Again, we can see the configuration and virtual disk is now being accessed from the Dropbox account by the east coast ESX host:

We also confirm that the virtual machine is now registered on the east coast ESX and there is nothing running on the west coast ESX host:

As you can see from this simple demonstration, you can easily run a virtual machine and have it accessible via multiple ESX host, though I would not recommend you go out and start putting your production VMs on Dropbox, but it is an interesting idea for cloud storage ;)

Note: While testing, I found the synchronization process between the two ESX host to be slightly off and were not always up to date when a change was made. I had to restart the Dropbox daemon several times after I created the virtual machine for the other ESX host to see the updated files. I am pretty sure it is an issue with the daemon versus dropbox, since I can see the files updated on the web browser immediately. I would be very careful to ensure that only one host is accessing the files, else you could see some discrepancies.

How to backup VMs in ESX onto Dropbox

I previously wrote about backing up virtual machines directly from ESX and ESXi onto both Amazon S3 and Google Storage, but there was actually a third online file hosting company that I wanted to get working, Dropbox. Dropbox is a relatively new file hosting company that launched back in 2008 and has gain popularity for its ease of use and ability to access and share files across multiple devices. While researching Dropbox initially, I discovered there was a python-based CLI which I was hoping would install and function on ESX(i). This unfortunately did not pan out due to the various python library dependencies including a newer version of python.

While scouring the web, I recently found out that Dropbox actually released a Linux client binary and I thought I'd kick the tires and see if I could get it running on ESX(i). After a few minutes of testing, I found out that it was possible to get it running on classic ESX, but there are still certain python dependencies that prevent the Dropbox client to run on ESXi.

Before you begin, you will need to sign up for a free Dropbox account. With the free account you automatically get 2GB of free online storage, if you want more, you can pay for up to 100GB of online storage. The following has been validated on ESX 4.1, I have not tested this on any other ESX version and your results may vary. ESXi is not supported as mentioned earlier.

1. Download the latest Dropbox Linux Client here.

2. You will need to upload the Dropbox tar ball to your ESX host, you can use scp on UNIX/Linux or winSCP if you are using a Windows system.

3. You should not have the tar ball file sitting in the root directory of your ESX host:

4. You will now need to extract the contents of the tar ball, by running the following:

tar -zxvf dropbox-lnx.x86_64-0.7.110.tar.gz

5. You should now have a hidden directory called .dropbox-dist in your current working directory:

6. If you are starting the Dropbox client for the first time, it will default to using home directory of the current user to access your Dropbox share. In our case, it will be stored as /root/Dropbox which is probably not what you want. We will actually update our home directory to point to a VMFS volume path, by setting the HOME environmental variable:

As you can see, now our new home directory is set to a VMFS datstore. Once the Dropbox client starts, it will create a 64bit encoded string of the path which is stored in a configuration file once you have authorized the addition of this system to your Dropbox account.

Note: Once the default Dropbox path is set, you need to ensure that environmental HOME dir is always set to the one you specified above, else when you start Dropbox it will think it is a new setup.

7. We will now start the Dropbox daemon and ensure that we run it in the background:

If you have successfully started the Dropbox daemon, a unique URL will be generated based on your system which is used to authorize this system to access your Dropbox account. Take the URL and paste it into a browser, it should ask you to login to authorize the system.

8. If this is the first time you are using Dropbox, once you have signed in, you will be brought to the files tab in which all the folders and files that are currently accessible to you. By default, you will have a Public and Photos folder in which both are empty:

9. If you go back to your ESX host, you will now notice some output regarding nautils, you can ignore this error as the packages are not required for functionality:

10. Now, if you remember when we set the home directory to trick Dropbox to put folders under a VMFS datastore, you will now see a new directory called Dropbox which will contain the Public and Photos folder you saw on your web browser:

11. For demo purposes, I created a Backup directory in Dropbox root folder on the ESX host which will be visible from your web browser:

I then created a dummy 1MB VMDK in the same folder as if you were copying a VM to Dropbox account:

You can now go back to your web browser and see the VMDK file that was just created in the Backup directory:

There you have it, you can now transfer files or backup your virtual machines from your ESX host to your Dropbox account.

Earlier I mentioned that there are configuration files that tells the Dropbox client that this system is authorized to connect to your Dropbox account and it stores both the system ID along with the Dropbox path. If you do a long listing in your VMFS volume that was used to store your Dropbox folder, you will notice a hidden directory called .dropbox:

There are two database files, dropbox.db that contains the files and folder structures as it is being synced down to the host and host.db which contains the system's ID and Dropbox path which is encoded as 64bit string. You can decode and verify the path by using this website: http://webnet77.com/cgi-bin/helpers/base-64.pl

Here is what the host.db file looks like:

The second line contains the Dropbox path and you should not try to edit this file manually as it may cause synchronization issues. If you look at the Dropbox documentation, there is a python script that allows you to change the path but it requires sqlite3 to be available which is not available by default on ESX.