virtuallyGhetto

vSphere 4

Nested Virtualization Resources

by 7 Comments

Here is a consolidated page on all the articles that I have written about the Nested Virtualizatoin (nested ESXi, Hyper-V, etc) and all the goodies that are "Not Supported".

vSphere / vCloud 5.1

vSphere / vCloud 5.0

Additional Info/Tips/Tricks/

Automating the Search for Lopsided Bootbanks in ESXi 4.x

by 1 Comment

I read an interesting article this weekend about lopsided bootbanks in ESXi and as Kyle states, the impact of this is very minor. If you would like to check to see if you would be impacted by this and remediate prior to upgrading from ESXi 4.x to 5.x, you can use the following vSphere SDK for Perl script findLopsidedBootbanks.pl which was modified from an existing script (queryDiskParition.pl) that allows you to query for partitions on a disk.

Note: Remediation is only necessary if you plan on using VUM (Update Manager) to upgrade from ESXi 3.x/4.x to 5.x as it expects both bootbanks to be the same size, else a scripted install/upgrade will extend bootbank1 and ensure it matches bootbank2 size. 

Here is an ESXi 3.5 host that was directly upgraded to ESXi 4.0 and as you can see from the partition layout, bootbank1 (partition 5) and bootbank2 (partition 6) differ in size.

Using the findLopsidedBootbanks.pl you can connect to either vCenter server or directly to an ESXi host and it will check to see if there are any lopsided bootbanks.

Download: findLopsidedBootbanks.pl script here.

Hidden vCenter Debugging Performance Metrics

by 1 Comment

While extracting the new performance metrics in vSphere 5 for a blog post, I came across a metric type that I had never noticed before, vcDebugInfo. These performance metrics seems to deal with some of the internal performance/counters in vCenter such as lock statistics, MoRef (Managed Object Reference) counts, etc. Majority of these metrics are available in either collection level 1 or 4 which is the highest level containing all statistics. VMware's best practice is to only enable collection level 1 or 2, 3 and 4 should only be enabled under VMware supervision for debugging purposes which is most likely when some of these stats may come in handy. So be warn, these are probably not supported by VMware

I found it interesting that these metrics are hidden from the vSphere Client UI, but they can easily be extracted when going through the vSphere API. It is just amazing on all the goodies you can find when going through the APIs 🙂

Metric Stat Level Description
vcDebugInfo
maximum.millisecond.activationlatencystats 4 The latency of an activation operation in vCenter
minimum.millisecond.activationlatencystats 4 The latency of an activation operation in vCenter
summation.millisecond.activationlatencystats 1 The latency of an activation operation in vCenter
maximum.number.activationstats 4 Activation operations in vCenter
minimum.number.activationstats 4 Activation operations in vCenter
summation.number.activationstats 1 Activation operations in vCenter
maximum.millisecond.hostsynclatencystats 4 The latency of a host sync operation in vCenter
minimum.millisecond.hostsynclatencystats 4 The latency of a host sync operation in vCenter
summation.millisecond.hostsynclatencystats 1 The latency of a host sync operation in vCenter
maximum.number.hostsyncstats 4 The number of host sync operations in vCenter
minimum.number.hostsyncstats 4 The number of host sync operations in vCenter
summation.number.hostsyncstats 1 The number of host sync operations in vCenter
maximum.number.inventorystats 4 vCenter inventory statistics
minimum.number.inventorystats 4 vCenter inventory statistics
summation.number.inventorystats 1 vCenter inventory statistics
maximum.number.lockstats 4 vCenter locking statistics
minimum.number.lockstats 4 vCenter locking statistics
summation.number.lockstats 1 vCenter locking statistics
maximum.number.lrostats 4 vCenter LRO statistics
minimum.number.lrostats 4 vCenter LRO statistics
summation.number.lrostats 1 vCenter LRO statistics
maximum.number.miscstats 4 Miscellaneous statistics
minimum.number.miscstats 4 Miscellaneous statistics
summation.number.miscstats 1 Miscellaneous statistics
maximum.number.morefregstats 4 Managed object reference counts in vCenter
minimum.number.morefregstats 4 Managed object reference counts in vCenter
summation.number.morefregstats 1 Managed object reference counts in vCenter
maximum.number.scoreboard 4 Object counts in vCenter
minimum.number.scoreboard 4 Object counts in vCenter
summation.number.scoreboard 3 Object counts in vCenter
maximum.number.sessionstats 4 The statistics of client sessions connected to vCenter
minimum.number.sessionstats 4 The statistics of client sessions connected to vCenter
summation.number.sessionstats 1 The statistics of client sessions connected to vCenter
maximum.number.systemstats 4 The statistics of vCenter as a running system such as thread statistics and heap statistics
minimum.number.systemstats 4 The statistics of vCenter as a running system such as thread statistics and heap statistics
summation.number.systemstats 1 The statistics of vCenter as a running system such as thread statistics and heap statistics
maximum.number.vcservicestats 4 vCenter service statistics such as events, alarms, and tasks
minimum.number.vcservicestats 4 vCenter service statistics such as events, alarms, and tasks
summation.number.vcservicestats 1 vCenter service statistics such as events, alarms, and tasks

How to Add a Splash of Remote Color to ESXi Shell

by 6 Comments

This morning I noticed a very interesting retweet by fellow vExpert Wil van Antwerpen from another vExpert: Richard Cardona (You may know him as rcardona2k on the VMTN Community Forums) about a neat little trick with the use of remote ESXi Shell (previous known as remote TSM).

For those of you who login remotely via SSH to the ESXi Shell (previously known as unsupported mode and Tech Support Mode) know that you can run the DCUI utility remotely by just typing "dcui". The remote DCUI works just like it does using the direct console, with the exception of displaying the famous yellow and black screen that we are familiar with.

Richard came upon a neat little trick by setting the terminal type to "linux" from the default "xterm" that the yellow and black can be enabled when using the remote DCUI.

Before launching DCUI utility, you will need to run the following command on the ESXi Shell:

export TERM=linux

Next you will just type "dcui" and hit enter

Here is an example of running remote DCUI in color on ESXi 5

Here is an example of running remote DCUI in color on ESXi 4.1

Note: As you can see this is not a new trick in vSphere 5, but has been there since 4.x days but one big change with vSphere 5 is the full resolution of DCUI which many have complained about in the past.

If you are interested in other ways of customizing the DCUI, take a look at this blog post How to add a splash of color to ESXi DCUI Welcome Screen

Don't forget to play some cool soundtrack music when using the DCUI 😉

vSphere ESX 4.0 – Crash VM Bug?

by Leave a Comment

We recently discovered an anomaly while backing up one of our development VMs using ghettoVCB.sh. When attempting to back up this powered on VM, the backup was successful however oddly, we were left with a powered off VM immediately following the first VMDK clone operation. After some investigation, we found that the problematic VM contained virtual disks spread across two datastores with dissimilar blocksizes (1MB and 2MB).

The VM configuration alongside its main OS disk was stored on the datastore with a 1MB blocksize while it’s data disk (>256 GB) resided on the other datastore which was initialized with a 2MB blocksize. We came to the conclusion that this might have had something to do with the VM configuration residing on a datastore with a blocksize that was smaller than what is needed for the larger VMDK (which was on a datastore with an ample blocksize). Manually snapshotting this VM apparently fails however different behavior was experienced when the commands are executed from a script.

Believing that this was a corner case, we decided that it was best practice to keep all VMFS volume block sizes consistent. This was to be remediated at a later time.

Today we noticed a blog post http://www.yellow-bricks.com/2009/08/24/vsphere-vm-snapshots-and-block-size/ from Duncan Epping regarding the snapshot issue. This may not be a corner case as we thought so we wanted to share this finding with everyone.

If you have a similar configuration from above, it is guaranteed that the VM will crash if you run a script that tries to take a snapshot of the described VM and then subsequently exports the VMDK using vmkfstools.

Here is a video displaying the symptoms decribed from above:
http://engineering.ucsb.edu/~duonglt/vmware/crashVM

Output from script execution on VM: Quentin:

[[email protected] ~]# ./crashVM.sh Quentin /vmfs/volumes/dlgCore-NFS-bigboi.VM-Backups/Quentin-clone.vmdk
thinking, give me a few ...
The power state of VM: "Quentin" is On
Extracted VmId and locating VM configured datastore (which should live on a smaller VMFS block size)
Located VMDK: "/vmfs/volumes/himalaya-local-SAS.Savvio/Quentin/Quentin.vmdk"
Trying to create snapshot ... (this should fail)
Create Snapshot:
Trying to vmkfstools copy "/vmfs/volumes/himalaya-local-SAS.Savvio/Quentin/Quentin.vmdk" (this should REALLY fail! right?)
Destination disk format: VMFS zeroedthick
Cloning disk '/vmfs/volumes/himalaya-local-SAS.Savvio/Quentin/Quentin.vmdk'...
Clone: 9% done.

Download crashVM.sh script