We recently discovered an anomaly while backing up one of our development VMs using ghettoVCB.sh. When attempting to back up this powered on VM, the backup was successful however oddly, we were left with a powered off VM immediately following the first VMDK clone operation. After some investigation, we found that the problematic VM contained virtual disks spread across two datastores with dissimilar blocksizes (1MB and 2MB).
The VM configuration alongside its main OS disk was stored on the datastore with a 1MB blocksize while it’s data disk (>256 GB) resided on the other datastore which was initialized with a 2MB blocksize. We came to the conclusion that this might have had something to do with the VM configuration residing on a datastore with a blocksize that was smaller than what is needed for the larger VMDK (which was on a datastore with an ample blocksize). Manually snapshotting this VM apparently fails however different behavior was experienced when the commands are executed from a script.
Believing that this was a corner case, we decided that it was best practice to keep all VMFS volume block sizes consistent. This was to be remediated at a later time.
Today we noticed a blog post http://www.yellow-bricks.com/2009/08/24/vsphere-vm-snapshots-and-block-size/ from Duncan Epping regarding the snapshot issue. This may not be a corner case as we thought so we wanted to share this finding with everyone.
If you have a similar configuration from above, it is guaranteed that the VM will crash if you run a script that tries to take a snapshot of the described VM and then subsequently exports the VMDK using vmkfstools.
Here is a video displaying the symptoms decribed from above:
http://engineering.ucsb.edu/~duonglt/vmware/crashVM
Output from script execution on VM: Quentin:
[[email protected] ~]# ./crashVM.sh Quentin /vmfs/volumes/dlgCore-NFS-bigboi.VM-Backups/Quentin-clone.vmdk
thinking, give me a few ...
The power state of VM: "Quentin" is On
Extracted VmId and locating VM configured datastore (which should live on a smaller VMFS block size)
Located VMDK: "/vmfs/volumes/himalaya-local-SAS.Savvio/Quentin/Quentin.vmdk"
Trying to create snapshot ... (this should fail)
Create Snapshot:
Trying to vmkfstools copy "/vmfs/volumes/himalaya-local-SAS.Savvio/Quentin/Quentin.vmdk" (this should REALLY fail! right?)
Destination disk format: VMFS zeroedthick
Cloning disk '/vmfs/volumes/himalaya-local-SAS.Savvio/Quentin/Quentin.vmdk'...
Clone: 9% done.
Download crashVM.sh script
Thanks for the comment!