Community stories of VMware & Apple OS X in Production: Part 7

Company: Fortune 150 (Retail)
Software: vSphere + vSphere Replication
Hardware: Apple Mac Pro

[William] - Hi Vitaliy, thank you for reaching out and wanting to share your experiences with the community on managing a VMware and Apple OS X infrastructure. Can you tell us a little bit about yourself and what you currently do?

[Vitaliy] - I am a Senior Systems Analyst for a Fortune 150 company that wishes to remain anonymous (aka I do not have legal clearance to use the company name). I am part of a team that is responsible for providing IT infrastructure for many creative and marketing applications -- think pre-press and advertising.

[William] - Can you provide us some details about the VMware and OS X infrastructure that you’re supporting? Software/Hardware specs that you decided to go with and the workload characteristics?

[Vitaliy] - Prior to virtualization we were running two dozen Xserves with OS X 10.6 running a wide range of applications from Open Directory to custom in-house scripts. We have virtualized the whole environment with just 4 Mac Pro machines, each machine has 12 cores and 64GB of memory giving us a total of about 128GHz and 256GB of memory.

We have exhausted all the PCI-X slots on the Mac Pro's by adding two dual port network cards and a dual port HBA. As a result we have two redundant management, data, and vMotion ports on each machine. Oh, one thing worth mentioning is that VMware officially only supports 32GB of memory per Mac Pro but we have been running 64GB with no issues. For the past year we have been running vSphere 5.1 and just upgraded to 5.5 last week.

We have been using HP 3PAR SAN for our storage back-end and over the last couple of weeks we have migrated to an Oracle SAN. The whole process was completely seamless and transparent to the users thanks to VMware.

Here is a picture of the Mac Pro setup courtesy of Vitaliy:

mac-pro-vitaliy
[William] - Wow, that’s great to hear you’ve been able to really push the Mac Pro’s. You must have been happy to be able to consolidate all those Xserves! What was your approach for virtualizing OS X from the physical Xserve to Mac Pro? Did you rebuild or leverage some type of V2V?

[Vitaliy] - We decided to rebuild from scratch. We were running an outdated version of OS X 10.6 and all the applications running on top of that were just as old.

[William] - Can you talk to how you provision your OS X Virtual Machines and Applications and how it gets to the end users? Do users get their own systems or is this a shared infrastructure?

[Vitaliy] - It's a shared infrastructure, generally a VM is dedicated to a particular application. We created a "base VM" that has basic settings like power/energy saver settings, local accounts, monitoring software, etc. preconfigured and whenever we need a new virtual machine we simply clone it and change the hostname and IP address on the new VM. Perhaps a template would've been a cleaner solution but this is what we do. We are currently looking into automating configuration with either Puppet or Casper.

When we initially rolled out a couple of OS X virtual machines we noticed that CPU usage on the VMware cluster spiked up to almost a 100% while the virtual machines were idle. It turned out that the default OS X screensaver uses GPU power to generate that flare effect and because not enough GPU memory was available it resorted to using up all the CPU. Disabling the screensaver or switching to a text based one quickly fixed that issue ...

[William] - Thanks for the excellent tip on OS X screensaver, this is a handy one to know about! How do you go about monitoring the Mac Pro infrastructure? What’s the process for replacing failed hardware components and have you had any challenges with this?

[Vitaliy] - We treat it the same way as the rest of our environment -- each vSphere node and virtual machine is monitored via Nagios. We have this cluster running for little over a year now and luckily we have not had to deal with any hardware failure.

[William] - For your OS X Virtual Machines, do you have a need for backups or a DR strategy? If so, could you share some details on what you are currently using?

[Vitaliy] - We have a replica of our production environment at a remote disaster recovery site and we use vSphere Replication to copy all the VMs nightly. We also heavily rely on the snapshot feature prior to making any operating system or application changes, it has been a lifesaver so far.

[William] - Vitaliy, I want to say thank you very much for taking some time out of your super busy schedule to have a chat. Before I let you go, do you have any words of wisdom for others looking to manage a similar infrastructure? Anything you would do differently and any resources you have found useful in aiding you to support a VMware / OS X infrastructure?

[Vitaliy] - Speak to your manager, legal department, or whoever is in charge about interpreting Apple EULA. I have heard of at least three different interpretations and all have legal implications. I am very happy with our environment and would not change a thing if I had to build it again. Your blog, virtuallyGhetto, has been a great resource as you are the only one talking about VMware products running on Apple hardware.

If you are interested in sharing your story with the community (can be completely anonymous) on how you use VMware and Mac OS X in Production, you can reach out to me here.

Want to issue a VAAI UNMAP operation using the vSphere Web Client?

Recently, I have seen several requests from both customers and partners wanting to be able to run the VAAI UNMAP operation from within the vSphere Web Client. For those of you not familiar with the VAAI UNMAP operation, I recommend you check out this blog post by my colleague Cormac Hogan. Today, the only way to issue the UNMAP operation is by using ESXCLI either remotely or in the ESXi Shell. There is currently not a vSphere API for this operation and therefore it would be difficult to build a native vSphere Web Client Plugin to provide this functionality.

Having said that, one way to provide this capability is through the use of a vCenter Orchestrator (vCO) Workflow which can remotely execute an ESXCLI command whether that is going through ESXCLI using a Linux jump box or through PowerCLI using a Windows jump box. Starting with vSphere 5.5, you can now extend the vSphere Web Client and attach a vCO Workflow to a vSphere Object and be able to execute the workflow right from the vSphere Web Client. This is great if you are already using vCO, but for those that are not, it can be somewhat complex to setup along with a steep learning curve depending on your experience.

Today, there was an exciting announcement from my Automation buddy, Alan Renouf for a new VMware Fling called PowerActions for the vSphere Web Client. This new Fling allows you to easily extend the vSphere Web Client in the following ways:

  • Access a PowerCLI console directly in the vSphere Web Client
  • Ability to run a context aware PowerCLI script directly from the vSphere Web Client

The prerequisite for setting up PowerActions is no different than vCO calling a PowerCLI script, you just need a Windows "jump-box" that has PowerCLI installed along with PowerActions. The added benefit, is that you do need to setup another piece of infrastructure like vCO if you are not already using it. This made setting up PowerActions extremely easy to setup and even I was able get it up and running in under 5minutes (minus a quick RTFM moment :)).

Given the number of inquiries regarding VAAI UNMAP operation via the vSphere Web Client, I thought that would be a great use case for my first PowerActions script! Below are the instructions on creating the VAAI UNMAP script for PowerActions:

Step 1 - Click on the PowerCLI Scripts option on the left hand side of the Object Navigator and then click on the "New Script" Icon. Select Datastore as the context aware object for the script.

unmap-command-in-vsphere-web-client-0
Step 2 - Provide a name and description for the script. Also make sure to select "Action".

unmap-command-in-vsphere-web-client-1
Step 3 - Copy and paste the following script from https://github.com/lamw/vghetto-scripts/blob/master/powershell/unmap-poweraction.ps1 inside the script window and then save the script. What the script does is takes the Datastore object and retrieves a list of ESXi hosts that has access to the Datastore and then randomly selects one of the host. This is required because ESXCLI operations on a per host level and we use that information to pass into Get-EsxCli cmdlet to issue the VAAI UNMAP operation.

Step 4 - To test the script, you just need to right click on a VMFS Datastore and click on PowerCLI->Execute a Script

unmap-command-in-vsphere-web-client-2
Note: Please be aware of the impact when running a UNMAP operation, you may want to run this on a non-production datastore for testing purposes or during off hours when your workload may not be as busy.

Step 5 - Select the VAAI UNMAP script you just created and once selected and you will be prompted to specify the number of VMFS blocks to unmap per iteration which is exactly the same input when manually ESXCLI.

Screen Shot 2014-09-17 at 10.30.09 PM
At this point, if everything was successful the VAAI UNMAP operation should begin and you can tail /var/log/hostd.log to see the UNAMP operation. Once completed, you should see the prompt return true.

As you can see, it was extremely easy to create my own PowerAction script that expose new functionality and making it available within the vSphere Web Client. I think this is going to be a pretty popular Fling and remember if this is something you would like to see officially in the product, be sure to leave a comment on the PowerAction for vSphere Web Client Fling page, product managers are listening! The only feedback I have is that I would love to see this get extended beyond just PowerCLI and into a generic script extension framework, just imagine the possibilities!

How to run Qemu & KVM on ESXi?

Last week I was asked whether ESXi could run the KVM hypervisor as a Virtual Machine (often referred to as Nested Virtualization). I personally have not used KVM before or run it on top of ESXi, but I have heard of many folks successfully virtualizing KVM as a Virtual Machine on top of ESXi. I figure since I have already written several articles on Nesting VMware ESXi, Microsoft Hyper-V and Xen on top of ESXi, I might as well also take a look at KVM!

Disclaimer: Nested Virtualization is not supported by VMware, please use at your own risk.

As mentioned already, I have not used KVM before and one thing I wanted to understand before trying to run it as a Virtual Machine is what the difference is between Qemu and KVM as I have heard both these terms used in-conjunction before. I found this post to be quite helpful in helping me understand the differences between Qemu, KQemu and KVM. I recommend a read if you are new to Qemu or KVM like I am.

From the article above, we now see that you can run either Qemu as a standalone system or KVM which is an accelerator that runs on top of Qemu. With this, I will now demonstrate how you can run Qemu as well as KVM as Virtual Machine on top of ESXi. In the example below, I have selected the latest Ubuntu release (14.04.1) to run both Qemu and KVM.

To be able to run either Qemu or KVM on top of ESXi, you just need to create a Virtual Machine running Virtual Hardware 10 and enable the (VHV) Hardware Assisted Virtualization feature which available in the vSphere Web Client as seen in the screenshot below:

qemu-and-kvm-on-esxi-0
To validate that you have properly enable VHV, you run the following command:

egrep '(vmx|svm)' /proc/cpuinfo

You should see some output when running the command, else it was not properly enabled as seen in the screenshot below:

qemu-and-kvm-on-esxi-1
Another command you can also run to check if the VM has been properly configure is the following:

kvm-ok

You should also see a message stating that KVM acceleration is possible:

qemu-and-kvm-on-esxi-2
Installing/Running Qemu on Ubuntu VM

Step 1 - Install Qemu by running the following command:

sudo apt-get -y install qemu

Step 2 - Download and extract a simple Linux Qemu image by running the following two commands :

wget http://wiki.qemu.org/download/linux-0.2.img.bz2
bunzip2 linux-0.2.img.bz2

Note: You can also find other Qemu images here.

Step 3 - Launch the Linux image by running the following command:

qemu-system-x86_64 linux-0.2.img -curses

qemu-and-kvm-on-esxi-3

Installing/Running KVM on Ubuntu VM

Step 1 - Install KVM by running the following command:

sudo apt-get install qemu-kvm libvirt-bin

Step 2 - Install bridge networking components by running the following command:

sudo apt-get install bridge-utils

Step 3 - Add a bridge interface by running the following command:

sudo brctl addbr br0

Step 4 - Append the following configuration to /etc/networking/interfaces:

Step 5 - Restart the networking service by running the following command:

sudo /etc/init.d/networking restart

Step 6 - Next, we will need to create a VM which is based on an XML file. Here is what my VM definition looks like:

Step 7 - You will need to make some changes to the XML file such as the location of the Ubuntu ISO (which is required to boot the VM) as well as the lP Address of the VNC service for console access to VM and lastly the UUID identifier of the VM which can be generated by running 'uuid' command.

Step 8 - Before we can create our VM, we will need to create the Image file by running the following command:

qemu-img create -f qcow2 /home/lamw/alice.img 5G

Step 9 - We are now ready to initialize and boot up our VM by running the following command:

virsh --connect qemu:///system create alice.xml

Step 10 - We can get information about the VM we just created by running the following command:

virsh --connect qemu:///system dominfo alice

qemu-and-kvm-on-esxi-4
Step 11 - Lastly, we can connect to the VM console using a VNC client and if you modified the XML definition of the VM to listen on the public address of your Ubuntu host VM, then you should be able to see the installer of ISO bootup (in my case, it's Ubuntu Desktop Edition) as seen in the screenshot below:

qemu-and-kvm-on-esxi-5
Note: The instructions have been modified from this site here http://xmodulo.com/2014/02/use-kvm-command-line-debian-ubuntu.html There were a couple of errors that I ran into and hence re-writing

Additional Resources:

  • https://gmplib.org/~tege/qemu.html

 

Quick Tip - Automate the enabling of the Customer Experience Improvement Program (vTelemetry) in VCSA

The VMware Customer Experience Improvement Program is a new feature that was introduced with the latest release of vCenter Server 5.5 Update 2 (vCenter Server for Windows & the VCSA). This feature provides the following per the documentation:

If you choose to participate in the Customer Experience Improvement Program (Program), VMware receives anonymous information to improve the quality, reliability, and functionality of VMware products and services. VMware wants to understand better your vSphere deployment and business needs, and improve VMware response to customer requirements. You can choose to participate in the Program for vSphere at any time.

In my opinion, this has been needed for quite some time now. If you have ever installed any consumer based software, there is usually an option that allows customers to provide basic telemetry data back to the vendor so that they can better understand how the product is being used and more importantly leverage that data to help improve the product and features future.

The process of collecting basic telemetry (sometimes also known as phone home) is not a new concept in the Enterprise. In fact, for those of you who manage a storage array, this has been a standard practice for many many years now where every night, the storage array sends back a variety of telemetry data that may include performance information, utilization, logs, etc. to the vendors HQ. This data is then analyzed and the vendor maybe able to sport trends of a potential issue and proactively alert customers to take action before a problem even arises. Michael White has also recently written about the topic here which I recommend a read as well.

There are four categories of data that is being collected:

  • esxcfg-info.xml
  • Extension.json
  • AboutInfo.json
  • performance-stats.txt

If you wish to learn more about what is being collected and how to view the data before it is sent, please take a look at the documentation here.

One thing I had noticed when deploying the latest VCSA 5.5 Update 2 is that there is now an option to enable the Customer Experience Improvement Program and of course I was interested in automating this configuration as part of my VCSA deployment script.

automate-telmetry-customer-improvement-program-0
Taking a look at the logs, I found that there is a new option that has been introduce in /usr/sbin/vpxd_servicecfg called telemtry:

Telemetry data collection modes:
read         : read and return the current status of the collector
enable       : enables the telemetry data collection
disable      : disables the telemetry data collection

To enable the Customer Experience Improvement Program as part of the VCSA setup, you must enable it after vpxd (vCenter Server) has started. Here is the modified VCSA configuration shell script:

If you decide not to enable this feature during the initial deployment or if you have upgraded from an existing vCenter Server, this feature can also be enabled after the fact. To do so, you will need to login to your vCenter Server using the vSphere Web Client and under the Settings tab of your vCenter Server, there is an option to enable or disable the Customer Experience Improvement Program

automate-telmetry-customer-improvement-program-1
Note: When enabling or disabling the Customer Experience Improvement Program, a restart of vCenter Server is not necessary.

Hopefully customers will see the benefit and value in joining the VMware Customer Experience Improvement Program and over time, I think you will start to see some really neat benefits for those who participate in this program.

How to build custom ESXi ISO for Apple Mac Mini?

For those of you who own an Apple Mac Mini 6,2 may recall some of the, lets call them "challenges" on getting ESXi to run on the Mini. These challenges range from known SMC issues from Apple to missing or updated Broadcom tg3 network drivers. Though there workarounds for these issue, the process was quite complex. I took it upon myself to help simplify it by building custom ESXi ISO's for most of the major ESXi releases so that users could simply install ESXi as they normally would and by-pass all the complexity.

This has worked for the majority of folks but I have received several requests for those that may not be comfortable with just downloading a random ISO on the internet, which I can fully understand. The other reason is that some folks would like to build their own custom ISO and include other drivers/packages and others are just interested in the process. This has been on my to-do list for awhile but it was finding the time to document the process but also I normally like to take it a step further and see how I can make it even more simple :)

Disclaimer: Running ESXi on an Apple Mac Mini is not officially supported by VMware, please use at your own risk

With the recent release of vSphere 5.5 Update 2, I thought this would be the perfect opportunity to show how you can build your own custom ESXi ISO to run on the Apple Mac Mini 6,2.

Note: Earlier versions of Mac Mini should work fine for the most part without additional modifications.

Before I get started, I would also like to mention that several of the "challenges" such as having an updated Broadcom tg3 drivers have been fixed in the latest ESXi 5.5 Update 2 release, so out of the box you will be able to see the on-board network device working as expected and Ethernet Thunderbolt will also be functional if you are using that device with no additional drivers being required. I have been able to successfully install the default out of the box ESXi 5.5 Update 2 ISO from VMware on my Apple Mac Mini 5,3 without any additional changes.

Here is the process for building your own custom ESXi ISO for your Mac Mini:

Step 1 - Download the ESXi ISO you wish to work with

Step 2 - You will need access to a Linux system (recommend CentOS) that has mkisofs utlity, which is used to author an ISO

Step 3 - Download my custom.tgz which will automatically handle the SMC issue for Apple Mac Mini 6,2

Step 4 - Download my ghettotize-esxi-iso.sh which is a shell script that will automatically take an ESXi ISO and author a new ISO containing the fixes. The script is pretty straight forward and you can take a look at the script for all the details.

Here is an example of running the script against the latest ESXi 5.5 Update 2 ISO:

build-custom-esxi-iso-for-mac-min-0
As you can see at the end of the script, you should get a new authored ISO with a -NEW in the filename:

build-custom-esxi-iso-for-mac-min-1
Once you have the new ISO, you can then take that and load that onto a USB device. I like using unetbootin which is a handy utility that is supported on all platforms and creates a bootable USB device with the ISO provided. As you can see the process is pretty straight forward and though it took a bit of "experimentation" on my end to make it completely seamless, you can see there is too much to the process in general.

For those of you who rather not go through the process and just wants the goods and trusts me, I have also built out a new updated image for latest ESXi 5.5 Update 2 release. I did not have access to a Mac Mini 6,2 to confirm the build, but I am pretty confident it should work as I tested an earlier 5.5 Update 2 build several months back. You can find the latest ISO along with past ones I have created below.