New VMware Fling to improve Network/CPU performance when using Promiscuous Mode for Nested ESXi

I wrote an article awhile back Why is Promiscuous Mode & Forged Transmits required for Nested ESXi? and the primary motivation behind the article was in regards to an observation a customer made while using Nested ESXi. The customer was performing some networking benchmarks on their physical ESXi hosts which happened to be hosting a couple of Nested ESXi VMs as well as regular VMs. The customer concluded in his blog that running Nested ESXi VMs on their physical ESXi hosts actually reduced overall network throughput.

This initially did not click until I started to think about this a bit more and the implications when enabling Promiscuous Mode which I think is something that not many of us are not aware of. At a very high level, Promiscuous Mode allows for proper networking connectivity for our Nested VMs running on top of a Nested ESXi VMs (For the full details, please refer to the blog article above). So why is this a problem and how does this lead to reduced network performance as well as increased CPU load?

The diagram below will hopefully help explain why. Here, I have a single physical ESXi host that is connected to either a VSS (Virtual Standard Switch) or VDS (vSphere Distributed Switch) and there are two portgroups one for my Nested ESXi VMs which has Promiscuous Mode enabled and another portgroup which does not and is connected to a regular VM. Lets say we have 1000 Network Packets destined for our regular VM, one would expect that the red boxes (representing the packets) will be forwarded to our regular VM right?

nested-esxi-prom-impact-0
What actually happens is shown in the next diagram below where every Nested ESXi VM gets a copy of those 1000 Network Packets on each of their vNICs even though they were not destined to the Nested ESXi VMs. This process of performing the shadow copies of the network packets and forwarding them down to the Nested ESXi VMs is a very expensive operation. This is why the customer was seeing reduced network performance as well as increased CPU utilization to process all these additional packets that would eventually be discarded by the Nested ESXi VMs.

nested-esxi-prom-impact-1
This really solidified in my head when I logged into my own home lab system which I run anywhere from 15-20 Nested ESXi VMs at any given time in addition to several dozen regular VMs just like any home/development/test lab would. I launched esxtop and set the refresh cycle to 2seconds and switched to the networking view. At the time I was transferring a couple of ESXi ISO’s for my kicskstart server and realized that ALL my Nested ESXi VMs got a copy of those packets.

nested-esxi-mac-learning-dvfilter-0
As you can see from the screenshot above, every single one of my Nested ESXi VMs was receiving ALL traffic from the virtual switch, this definitely adds up to a lot of resources being wasted on my physical ESXi host which could be used for running other workloads.

I decided at this point to reach out to engineering to see if there was anything we could do to help reduce this impact. I initially thought about using NIOC but then realized it was primarily designed for managing outbound traffic where as the Promiscuous Mode traffic is all inbound and it would not actually get rid of the traffic. After speaking to a couple of Engineers, it turns out this issue had been seen in our R&D Cloud (Nimbus) which provides IaaS capabilities to the R&D Organization for quickly spinning up both Virtual/Physical instances for development and testing.

Christian Dickmann was my go to guy for Nimbus and it turns out this particular issue has been seen before. Not only has he seen this behavior, he also had a nice solution to fix the problem in the form of an ESXi dvFilter that implemented MAC Learning! As many of you know our VSS/VDS does not implement MAC Learning as we already know which MAC Addresses are assigned to a particular VM.

I got in touch with Christian and was able to validate his solution in my home lab using the latest ESXi 5.5 release. At this point, I knew I had to get this out to the larger VMware Community and started to work with Christian and our VMware Flings team to see how we can get this released as a Fling.

Today, I am excited to announce the ESXi Mac Learning dvFilter Fling which is distributed as an installable VIB for your physical ESXi host and it provides support for ESXi 5.0, 5.1 & 5.5.

esxi-mac-learn-dvfilter-fling-logo
You can download the MAC Learning dvFilter VIB here vmware-esx-dvfilter-maclearn-0.1-ESX-5.0.vib or you can install directly from the URL.

To install the VIB, run the following ESXCLI command if you have VIB uploaded to your ESXi datastore:

esxcli software vib install -v /vmfs/volumes/<DATASTORE>/vmware-esx-dvfilter-maclearn-0.1-ESX-5.0.vib -f

To install the VIB from the URL directly, run the following ESXCLI command:

esxcli software vib install -v http://http://download3.vmware.com/software/vmw-tools/esxi-mac-learning-dvfilter/vmware-esx-dvfilter-maclearn-1.0.vib -f

A system reboot is not necessary and you can confirm the dvFilter was successfully installed by running the following command:

/sbin/summarize-dvfilter

You should be able see the new MAC Learning dvFilter listed at the very top of the output.

nested-esxi-mac-learning-dvfilter-2
For the new dvFilter to work, you will need to add two Advanced Virtual Machine Settings to each of your Nested ESXi VMs and this is on a per vNIC basis, which means you will need to add N-entries if you have N-vNICs on your Nested ESXi VM.

    ethernet#.filter4.name = dvfilter-maclearn
    ethernet#.filter4.onFailure = failOpen

This can be done online without rebooting the Nested ESXi VMs if you leverage the vSphere API. Another way to add this is to shutdown your Nested ESXi VM and use either the “legacy” vSphere C# Client or vSphere Web Client or for those that know how to append and reload the .VMX file as that’s where the configuration file is persisted
on disk.

nested-esxi-mac-learning-dvfilter-3
I normally provision my Nested ESXi VMs with 4 vNICs, so I have four corresponding entries. To confirm the settings are loaded, we can re-run the summarize-dvfilter command and we should now see our Virtual Machine listed in the output along with each vNIC instance.

nested-esxi-mac-learning-dvfilter-4
Once I started to apply this changed across all my Nested ESXi VMs using a script I had written for setting Advanced VM Settings, I immediately saw the decrease of network traffic on ALL my Nested ESXi VMs. For those of you who wish to automate this configuration change, you can take a look at this blog article which includes both a PowerCLI & vSphere SDK for Perl script that can help.

I highly recommend anyone that uses Nested ESXi to ensure you have this VIB installed on all your ESXi hosts!

Community stories of VMware & Apple OS X in Production: Part 5

Company: Artwork Systems Nordic A/S (AWSN)
Software: VMware vSphere
Hardware: Apple Mac Pro

[William] - Hi Mads, thank you for taking some time this morning to share with the community your past experiences managing a VMware and Apple OS X environment. Before we get started, can you introduce yourself and what you currently do?

[Mads] - My name is Mads Fog Albrechtslund, and I currently work as a vSphere Consultant for Businessman A/S Denmark. The reason for my current employment, is primarily a Mac based vSphere project I did at my former employer, Artwork Systems Nordic A/S also in Denmark. Before I became a vSphere Consultant, my primary job function was as a Mac Consultant, in which I have several Apple related certifications.

[William] - Could you describe what your vSphere project was about?

[Mads] - The vSphere Project, was that of virtualizing and consolidating the infrastructure of Artwork Systems Nordic A/S (AWSN). AWSN is a reseller of hardware and software to the graphical industry, thereby running a lot of Apple systems and software that require Mac OS X underneath.

When I started at the company in early 2009, there were around 8-10 servers, and only 9 employees. Every server was just a desktop Mac or PC, running multiple services at once, trying to use the hardware at best. I started by consolidating and somewhat standardizing all these machines, into a Rack cabinet.

But I still wanted to make it better, more flexible and faster to deploy new OS'es when they are needed. I also wanted to move away from running multiple services on a single OS. I started looking into virtualization around late 2010, before VMware even made vSphere compatible with the Mac's. And we started working with a competitor of VMware, which at the time was about to release a bare-metal hypervisor that was compatible with Mac hardware.

We invested time, money and hardware in that initial project, only to around 6 month later to find out that the vendor would drop that bare-metal software again.

[William] - Ouch! I guess that is one of the risks when working with a new company/startup. So what did you end up doing after the company dropped support for bare-metal support?

[Mads] - So when VMware release vSphere 5.0 which was compatible with Apple hardware, I asked my boss to try again. He said "Sure, go ahead…. but we don't have a lot of money to do this with". So I needed to make this project as cheap as possible.

What I ended up with was 3 Mac Pro's (2x 2008 and 1x 2009), which I got almost free from a customer, extra RAM (32GB in each Mac Pro), extra NIC's (4 NIC's in each Mac Pro), a Synology RS812+ NAS and VMware vSphere Essentials bundle.

Here is a picture of the 3 Mac Pros:

awsn-mac-pro
[William] - I too remember when VMware announced support for Apple Hardware with vSphere 5.0, that was a huge deal for many customers. Were there any performance or availability requirements that you had to take into considerations while designing this solution? Did all Virtual Machines run off of the NAS system or was it a mix between local and remote storage?

[Mads] - All VM's ran off the NAS over iSCSI. I did consider the availability of that design, but given the constraints of the money of the project, there was not much of a choice. I did not want to run the VM's on the local disks inside each Mac Pro, considering that if one Mac Pro died, I would not easily have the possibility to power-on that VM on another Mac Pro.

The performance of the NAS was not great, but good enough. After I left, the NAS was upgraded to a Synology DS1813+, and then using the old Synology RS812+ as a backup destination. The load on the VM's was light, as there only was 10 employees in the company, and most of the VM's was only for testing or designing solutions for the customers.

[William] - What type of Virtual Machines and applications were you running on the Mac Pros?

[Mads] - The 3 Mac Pro's are running around 20 VM's, where most of them are either OS X based or Linux Virtual Appliance's. My plan was to do one service per OS, to keep it as simple as possible. Almost all the OS X based VMs are running OS X 10.8 Mountain Lion. Some of them are just plain Client installations, but most of them have the Server app installed, to run Open Directory, DNS or File Server.

The Client installations are running specific software that the company sells, like graphical processing software from Enfocus or FTP software from Rumpus. There is also an older OS X based VM, running Mac OS X Server 10.6, which runs a special graphical procession software called Odystar from a company called Esko. This software only exist on Mac OS X, and it also requires a HASP USB dongle for its license. Most of the VM's are configured as low as possible, which for most is 1 vCPU and 2GB ram.

The Mail server for the company, is based on Kerio Connect software, which is also something that the company is a reseller of for its smaller graphical customers. That software exist either as a virtual appliance, a Windows install or a Mac based install. We ended up with choosing a Mac based installation, because we knew it better.

[William] - How did you go about monitoring the Virtual Machines as well as the underlying hardware? Any particular tools that you found worked well for your organization?

[Mads ] - We did not do much of monitoring, of neither the VMs or the hardware. I was onsite, and sitting almost beside the rack most of the time, so if there was any trouble either physically or virtual, I could fix it fast. I had configured email reporting in all the solutions that gave the option (vCenter Server, Synology NAS and some of the applications).

[William] - I know you had started this project back in 2010 and there was definitely a limited amount of hardware options to run Apple OS X VMs. Today, there are a few more options and if you were to do it again, would you have done anything differently? Would you still consider the Mac Pro (Tower) or look at potentially the newer Mac Pro (Black) or even the Mac Mini’s?

[Mads] - We did start out by looking at the Mac Mini's, but considering that we could only run 3 hosts because of the vSphere Essential license, we needed to get more RAM in each host, than the Mac Mini's could provide. The Tower based Mac Pro is still the best option for this installation, given that it is available for a reasonable price, runs more than 16GB ram and you can get 2x CPU sockets in each host.

The new black version of the Mac Pro, is especially not a good fit, primarily because of the price and because of the dual GPU's and only 1 CPU. I would love a Mac Mini with 32GB ram, that would properly fit perfectly, considering the advances in CPU technology over the 2008/2009 CPU's in the Mac Pro's currently running the environment.

[William] - Mads, thank you very much for spending your morning and sharing with us your experiences with running vSphere on the Mac Pros. You have provided a lot of good information that I know will surely help the VMware and Apple community. One final question before I let you go. Is there any tips/tricks you would recommend for someone looking to start a similar project? Any particular resources you would recommend people check out?

[Mads] - First of a big thanks to yourself, for provide great content on http://www.virtuallyghetto.com. I have also provided my own experiences both on my personal blog www.hazenet.dk and on businessman's company blog bmspeak.businessmann.dk

On my own blog, I have written about issues with screensavers in Mac OS X VM's and I have also written a long blog post about how make a never booted Mac OS X template VM, which don't have any UUID's set.

If you are interested in sharing your story with the community (can be completely anonymous) on how you use VMware and Mac OS X in Production, you can reach out to me here.

Support my challenge - Bike MS: Waves to Wine Ride 2014

CAN 2014 Bike - Event Details Banner
When I am not playing with the latest VMware products or exploring new technologies, one of my passions outside of work is cycling (road bike). Yesterday, I decided to take the plunge and signed up for the annual Bike MS: Waves to Wine Ride 2014 which takes place the weekend of September 20-21, 2014. The ride is to help bring both awareness as well as help raise funds towards curing MS (multiple sclerosis). I personally do not know of anyone with MS or have it myself, but I think it is a worthy cause to ride for and here is some more details regarding the disease from the website:

Multiple sclerosis is an unpredictable, often disabling, disease of the central nervous system that interrupts the flow of information within the brain, and between the brain and body. Millions of people are affected by MS and the challenges of living with its unpredictable symptoms, which range from numbness and tingling to blindness and paralysis. The progress, severity and specific symptoms of MS in any one person cannot yet be predicted, but advances in research and treatment are moving us closer to a world free of MS.

I have decided to challenge myself by taking on the century route (104 miles) which starts from San Francisco and goes all the way up to Sonoma County. This will be an interesting ride as it will be my first "official" century (accidentally did one about month ago from Sunnyvale to San Francisco) and will be my first time riding with large number of cyclists. I think it will be fun and quite challenging and I am really looking forward to it.

To participate in the event, I will need to personally raise a minimum of $350 USD (33 days left), which I have already donated a small amount myself. I am reaching out to greater Virtualization community to ask for your help and support. If you would like to support me and the National MS Society, please consider making a small donation here. I have also created a group called Team VMware, as one did not exists and others would like to join. I have also set an optimistic fundraising goal for myself and the team of $1,000 each. I will definitely be leveraging our VMware Foundation benefits to match whatever amount I have been able to raise. I hope you will share this with your colleagues, friends and family and assist me with this worthy cause! I also plan on renting a GoPro to capture my ride and the spectacular views as a token of my appreciation.

Click here to Donate!

Thanks for your considerations!

pyvmomi (vSphere SDK for Python) 5.5.0-2014.1 released!

I just saw an awesome update from Shawn Hartsock, a fellow VMware colleague. For those of you who do not know him, Shawn works in our Ecosystems and Solutions Engineering (EASE) organization and is the primary maintainer of VMware's pyvmomi (vSphere SDK for Python) open-source project. The pyvmomi project was open sourced since last December which I had written about here, it has received over 3K+ downloads and has a very active community. Much of this success has been due to the hard from Shawn fostering an active community around pyvmomi.

The announcement today from Shawn is a new release of pyvmomi at version 5.5.0-2014.1:

As mentioned earlier, the pyvmomi project is a very active project and Shawn is constantly engaging with users looking for feedback, suggestions or requests for new samples to build. If you are interested in vSphere Automation and would like to leverage Python, be sure to check out the pyvmomi Github repository! Lastly, if you have written some cool scripts/applications or would like to request specific sample scripts, be sure to send a pull request to Shawn as we would love to see more contributions and collaborations from the community!

Community stories of VMware & Apple OS X in Production: Part 4

Company: AutoTrader.com
Product: VMware vSphere
Hardware: Apple Mac Mini

[William] - Hi Chris, good afternoon. I know we have chatted a few times on Twitter before but for the folks that do not know you, can you quickly introduce yourself and what you do?

[Chris] - My name is Chris Nakagaki and I work for Autotrader.com as Sr. Systems Engineer. My current role involves day-to-day operations of VMware vSphere products in addition to defining best practices around the virtual infrastructure. Not to mention, help drive automation in my organization. Occasionally, I'll post something useful on my tech.zsoldier.com blog.

[William] - You had sent me an email after I published the first community story around how VMware leverages Mac Mini’s. I hear you are doing something pretty cool with the Mac Mini’s as well for your organization? Could you share some details on how your organization is using VMware and Mac Minis?

[Chris] - A couple of our subsidiary companies, in this case (vAuto and AutoTrader.com) essentially needed OS X VM's for QA testing of iOS applications, and general Mac browser testing. Rather than delivering individual Mac Mini to every developer and/or VMware fusion, etc. vAuto first approached us and we came up with this idea of clustering some Mac Mini's. They wanted to run ESXi on them to host relatively small VM's that could be centrally managed and accessed from any number of developers and with the Apple EULA, this was the only option due to the restriction. Besides that, it was just a really cool idea since we're all Mac/Apple fans anyways

[William] - That’s awesome, never seen a customer come up with both the request and a solution at the same time ;) Have you had any experiences running vSphere on the Mac Mini’s before? Any challenges you faced while exploring this solution?

[Chris] - Thankfully you (William Lam) had run into a lot of the problems for us already.  So it was really easy to setup using the custom ISO and VIB you created to put our little 'MacCloud' together. The other non-software aspect we ran into though was the fact that the Mac Mini's do not have an out of band management interface. So we are currently looking for some smart power supplies and/or iKVM so that we can actually place these in our 'real' datacenters.

[William] - Hey no worries, I rather be the guinea pig and get all the kinks out so customers like yourself can just enjoy the benefits of running vSphere and ESXi on Apple hardware! How large is the MacCloud right now and what is the current hardware and software configuration?

[Chris] - Our MacCloud is only 3 Mac Mini's right now as we're still kind of 'feeling' it out. But soon after vAuto started using it, the word got out and we setup some test systems for our AutoTrader.com developers. In addition, our client engineering group uses a tool called Casper to manage our Macs. He needed a Distribution Point, preferably a system that had AFP, so we set him up with one and he was able to use it to deliver updates/applications. And the I/O for VSAN is so good with the SSD's, it screams.

Each Mac Mini is the 'Server' version, i7@ 2.6GHz, 1 SSD (128GB), 1HDD (1TB) and 16GB memory using USB to boot into ESXI. For the software, we are currently using vSphere Enterprise Plus and the vCenter Server happens to be the VCSA. The MacCloud is also being monitored by vCOPs

Here is a picture of the front/back of the Mac Mini rack:

autotrader-mac-mini-1

autotrader-mac-mini-2
[William] - That’s amazing, it sounds like the environment is really satisfying your developers and I can see why word has spread. So, did I read that right? You are currently using VSAN on the Mac Mini’s!? How has the performance been and what made you decide to leverage VSAN?

[Chris] - The minute VSAN went into beta, that's all I could personally think about for my own home lab with the Mac Mini's. That just naturally translated when the business had a need and I could satisfy my curiosity in one fell swoop. So being that these workloads aren't heavy I/O, I haven't been all that concerned with it. The VM Storage Policies have been all left at default because I don't see a need to change right now.

My team and I are actively keeping a pulse on all the users of the VM's hosted on here though. vCOPs shows that everything is working efficiently, but we want to make sure that is translating to a good user experience. The Casper DP is one in particular that I'm curious about since the disk I/O profile on that one should be a bit more consistent.

[William] - That is really cool to see customers already leveraging VSAN for their production usage and great to hear the experiences has been solid so far. You mention the use of vCOPs for monitoring the VMs, are you also using vCOPS to monitor the underlying Mac Mini’s and how do you handle hardware issues?

[Chris] - Honestly, right now, we're just relying on the vCenter CIM service to tell us if it finds a hardware problem. The first obvious problem I have with that is I'm not so sure it would notice a hard drive failure. Like VMware, we'd probably just bring it into a local Apple store and have any components still under warranty replaced. 'Normally' we have SNMP traps sent from vCenter to HP BSM. Being that this is such a small environment with lots of questions, we simply use vCOPs to alert us of any unusual behavior or problems. Many of of our vCenter alarms are 'self-correcting' alarms.

[William] - It sounds like your MacCloud is quite mature with so many different capabilities. Any plans in the near term to expand, I can already see more developers asking for similar setup? Will you be increasing your Mac Mini VSAN Cluster or potentially create a new one?

[Chris] - Most likely yes. My hope is that Apple and VMware will see the value in these community initiatives to hopefully make a 'support' Mac Mini with some native 10Gb capabilities. In the meantime, I can only see this growing to some really awesome potential.

[William] - Awesome to hear! Well, I do not want to take up any more of your time but before we conclude. Is there any tips or recommendations you would offer other fellow vSphere Administrators looking to run vSphere on Mac Mini? Any words of wisdom that you can offer?

[Chris] - Download William's ISO, upload and install the VIB from local system. Remote VIB install doesn't work because of static line (might be specific to windows) and last but not least. TRY IT!

[William] - haha. Thanks for the plug!

If you are interested in sharing your story with the community (can be completely anonymous) on how you use VMware and Mac OS X in Production, you can reach out to me here.