osx

Community stories of VMware & Apple OS X in Production: Part 9

10/09/2014 by William Lam Leave a Comment

Company: Connell Insurance
Software: VMware vSphere
Hardware: Apple Mac Mini

[William] - Hi Bryan, thanks for your time this afternoon. I know you have been pretty busy these last couple of weeks and I am glad we got some time with you to chat about your environment. Before we get started, can you please introduce yourself and what you are currently responsible for?

[Bryan] - Hi William. Thanks for the interest! My name is Bryan Linton. I'm currently the IT Director for Connell Insurance, an independent Insurance Agency in Southwest Missouri. We’ve been around for a long time but have been growing more rapidly in recent years. We currently have around 40 employees.

We have most of our systems in a secure off-site datacenter, but I still need some support systems onsite, and that's where your project bringing together inexpensive Mac Minis with ESXi caught my attention.

[William] - Can you tell us a bit more about your Mac Mini infrastructure? What is the hardware and software configuration and the type of workload you are currently running on them?

[Bryan] – First of all, I didn't buy the "server" versions of the Mac Mini - I just ordered the standard Mac Minis with stock RAM and storage. My only exception to going stock was on the CPU. For that, I went all-out and ordered the i7 quad-core processors since I knew I’d be using them as servers. But I bought a large SSD, a data doubler kit that allowed me to mount the second hdd, and 16GB of RAM, and installed all those myself, since it was cheaper than spec’ing it that way from Apple.

I also bought a 16GB low-profile USB flash drive to use as my install point and boot device. It's working fine booting and running ESXi 5.5. I’ve always used ESXi embedded on my servers going all the way back to version 3.5, so I was already comfortable and familiar with booting and running ESXi from a flash drive. So to summarize the hardware, that's a quad-core i7, 16GB of RAM, a 1TB spinning HDD and a 500GB SSD as datastores, booting ESXi 5.5 from a tiny flash drive that barely protrudes from its USB port.

On the software side, I downloaded your turnkey ISO for ESXi 5.5 for the Mac Mini. Your advice to enable support for the Thunderbolt Ethernet adapter was easy to follow, and that gave me two 1GigE NICs for whatever I need. The installation was about as simple as you can get.

My workload currently is very light. We just opened a branch office, so that’s where I’m using the Mac Mini. I installed a DC for the site, and have set up just one other support machine so far. It runs the management software for our video surveillance, alarm system remote administration, and the controller software for my WiFi access points. I’ll probably add (or move) other support roles to this machine before long. It's not working that hard and to be honest, is probably faster than the 6-year-old Xeon-based server currently filling the support-system role inside our main office. The two offices are very well-connected, so I can move around support systems almost without concern for which site uses them most. So I do foresee loading up the Mac Mini with more work.

Here is a picture of Bryan's setup:

[William] - What was the deciding factor on choosing a Mac Mini versus Mac Pro or any other platform? Given the Mac Mini is consumer grade hardware and is currently not a supported platform, were there any architecture decisions you had to make on the infrastructure or application side to accommodate for this fact?

[Bryan] – The Mac Pro is fantastic hardware, so if you need heavy-duty power behind your VMs it's certainly worth looking at. But if you really beef up a Mac Pro you're back in the cost realm of what server hardware typically costs. I honestly didn’t look at the compatibility or support of the Mac Pro with ESXi because we didn't need that kind of power for our new branch office.

As for other platforms considered, we’re largely a Dell shop. We use their small-form-factor desktops for our user workstations. I considered using one of our retired workstations as a “server”, but was afraid it would be too slow, even with an SSD, and it didn't have any more RAM capacity than the Mac Mini. Plus it's obviously bigger and less power-efficient, and it’s probably less hardware-compatible with ESXi. That’s why we ultimately chose the Mac Mini.

The main limitation of the consumer-grade hardware in the Mac Mini, for us, was RAM. The Xeon processors in my "proper" servers running in our CoLo aren't ever taxed nearly as much as the other compute resources, so to me the i7 quad-core CPU seemed more than adequate. Having the RAM maxed out at only 16GB, though, made me put extra thought into how I can make the best use of ESXi’s transparent page sharing between VMs. We’re mostly a Windows shop, so it made sense to me to standardize on a single Windows Server version for a given Mac Mini, and try to keep all those VMs at the same patch level. That way, the odds are better that many of the core OS pages in RAM will be identical, meaning the use of TPS will increase, and more RAM will be available to run applications (or even more VMs). We chose Server 2012 Standard as the go-to Windows server OS, and I look forward to loading up the Mac Mini to see just how far ESXi’s advanced memory management techniques will let me push it.

The other constraint was network connectivity. For example, I have just 2 NICs, and I'm currently in the process of working out a backup strategy. I think I'm going to use a backup appliance running on the Mac Mini with a relatively inexpensive NAS as the backup destination, and if needed I can dedicate a physical NIC to that, but with only two NICS, I have to think a lot about network traffic management. I have a couple of other ideas that involve using a thunderbolt dock and/or USB 3.0 NICs to increase the number of USB ports and NICs available to me. The big question there is driver support in ESXi, but I haven’t yet researched or tested those ideas at all.

[William] - You mentioned the current workload is pretty light and you plan to deploy additional Window Servers. What type of workloads will these Virtual Machines be running? Do you see a need to scale up your Mac Mini infrastructure from what you have today?

[Bryan] - I'm not yet ready to put critical production data on them. Not until I have a better feel for what backup looks like, and until I really push the limits of the hardware resources. But support systems like I mentioned above - Wi-Fi controller, NVR for surveillance (with the data stored off-box), things like AV management, Spiceworks, syslog servers, additional DCs to provide a global catalog server, a DFS namespace server, and maybe a replica file server synced with DFS replication are all good possible candidates for running on a Mac Mini.

We have an email archiving system that has to run somewhere. Its data is stored on a remote share and DB, but the processing can happen anywhere, and Mac Mini will handle it fine. That lets me keep resources free for the mission-critical apps that run on my "real" servers that DO contain our critical production data. If I can improve user experience by offloading non-critical support systems to the Mac Mini, there's less resource contention on my vCenter-managed hosts, and user responsiveness should benefit from that.

I also have software firewalls that I use to create a "double-router, Dual-NAT" environment so I can run or build machines in a test, isolated environment, with internet connectivity, with the IP addresses they use (or will use) in production without conflict. They run in perfect isolation. The Mac Mini can host those software firewalls along with any machines that are either being built, or perhaps being restored from backup for exploratory purposes, or even for testing of upgrades or new software in a mirror environment before it goes into production.

As for scaling up, so far I haven't considered adding a Mac Mini ESXi host to my vCenter environment. But I might consider that if two things happened:

Apple starts supporting more RAM in the Mac Mini.
VMware decides to support the Mac Mini hardware officially.

Actually, as I get more time and experience using this setup, number 2 may diminish in importance. Time will tell.

[William] - Your last reply was quite interesting. You mentioned you have not considered adding your Mac Mini ESXi hosts to vCenter Server? I’m curious to hear why the support of Mac Mini would dictate the ease of centralize management? Is it from a support standpoint that you did not want to do this or additional licenses?

[Bryan] - My Dell ESXi servers are managed by vCenter, but I have three of them, which is my license limit currently. If I had a free slot for the Mac Mini I'd certainly use it. But I'd have a hard time justifying the purchase of *additional* licenses for vCenter to bring machines under management that aren't even officially supported. But yeah - if I had the licenses I'd have no hesitation in managing them via vCenter. I get around it currently thru the use of shared data stores.

[William] - Bryan, it was really great chatting with you and thanks again for sharing your experiences on how you have leveraged VMware and Apple Mac Mini’s in your production environment. Here is the last question that I have asked all my past interviewees, do you have any final advice or words of wisdom for someone looking to embark on a similar journey? Any particular resources you would recommend for someone to start with?

[Bryan] - Well, virtuallyGhetto is THE place to go for those resources. Without your pages I would not have made the leap. My advice would be: Don't expect it to do what a $5,000 investment will do. If you plan to run mission-critical apps or host production data, KNOW what your backup and recovery process looks like and TEST it.

If you understand you're striking out somewhat on your own, and you don't mind being a pioneer for the fun of it, do your due diligence, and if it seems like a fit, enjoy it! It's honestly fun to show people my rack. "That's our server. It's running ESXi 5.5." "...Really?!?"

If you are interested in sharing your story with the community (can be completely anonymous) on how you use VMware and Mac OS X in Production, you can reach out to me here.

Community stories of VMware & Apple OS X in Production: Part 8

09/25/2014 by William Lam 3 Comments

Company: Mid-Pacific Institute (Private School in Hawaii)
Software: VMware vSphere and Fusion
Hardware: Apple Mac Pro

[William] - Hi Derick, I appreciate you taking some time out of your busy schedule to talk with us all the way from Hawaii 🙂 Before we get started, can you quickly introduce yourself?

[Derick] - Sure William. My name is Derick Okihara, and I work for Mid-Pacific Institute. We are a private K-12 institution with about 1600 students. My role here is general IT and server administration. I've been working with computers since I was in high school. I have been a long time Apple user (since //gs), but really started working with them professionally about 10 years ago. We currently have a 1:1 iPad program for the students, and 2:1 iPad+laptops for faculty, so there's a lot of technology to support.

[William] - That’s awesome Derick. Look forward to hearing more about your environment. Speaking of which, I hear you are currently managing some Apple hardware running on VMware? Could you tell us a little bit about the hardware configuration and the VMware software you are currently using?

[Derick] - We are currently using vCenter with ESXi 5.5, we have 2 Mac Pro (5,1s) in a cluster with a Synology 1813+ shared storage. The network storage is connected via the iSCSI software initiator using round-robin. We also use VMware Fusion for the Mac. The Mac Pros have 24GB of RAM and 4 port Intel Gigabit Ethernet cards for a total of 6 ports each.

[William] - What made you decide on using a Synology for shared storage and what configuration/capacity did you go with? Were there any other options you were looking at?

[Derick] - A lot of the decisions for this setup was made on price. How this all started, was that I was asked to create an ESXi server to host a VM Appliance to run our campus wide Informacast speaker system. I had already been planning an ESXI deployment on the Mac, testing on Mac Mini. Instead of building a PC server just for this appliance, I was given the OK to build on an existing Mac Pro, so it could serve multiple purposes.

Being forward thinking, I knew we needed redundancy, so I opted to go for network storage. With a tight budget, and being able to use CPUs we already had, I decided on the Synology 1813+ for it's 4 gigabit ports. This allowed me to later add the 2nd host in our vcenter cluster when we expanded.

[William] - Can you talk a little bit about the type of workloads and applications you are currently virtualizing on the Mac Pro’s and are these all OSX VMs?

[Derick] - Since this is still version 1.0, we aren't heavily taxing our cluster. Right now it hosts 5 VMs (2 Virtual Appliance, 2 OS X Server, 1 Windows Server). I'd want to add more RAM as well, OS X VMs are very RAM hungry.

The OS X servers are a student file server (AFP/SMB) and an Apple Caching server / Munki repository. The Windows server is mostly a test bed, the Virtual Appliances are the aforementioned Informacast manager and VCenter Appliance.

Here is a picture Derick's two Apple Mac Pros:

[William] - The Mac Pro’s have a maximum amount of memory that it supports, do you plan on expanding the infrastructure to accommodate additional workloads or would you be looking at upgrading to the latest generation of Mac Pro (black)?

[Derick] - Honestly, with our current needs and budget, I think I would be looking at the next generation Mac Mini combined with some sort of PCI-E enclosure. Like the Sonnet XMac server. I know the Mini will likely not be fully supported, but I like what i've seen on virtuallyGhetto with the current generation 🙂 That is, if the next gen Mac Mini supports 32GB of RAM!

[William] - Very cool! So from your point of view, you would rather have more Mac Mini’s than a couple of Mac Pro’s? It sounds like cost plays a huge factor, but what other constraints or requirements that is making you lean more towards Mac Mini’s instead of going to a new Mac Pro which can get up to 64GB of memory and 6-Core CPU?

[Derick] - Footprint - the Mac Pros we currently have take up a large amount of rack space. Even the new mac pros would not be rack mountable without an additional enclosure. For us, having 3 x Mac Mini with 32GB of RAM would be ideal price/performance ratio. (We have a 3 CPU license). Eventually our Mac Pro 5,1's will die, so I'm already thinking about what's next. Having 3 x Mac Mini servers in a cluster, that takes up only 3U would be pretty sweet!

[William] - Speaking of support, did you purchase any type of extended contracts with Apple on the hardware or you going to deal with them on a case by case basis? Have you had any issues with failing hardware on the Mac Pro’s?

[Derick] - We only had the initial Apple Care (now since expired). We have 1 spare Mac Pro currently running other loads but that could be migrated in the event that we have a hardware failure. We have not had many issues on the Mac Pro 5,1s other than internal hard drive failure. They've been rock solid.

[William] - Has there been any interesting issues or challenges you had faced while setting up this infrastructure? Either the hardware, software or managing the VMs?

[Derick] - This whole process was a learning experience for me. At a previous job, I had inherited an ESXI server running multiple CentOS and Ubuntu VMs, but I had never set it up myself, let alone on a Mac. Thanks to the multiple resources on the web (Rich Trouton's blog, VirtuallyGhetto, and a P2V script from Alan Gordon at MacSysAdmin) the process and gotchas were mostly done before me

The biggest challenge for me was configuring the Synology for iSCSI-round robin. In my research I found that one could utilize multiple gigabit connections with Multi-path IO for higher bandwidth. After lots of configuring and back and forth with Synology / VMware support, I finally found the proper settings that allowed me to utilize more than 1 gigabit link.
However, after I updated to ESXi 5.5, it broke.

I was stuck, because I needed to upgrade to 5.5 in order to run an OS X caching server (12-character serial number fix in 5.5). But Synology said the 1813+ was not compatible with 5.5 and would not help me. Long story short, one of my hosts is running 5.5 (with OS X Caching server) and the other host is running 5.1 (file services) because it needs the greater throughput.

[William] - Derick, I want thank you for taking the time and sharing with us your experiences with managing VMware and Apple hardware. Before I let you go, do you have any tips for our readers that may be in a similar environment (academic) and needs to build out an infrastructure to support their end users? Any gotchas or things you would recommend if you had to do this all over again?

[Derick] - Anyone looking to reduce their machine footprint should definitely look into virtualization. VMware has very attractive pricing for the EDU market if you're looking to build a cluster with high availability, or you can run a single host for free. The best piece of advice I can give is just to test thoroughly. Virtualization is very complicated, and combines a multitude of areas of expertise (Storage, Networks, Workloads, and ESXI platform itself). It can be daunting but it's very rewarding. If you get stuck, just ask William on Twitter @lamw, jk

If you are interested in sharing your story with the community (can be completely anonymous) on how you use VMware and Mac OS X in Production, you can reach out to me here.

Community stories of VMware & Apple OS X in Production: Part 7

09/18/2014 by William Lam Leave a Comment

Company: Fortune 150 (Retail)
Software: vSphere + vSphere Replication
Hardware: Apple Mac Pro

[William] - Hi Vitaliy, thank you for reaching out and wanting to share your experiences with the community on managing a VMware and Apple OS X infrastructure. Can you tell us a little bit about yourself and what you currently do?

[Vitaliy] - I am a Senior Systems Analyst for a Fortune 150 company that wishes to remain anonymous (aka I do not have legal clearance to use the company name). I am part of a team that is responsible for providing IT infrastructure for many creative and marketing applications -- think pre-press and advertising.

[William] - Can you provide us some details about the VMware and OS X infrastructure that you’re supporting? Software/Hardware specs that you decided to go with and the workload characteristics?

[Vitaliy] - Prior to virtualization we were running two dozen Xserves with OS X 10.6 running a wide range of applications from Open Directory to custom in-house scripts. We have virtualized the whole environment with just 4 Mac Pro machines, each machine has 12 cores and 64GB of memory giving us a total of about 128GHz and 256GB of memory.

We have exhausted all the PCI-X slots on the Mac Pro's by adding two dual port network cards and a dual port HBA. As a result we have two redundant management, data, and vMotion ports on each machine. Oh, one thing worth mentioning is that VMware officially only supports 32GB of memory per Mac Pro but we have been running 64GB with no issues. For the past year we have been running vSphere 5.1 and just upgraded to 5.5 last week.

We have been using HP 3PAR SAN for our storage back-end and over the last couple of weeks we have migrated to an Oracle SAN. The whole process was completely seamless and transparent to the users thanks to VMware.

Here is a picture of the Mac Pro setup courtesy of Vitaliy:

[William] - Wow, that’s great to hear you’ve been able to really push the Mac Pro’s. You must have been happy to be able to consolidate all those Xserves! What was your approach for virtualizing OS X from the physical Xserve to Mac Pro? Did you rebuild or leverage some type of V2V?

[Vitaliy] - We decided to rebuild from scratch. We were running an outdated version of OS X 10.6 and all the applications running on top of that were just as old.

[William] - Can you talk to how you provision your OS X Virtual Machines and Applications and how it gets to the end users? Do users get their own systems or is this a shared infrastructure?

[Vitaliy] - It's a shared infrastructure, generally a VM is dedicated to a particular application. We created a "base VM" that has basic settings like power/energy saver settings, local accounts, monitoring software, etc. preconfigured and whenever we need a new virtual machine we simply clone it and change the hostname and IP address on the new VM. Perhaps a template would've been a cleaner solution but this is what we do. We are currently looking into automating configuration with either Puppet or Casper.

When we initially rolled out a couple of OS X virtual machines we noticed that CPU usage on the VMware cluster spiked up to almost a 100% while the virtual machines were idle. It turned out that the default OS X screensaver uses GPU power to generate that flare effect and because not enough GPU memory was available it resorted to using up all the CPU. Disabling the screensaver or switching to a text based one quickly fixed that issue ...

[William] - Thanks for the excellent tip on OS X screensaver, this is a handy one to know about! How do you go about monitoring the Mac Pro infrastructure? What’s the process for replacing failed hardware components and have you had any challenges with this?

[Vitaliy] - We treat it the same way as the rest of our environment -- each vSphere node and virtual machine is monitored via Nagios. We have this cluster running for little over a year now and luckily we have not had to deal with any hardware failure.

[William] - For your OS X Virtual Machines, do you have a need for backups or a DR strategy? If so, could you share some details on what you are currently using?

[Vitaliy] - We have a replica of our production environment at a remote disaster recovery site and we use vSphere Replication to copy all the VMs nightly. We also heavily rely on the snapshot feature prior to making any operating system or application changes, it has been a lifesaver so far.

[William] - Vitaliy, I want to say thank you very much for taking some time out of your super busy schedule to have a chat. Before I let you go, do you have any words of wisdom for others looking to manage a similar infrastructure? Anything you would do differently and any resources you have found useful in aiding you to support a VMware / OS X infrastructure?

[Vitaliy] - Speak to your manager, legal department, or whoever is in charge about interpreting Apple EULA. I have heard of at least three different interpretations and all have legal implications. I am very happy with our environment and would not change a thing if I had to build it again. Your blog, virtuallyGhetto, has been a great resource as you are the only one talking about VMware products running on Apple hardware.

If you are interested in sharing your story with the community (can be completely anonymous) on how you use VMware and Mac OS X in Production, you can reach out to me here.

Community stories of VMware & Apple OS X in Production: Part 6

09/10/2014 by William Lam Leave a Comment

Company: Public Education K-12
Software: VMware vSphere
Hardware: Xserve

[William] - Hi Pete, thanks for reaching out on Twitter and offering to share your experiences in managing VMware and Apple OS X in an academic environment. Can you start off by quickly introducing yourself and what your role is currently?

[Pete] - My name is Pete Wann, I've been a sysadmin for over 15 years, mostly in education. I switched to Mac at the OSX transition because I was really interested in the Unix (BSD) foundation. My interest in Unix was piqued by my exposure to Solaris in the military, and since then I've tried to focus my career around all the various flavors out there, it just so happens that I like Macs, and it's been a good niche to be in. The community is awesome and ridiculously supportive.

My current role is as a Principal Systems Technologist at Oracle. I work for our Global IT group, but I primarily support a subset of our Marketing department. I'm responsible for the infrastructure around our video, print, and web production efforts. Although, the specific implementation we're going to discuss was done at my last position, with a large school district in Alaska.

[William] - Thanks for the background Pete. So I hear you were involved in an implementation that involved VMware and Apple OS X Technologies, can you share with us some more details about the environment?

[Pete] - Well, as you know, Apple discontinued the Xserve in 2010. (boo! hiss!) This was disastrous for that environment since the schools were very far apart, and our WAN links were slow and sometimes tenuous, in addition to some decisions made before I arrived about how home directories were handled, we needed to have some kind of server presence in every school. Since we couldn't count on having someone in each school who was comfortable going into a server closet to reset a system, we really needed Lights-Out-Management on whatever hardware we put out there.

Additionally, this was by far the largest Open Directory deployment that we (or Apple) had ever heard of. We had both computers and users in OD, and with our sometimes rickety WAN, we needed to have OD replicas as close to the clients as we could get, so again, a server presence in every school.

Eventually we migrated all of our user authentication over to AD, but still used OD for some computer management functions (mostly we used JAMF Casper for imaging and package deployment), so we still needed separate OD replicas for each school. (Each school was its own OU within OD so that we could distribute computer management tasks.)

[William] - I too remember the EOL announcement of the Xserve, it definitely had an impact to everyone who relied on that hardware. It sounds like you had a decent Apple Infrastructure, where was all this running? Physical or Virtual?

[Pete] - At the time, ESX did not support the Apple RAID card, so I could not use the internal storage with any of the systems I had available, which was fine with me, since I didn't want any moving parts on the hosts if I could avoid it, to hopefully increase longevity.

So, after much bugging of the powers-that-be, I got three licenses for vSphere for the three Xserves I scrounged from our secondary schools, removed all internal storage, then installed ESXi on a small USB drive on each host. I used the built-in iSCSI support in ESXi to connect to our NetApp storage, and integrated the Xserves with the rest of our vSphere environment, with full support for vMotion and everything. It was really easy, and worked insanely well.

We wound up virtualizing about 20 hosts across the three Xserves, mostly OS X, but also a couple of Linux hosts to act as web front-ends for our Casper environment. I fought hard to make the Xserves full-fledged members of the vSphere deployment, but my counterparts on the Windows side resisted harder. I still think that was a waste of available CPU power, but such is life.

[William] - Wow, this is pretty cool! I think this is the first implementation that I have heard of that leverages external storage w/Apple hardware. Could you share some details about the hardware specs for the Xserve and how you came to this particular configuration?

[Pete] - Well, in the case of the Xserves, we lucked out by having already ordered 77 of the last generation before Apple announced the end-of-production. We were in the process of transitioning from Xserve G5s to Intel in all the schools.

I was at the MacTech conference in LA when word came out that the Xserve was killed (Can you imagine the mood in that room?) and immediately got in touch with my boss to ask for as many more of the last generation we could afford to buy. Initially my intention was to go with Parallels Server, and we did buy it and deploy it at a couple of sites, but let's just say that didn't go well, and I jumped off that path as soon as ESXi 5 was released.

Initially I wanted dual-processor systems with the internal SSD and maxed RAM (I believe 48GB on that model), and since I was still thinking in terms of what Parallels Server supported, I got 3 internal 1Tb drives to use for local storage. Unfortunately, the option of adding the internal SSD as a fourth drive disappeared almost as quickly as it appeared, and we missed the window. I got the rest of what i asked for, though.

Once I discovered that ESXi 5 didn't support the Apple internal RAID controller, I had to find another solution for storage, since I didn't want to run everything, Hypervisor and VM Storage on USB drives. Fortunately for me, our vSphere environment was already configured to connect to our NetApp NAS, so it was trivial to add that storage for the VMs once the Xserves were added as hosts to the vSphere DC.

I also managed to scrounge additional NICs for the Xserves to give the nodes more network capacity for the guest VMs. So I think ultimately we wound up with 6 total 1Gb connections — 1 management, 1 vMotion etc., and 4 on a vSwitch for guest VMs. The three Xserves were segregated into their own vDC to avoid confusion for our management and SysAdmins.

[William] - How did you go about monitoring this infrastructure? Any challenges or gotchas you found while building and managing this environment?

[Pete] - Honestly, no. We used all of the same management tools that we used for our wider vSphere environment, and it all just worked.

At the time, I believe they were implementing some monitoring tools from Symantec, but I left while that was still being implemented. Before that was in place, it was largely a manual process. I stayed as hands-off as possible once I had my environment up and running because I take a "less is more" approach to being a SysAdmin. 🙂

The ONLY gotcha, and it was very easily overcome, was the lack of support in ESXi 5.0 for the Apple internal RAID controller. That turned out to be good for us, as it forced us to use the existing vSphere infrastructure.

As for management, we just had to embrace a new way of deploying VMs, but there again, once I built a template for vSphere, it was trivial to deploy new Mac VMs, which I then configured as needed. If we'd had a larger environment, I would have leveraged tools like Puppet or Casper to auto-configure hosts to our needs.

[William] - In building out this environment, it sounds like you learned quite a bit. Was this something you already had some experienced with or were you learning on the job? If the latter, were there any key resources you leveraged that helped you build and manage such an infrastructure?

[Pete] - I had experience with VMware from my previous job, where I got involved in deploying new VMware nodes to help transition to a virtual datacenter. In truth, it worked so well and was so easy to set up, I didn't really need support except for gathering the specifics of our environment.

There was literally no difference between the setup for generic x86 hardware and Xserve as far as I could see. The only difference was that in addition to all the other guest OSes, we could also run OS X on these hosts.

[William] - Pete, I would like to thank you very much for your time this afternoon and sharing with us your experiences. I think this has been very informative/educational and should help others thinking about building or managing a similar type of environment. Before we finish up, do you have any words of wisdom or advice to others looking to start a similar project and perhaps also working in the academic/education field?

[Pete] - I would say that if you're thinking about it and if you think that virtualizing OS X will help, then go for it. It's actually easier than you probably think. Also, I'd say to remember that as a SysAdmin, managing up is just as important as managing your systems. Keep your eyes open to what's happening in your industry, and try to be prepared for new things and opportunities to save money and improve efficiency. Especially in public K12, budgets are shrinking, but demands (particularly on IT) are increasing. Don't be afraid to speak up if you think you can find a way to save money and provide the same or a better level of service for your students.

If you are interested in sharing your story with the community (can be completely anonymous) on how you use VMware and Mac OS X in Production, you can reach out to me here.

Community stories of VMware & Apple OS X in Production: Part 5

08/19/2014 by William Lam 2 Comments

Company: Artwork Systems Nordic A/S (AWSN)
Software: VMware vSphere
Hardware: Apple Mac Pro

[William] - Hi Mads, thank you for taking some time this morning to share with the community your past experiences managing a VMware and Apple OS X environment. Before we get started, can you introduce yourself and what you currently do?

[Mads] - My name is Mads Fog Albrechtslund, and I currently work as a vSphere Consultant for Businessman A/S Denmark. The reason for my current employment, is primarily a Mac based vSphere project I did at my former employer, Artwork Systems Nordic A/S also in Denmark. Before I became a vSphere Consultant, my primary job function was as a Mac Consultant, in which I have several Apple related certifications.

[William] - Could you describe what your vSphere project was about?

[Mads] - The vSphere Project, was that of virtualizing and consolidating the infrastructure of Artwork Systems Nordic A/S (AWSN). AWSN is a reseller of hardware and software to the graphical industry, thereby running a lot of Apple systems and software that require Mac OS X underneath.

When I started at the company in early 2009, there were around 8-10 servers, and only 9 employees. Every server was just a desktop Mac or PC, running multiple services at once, trying to use the hardware at best. I started by consolidating and somewhat standardizing all these machines, into a Rack cabinet.

But I still wanted to make it better, more flexible and faster to deploy new OS'es when they are needed. I also wanted to move away from running multiple services on a single OS. I started looking into virtualization around late 2010, before VMware even made vSphere compatible with the Mac's. And we started working with a competitor of VMware, which at the time was about to release a bare-metal hypervisor that was compatible with Mac hardware.

We invested time, money and hardware in that initial project, only to around 6 month later to find out that the vendor would drop that bare-metal software again.

[William] - Ouch! I guess that is one of the risks when working with a new company/startup. So what did you end up doing after the company dropped support for bare-metal support?

[Mads] - So when VMware release vSphere 5.0 which was compatible with Apple hardware, I asked my boss to try again. He said "Sure, go ahead…. but we don't have a lot of money to do this with". So I needed to make this project as cheap as possible.

What I ended up with was 3 Mac Pro's (2x 2008 and 1x 2009), which I got almost free from a customer, extra RAM (32GB in each Mac Pro), extra NIC's (4 NIC's in each Mac Pro), a Synology RS812+ NAS and VMware vSphere Essentials bundle.

Here is a picture of the 3 Mac Pros:

[William] - I too remember when VMware announced support for Apple Hardware with vSphere 5.0, that was a huge deal for many customers. Were there any performance or availability requirements that you had to take into considerations while designing this solution? Did all Virtual Machines run off of the NAS system or was it a mix between local and remote storage?

[Mads] - All VM's ran off the NAS over iSCSI. I did consider the availability of that design, but given the constraints of the money of the project, there was not much of a choice. I did not want to run the VM's on the local disks inside each Mac Pro, considering that if one Mac Pro died, I would not easily have the possibility to power-on that VM on another Mac Pro.

The performance of the NAS was not great, but good enough. After I left, the NAS was upgraded to a Synology DS1813+, and then using the old Synology RS812+ as a backup destination. The load on the VM's was light, as there only was 10 employees in the company, and most of the VM's was only for testing or designing solutions for the customers.

[William] - What type of Virtual Machines and applications were you running on the Mac Pros?

[Mads] - The 3 Mac Pro's are running around 20 VM's, where most of them are either OS X based or Linux Virtual Appliance's. My plan was to do one service per OS, to keep it as simple as possible. Almost all the OS X based VMs are running OS X 10.8 Mountain Lion. Some of them are just plain Client installations, but most of them have the Server app installed, to run Open Directory, DNS or File Server.

The Client installations are running specific software that the company sells, like graphical processing software from Enfocus or FTP software from Rumpus. There is also an older OS X based VM, running Mac OS X Server 10.6, which runs a special graphical procession software called Odystar from a company called Esko. This software only exist on Mac OS X, and it also requires a HASP USB dongle for its license. Most of the VM's are configured as low as possible, which for most is 1 vCPU and 2GB ram.

The Mail server for the company, is based on Kerio Connect software, which is also something that the company is a reseller of for its smaller graphical customers. That software exist either as a virtual appliance, a Windows install or a Mac based install. We ended up with choosing a Mac based installation, because we knew it better.

[William] - How did you go about monitoring the Virtual Machines as well as the underlying hardware? Any particular tools that you found worked well for your organization?

[Mads ] - We did not do much of monitoring, of neither the VMs or the hardware. I was onsite, and sitting almost beside the rack most of the time, so if there was any trouble either physically or virtual, I could fix it fast. I had configured email reporting in all the solutions that gave the option (vCenter Server, Synology NAS and some of the applications).

[William] - I know you had started this project back in 2010 and there was definitely a limited amount of hardware options to run Apple OS X VMs. Today, there are a few more options and if you were to do it again, would you have done anything differently? Would you still consider the Mac Pro (Tower) or look at potentially the newer Mac Pro (Black) or even the Mac Mini’s?

[Mads] - We did start out by looking at the Mac Mini's, but considering that we could only run 3 hosts because of the vSphere Essential license, we needed to get more RAM in each host, than the Mac Mini's could provide. The Tower based Mac Pro is still the best option for this installation, given that it is available for a reasonable price, runs more than 16GB ram and you can get 2x CPU sockets in each host.

The new black version of the Mac Pro, is especially not a good fit, primarily because of the price and because of the dual GPU's and only 1 CPU. I would love a Mac Mini with 32GB ram, that would properly fit perfectly, considering the advances in CPU technology over the 2008/2009 CPU's in the Mac Pro's currently running the environment.

[William] - Mads, thank you very much for spending your morning and sharing with us your experiences with running vSphere on the Mac Pros. You have provided a lot of good information that I know will surely help the VMware and Apple community. One final question before I let you go. Is there any tips/tricks you would recommend for someone looking to start a similar project? Any particular resources you would recommend people check out?

[Mads] - First of a big thanks to yourself, for provide great content on http://www.virtuallyghetto.com. I have also provided my own experiences both on my personal blog www.hazenet.dk and on businessman's company blog bmspeak.businessmann.dk

On my own blog, I have written about issues with screensavers in Mac OS X VM's and I have also written a long blog post about how make a never booted Mac OS X template VM, which don't have any UUID's set.

If you are interested in sharing your story with the community (can be completely anonymous) on how you use VMware and Mac OS X in Production, you can reach out to me here.

How to run Nested Mac OS X guest on ESXi VM on top VMware Fusion?

08/08/2014 by William Lam Leave a Comment

You might be asking, why would anyone want to do this? Well, luckily this is not a "because you can" type of answer but was it was an interesting solution that one of our VMware Engineers (Darius) had shared with me after helping out on this VMTN Community forum thread.

The user was running VMware Fusion on his physical Mac OS X system and wanted to be able to test OS X Mavericks under ESXi. Not having a physical ESXi host to test with, the next best thing was to run a ESXi VM under VMware Fusion and then run the Mavericks guest on top of that.

Here is a quick diagram of the user setup:

The issue with just simply doing this is that for a Mac OS X guest to properly run on ESXi, the underlying hardware must be Apple Hardware. The reason for this is not a technical challenges, but rather a legal one per Apple's EULA. The way in which ESXi detects that the underlying hardware is Apple is by checking whether Apple's SMC (System Management Controller) is available.

In the scenario above, the Nested ESXi VM is not automatically passing through the SMC from the physical Mac OS X system and hence the Mac OS X VM at the very top of the stack will not properly function. The solution that Darius found was to add the following two Advanced VM Settings (VMX) entries to the ESXi VM:

smc.present = "TRUE"
smbios.reflectHost = "TRUE"

This will allow the passing of the underlying SMC up into the Nested ESXi VM which will then allow Mac OS X guest VMs to properly function. We can also confirm this by check the Nested ESXi MOB by pointing a browser to the following URL: https://[ESXI-IP]/mob/?moid=ha-host&doPath=hardware

If you did not add the two entries above, then the smcPresent property would show up as false. In our case, we did add the following two entries and we now run our Mac OS X Guest. Here are a couple of screenshots of performing this on my iMac at home running the same exact configuration:

Thanks Darius for sharing this with me and the community! I am sure this will come in handy for anyone wanting to test Mac OS X guests under ESXi but do not have a physical ESXi host and can easily substitute using VMware Fusion.