apple

Apple Mac Pro 6,1 PCIe SSD issue resolved w/ESXi 6.0 Update 2

03/15/2016 by William Lam 6 Comments

Early last year, the new Apple Mac Pro 6,1 (aka black can design) was certified and fully supported on vSphere 6.0 which I had blogged about here. Several months later, customers discovered that some of the newer Mac Pro 6,1 units were shipping with different model of their PCIe SSD device than what was originally released at GA. This was problematic because ESXi was not aware of this newer device and could not detect during or after installation. Although a work around was identified for customers looking to install either ESXi 5.x or 6.x on the newer Apple Mac Pros, it definitely was not ideal.

It has taken a bit longer than expected, but the issue has now been resolved with the latest release of ESXi 6.0 Update 2. A similar fix will be available for customers running ESXi 5.5 in a future update. You can find the direct download for ESXi 6.0 Update 2 in link below which includes a pointer to the release notes in case you are interested in other fixes included in this release.

vSphere ESXi 6.0u2 - https://my.vmware.com/web/vmware/details?downloadGroup=ESXI60U2&productId=491&rPId=10348

ESXi 6.0 on Apple Xserve 3,1

11/17/2015 by William Lam 64 Comments

A couple of months ago, I shared a guest blog post from one of my readers John Clendenen who was able to get ESXi 6.0 running on an Apple Xserve 2,1. At the end of that article, it was hinted that John was also looking into getting ESXi 6.0 running on an Apple XServe 3,1 and you can the details below after several months of investigation.

Disclaimer: This is not officially supported by VMware, please use at your own risk.

*** This is a guest blog post from John Clendenen ***

First an update on my Xserve 2,1’s. I had them running for over 100 days without any issue! However, now that I have the 3,1 working reliably, it is time that I part ways with my Xserve 2,1’s. I currently have them up on eBay. Here is the link: http://www.ebay.com/itm/231752771080?ssPageName=STRK:MESELX:IT&_trksid=p3984.m1555.l2649

Anyway, onto the Xserve 3,1.

I came across an Xserve 3,1 on eBay about a year ago. It was badly photographed, and the seller didn’t really know what he/she had. It wasn’t getting much attention, so I thought I might get it cheap. I ended up paying $500 for it which I felt ok about, but not great.

When it arrived, it had no processors, heatsinks or airflow duct. I immediately messaged the seller, and was able to get $350 refunded to me. I found the missing parts for under $100 over the next few weeks, and developed an intimate understanding of the Xserve 3,1 hardware.

At this point, I had no familiarity with vSphere at all. I was running OS X server and virtualizing a few services in Fusion. It was only through researching the Xserve 3,1 to find the missing hardware that I discovered that VMware had supported once as an ESXi 5 host. This made me wonder if it might still be possible to run ESXi on it, despite it no longer being supported.

I have found, after a considerable time investment, that the Xserve 3,1 can run ESXi 6, just as I found the Xserve 2,1 can run ESXi 6. However, unlike the Xserve 2,1, the Xserve 3,1 took months of troubleshooting before I had it running as a reliable ESXi host.

As it turns out, despite how much time it took me to get it working, there are only 2 serious issues with the Xserve 3,1 running ESXi 6. The first is somewhat specific to my configuration, but the second will be relevant to all configurations.

The first issue concerns booting into ESXi on a headless Xserve 3,1. The issue is limited to configurations where ESXi is booting from a drive installed in the optical bay (my original configuration). I have since changed my configuration and swapped the ESXi boot drive from the optical bay to the first hard drive bay. I have had no issue since I made this change.

For my configuration, I used an OWC bracket to replace the optical drive with an SSD. I installed ESXi onto it without issue. During installation, it was connected to monitor, keyboard, etc. I ran some VM’s on it to make sure it worked, and there were zero issues. I was relieved! So, I put it in the rack, wired it up and turned it on. Nothing. The Xserve lit up, and it was clear that it got through POST, but ESXi was clearly not booting.

Long story short, when no monitor is plugged into the Xserve 3,1, it will not automatically boot into ESXi if the boot drive is installed in the optical bay. The Xserve boot options can even be programmed through the front panel, but no configuration will make it reliably boot from the optical bay when a hard drive is installed. It is truly baffling, and if anyone has some insight here, or if it is a problem specific to my particular Xserve, I would love to know.

The solution, in my case, was to plug a keyboard into the Xserve, and hold down option for a few minutes while it boots (bringing up the boot options). Once all LED activity has normalized and the fan has settled down, I released the option key and pushed the arrow buttons. I think you only need to push the up button, but I always just pressed all of them to be sure. Then I pressed enter, and ESXi will boot. I have since simply swapped the boot drive to the first drive bay. Ideally, I’d have the other drives in the hot-swap bays, but I felt it was too much trouble to keep it in the optical bay.

The second issue concerns the onboard NIC. Once I had ESXi up and running, everything worked fine for anywhere between a few hours and 2 days, after which the Xserve 3,1 host would disappear from the VCSA and become completely unresponsive (no ping/ssh/etc). The length of time before failure made this issue especially difficult and time consuming to diagnose.

After nearly a month of frustration and disappointment, I determined that ESXi actually continued to run, but all network connectivity was ceasing. The only solution I have found is to install a 3rd party NIC and completely avoid using the onboard NIC. Even in standby, the onboard NIC can cause problems, but when it is completely unused, both for management and VM traffic, it no longer causes any problems.

This has been superficially improved with the last update, but use of the onboard NIC should still be completely avoided. The ESXi host will remain accessible via the VCSA, but the network management will become grayed out after a day or so. I suspect this is a driver issue in ESXi, but I really do not know.

Beyond these 2 issues, I have had no problems. Since the last update, even the performance and hardware status tabs are functional. RDM is not available, but not recommended in the first place. The Apple RAID backplane will not be recognized, but this was even the case in ESXI 5 when it was officially supported by VMware.

I hope that my efforts here will save others a lot of time and frustration. I think that for a lot of IT infrastructures, ESXi on an Xserve might make sense. It can run non-critical OS X services (which are hopefully the only kind of services you’re trying to run in OS X).

Summary

Completely avoid using the on-board NIC. Silicom NIC’s are recommended.
Find a standard backplane. The RAID backplane is useless in ESXi.
A 2.5” drive can be installed in the optical bay, but booting from it is problematic

The Xserve 3,1 with the Silicom NIC installed

The 6 ports are a tight squeeze, but they just fit. My other 2 EXSi hosts are Supermicro Nodes, also with Silicom NIC’s and I had to use a Dremel to grind off part of the chassis to make all the ports accessible. But the Xserve works out of the box.

The OWC SSD “Data Doubler” bracket in the optical bay. Booting from here is a pain, but putting an additional SSD here works great for host caching.

The standard backplane is difficult to find, but is a great asset for vSphere. It is easy to distinguish it from the RAID backplane which would have a heat sink here.

There are no complications during installation/initial configuration.

Apologies for not having a longer uptime. I updated to ESXi6.0U1a 12 days ago, but I’ve had the Xserve 3,1 up for months. If something changes, I will post an update here, but I am confident that the system is stable.

This is the final stage of my home lab. The Xserve 3,1 is 1 of 3 ESXi hosts. These are accompanied by a primary domain controller (Samba4), a media server (Emby) and a home-grown NAS (Centos7). Networking in the back is Ubiquiti. I use this lab to prototype production environments for clients, and of course to run my home media services 🙂

Heads Up – Workaround for changing Mac OS X VM display resolution in vSphere & Fusion

10/22/2015 by William Lam 47 Comments

For customers who are running Mac OS X 10.9 (Mavericks) or newer in a Virtual Machine, you may have noticed that you can no longer set a custom display resolution beyond the default 1024x768 in either VMware Fusion and vSphere, regardless of the amount of video memory that has been allocated. The reason for this behavior is that Apple has changed the way in which it remembers previously used modes and would automatically fall back to this versus retaining the custom mode using the Display Preferences. Given this is a non-trivial fix, VMware Engineering has been working hard on a providing a workaround which would still allow users to set a custom resolution from within the GuestOS.

The workaround that has been developed is a tiny standalone command-line utility called vmware-resolutionSet which runs within the Mac OS X Guest and allows you to configure a custom display resolution. You will need to ensure you have VMware Tools installed and running before you can use this utility. As of right now, customers can get a hold of this utility by filing an SR with VMware Support and referencing PR 1385761. Although this tool has not been officially released and must go through the standard release process, the plan is to include it in a future update of VMware Tools and will available for use with both VMware Fusion and vSphere.

UPDATE (12/11/15) - Thanks to reader @elvisizer, it looks like the latest VMware Fusion 8.1 release now includes an updated version of VMware Tools (10.0.5) which includes the vmware-resolutionSet utility. You can find it under '/Library/Application Support/VMware Tools'. One thing to note is that there is a known issue right now for VMware Fusion 8.1 related to NAT and port forwarding, you may want to hold off on upgrading if you rely on this feature.

The syntax for the vmware-resolutionSet utility is pretty straight forward, it accepts a width and height argument. Make sure to use "sudo" if you want the display resolution to persist through a system reboot. For example, to set a 1920x1080 resolution, you would run the following command:

./vmware-resolutionSet 1920 1080

Note: Ensure you have sufficient video memory configured for your VM for larger display resolutions. In the example above, I have 16MB configured for my Mac OS X VM which would give you a max resolution of 2560x1600.

If everything was successful, you should see that both the "Requested resolution" and the "Effective resolution" match in the output. If output does not match, it most likely means you need to increase the video memory and you can refer to this VMware KB 1003 for more details. If we take a look at our Mac OS X VM, we should now see that our new custom display has taken effect. Below is a screenshot of a Mac OS X 10.11 (El Capitan) running on vSphere 6.0 Update 1 configured with a 1920x1080 resolution.

One other thing to note is if you plan on using higher display resolution than 2560x1600, you may need to configure some additional VM Advanced Settings due to use of framebuffers that are larger than 16MB. In this case, you would need to also add the following two advanced settings to the VM which can be done using the vSphere Web/C# Client or the vSphere API. For example, if I want a 2880x1800 display resolution, I would add the following:

svga.maxWidth = "2880"
svga.maxHeight = "1800"

Lastly, I would like to give a big thanks to Michael Udaltsov, the Engineer who is responsible for creating the workaround and providing me with some additional context to this change in behavior. I know our customers will greatly appreciate this workaround!

Building minimal vSphere demo lab using VMware Fusion/Workstation with only 8GB memory?

10/16/2015 by William Lam 7 Comments

After tweeting this update last week, I received quite a few questions on how I was able to squeeze a vCenter Server Appliance (VCSA) & ESXi 6.0 Update 1 along with a VMware Photon VM, all running on my Mac Book Air with only 8GB of memory. Although, I was not able to make use of my demo which was for my vSphere Content Library session at VMworld Europe this week; I thought I would still share the details on how I built this vSphere lab environment which could come in handy for others.

I was able to squeeze VCSA 6.0 & ESXi 6.0 Update 1 & Photon VM on Mac Book Air w/only 8GB of memory. Chrome & terminal ran fine as well!

— William Lam (@lamw) October 7, 2015

I wanted to run everything on my Mac Book Air primarily for the convenience factor so I did not have to bring my Mac Mini which may not be ideal for traveling aboard. The performance and responsiveness of the environment was actually pretty good and I was able to also access the vSphere Web Client using Google Chrome as well as OS X terminal for CLI operations without any problems. It definitely helps if you place all VMs on SSDs, which is especially useful if swapping occurs since we are overcommitting the physical memory.

Below are the instructions for building this environment and here is a quick summary of the expected memory configuration for the three VMs.

Virtual Machine	Memory
Embedded vCenter Server Appliance VM	5GB
ESXi VM	3GB
Photon VM	384 MB

Step 1 - Download the VCSA & ESXi 6.0 Update 1 ISO (or any other version you wish to run). You will need to extract the contents of VCSA ISO and the OVA is located in /vcsa/vmware-vcsa and you will need to add the .ova extension.

Source: Ultimate automation guide to deploying VCSA 6.0 Part 1: Embedded Node

Step 2 - We will need to configure memory overcommitment for VMware Fusion/Workstation to allow for the majority of the memory to be swapped to be able to run our minimal vSphere environment. You will need to set the value of prefvmx.minVmMemPct to 25 by adding the following line to the respective configuration file shown in the table below.

prefvmx.minVmMemPct = 25

Hypervisor	Configuration File
VMware Workstation	C:\ProgramData\VMware\VMware Workstation\config.ini
VMware Fusion	/Library/Preferences/VMware\ Fusion/config

Source: Quick Tip – How to enable memory overcommitment in VMware Fusion?

Step 3 - Deploy the VCSA OVA to either your VMware Fusion or Workstation deployment and ensure you do not power on the VM. We will need to make the following edits to the VCSA's VMX file to ensure it is properly configured when it is powered on. Below is an example of the VMX parameters you will need to add before powering on the VM.

guestinfo.cis.deployment.node.type = "embedded"
guestinfo.cis.vmdir.domain-name = "vghetto.local"
guestinfo.cis.vmdir.site-name = "vghetto"
guestinfo.cis.vmdir.password = "VMware1!"
guestinfo.cis.appliance.net.addr.family = "ipv4"
guestinfo.cis.appliance.net.addr = "192.168.1.54"
guestinfo.cis.appliance.net.pnid = "192.168.1.54"
guestinfo.cis.appliance.net.prefix = "24"
guestinfo.cis.appliance.net.mode = "static"
guestinfo.cis.appliance.net.dns.servers = "192.168.1.1"
guestinfo.cis.appliance.net.gateway = "192.168.1.1"
guestinfo.cis.appliance.root.passwd = "VMware1!"
guestinfo.cis.appliance.ssh.enabled = "true"

Source: Quick Tip – How to enable memory overcommitment in VMware Fusion?

Step 5 - Once the VCSA has successfully been configured and you can connect to it using the vSphere Web Client, you can then power it off and reduce the memory from 8GB to 5GB.

Step 4 - Create a new VM using the ESXi 6.x GuestOS type for running your Nested ESXi VM and stick with the defaults of 4GB of memory to be able to install ESXi. Once the VM has been created, go ahead and install ESXi using the ISO as you normally would.

Step 5 - Once the ESXi VM has successfully been installed and booted up, you can then power it off and reduce the memory from 4GB to 3GB.

Step 6 (Optional) - If you wish to play with VMware Photon, you can also install Photon using the ISO which can be downloaded from here or deploy using the OVA which can be downloaded from here.

For folks who have more memory in their system, you could add an additional two Nested ESXi VMs to then run a full VSAN setup and then you will have a pretty powerful, with minimal resource footprint that you can bring with you anywhere to run demos or for development and testing purposes. I also highly recommend making use of the "Suspend" operation when you need to quickly get access to memory or run other applications and this also allows you to quickly resume the entire environment in just a few seconds without having to power down the entire setup which will take much longer.

ESXi 6.0 on Apple Xserve 2,1

08/20/2015 by William Lam 7 Comments

I really enjoy hearing from my readers, especially when they share some of the unique challenges they have come across or boundaries they have pushed with our VMware software. Several weeks back I received an interesting email from a reader named John Clendenen who was able to get ESXi 6.0 running on both his Apple Xserve 3,1 as well as Xserve 2,1! To be honest, I was pretty surprised to hear that this worked and not only that, there was not a whole lot that John had to do to get it working. I thought this was a pretty cool setup and asked if John was interested in sharing more details with the VMware/Mac OS X community in case others were interested.

Disclaimer: This is not officially supported by VMware, please use at your own risk.

*** This is a guest blog post from John Clendenen ***

For the past 5 years, I have lived in New York where I work at various print and post-production studios. The IT situations in many of these locations is often home-grown and sub-adequate, if not barely functional, so I took it upon myself to learn OS X Server administration. I have a background in computer science and networking, so it wasn’t a huge leap to step into administration. I was able to gradually build one studio’s IT infrastructure form a single AFP share with incessant permissions problems, to an Open Directory driven, single sign-on infrastructure with mobile homes, messaging etc.. However, with each OS X Server update, things broke, and I was spending a lot of time putting out fires.

This led me to pursue virtualization in hopes of isolating services on separate machines, without telling the studio to go buy 8 Apple Mac Minis. However, I needed to learn how to do this, before pitching it to them, so I looked on Craigslist for some cheap hardware. I found an Xserve 2,1, which I was able to talk down to $100. I also found a brief post in a thread that said that the Xserve 2,1 ran ESXi 5.5 without issue. I figured I’d take the plunge and just resell it on eBay if it didn’t work out.

Since then, my home lab has grown considerably, and I have learned that the best way to provide Mac services, is simply to use Linux (I’ve had a great experience with Netatalk). That said, Open Directory, Software Update, Caching and a few other services still need to be run on a Mac, so it’s still necessary to have the hardware. For a while, I had 3 Xserves, all purchased very cheaply, running VMs. I just sold one, and will sell another in the next month or two in favor of some Supermicro hardware (I really am mostly running Linux VM’s at this point). I’ll keep the one to run a few Mac OS X VMs. I’m still working on getting Samba 4 to work as the PDC, but once that is running smoothly, I’ll have a functional environment that I can take to the studios where I work (with approved hardware of course). Then I’ll have an improved work experience, while also pulling some extra income on the installation/maintenance.

Anyway, you’ve come here to read about running ESXi 6.0 on an Xserve 2,1. Well, it works. It’s 7 years old, and not officially supported, but you already know that. So, if you were considering it, I doubt anything here will dissuade you. On top of that, there’s not much for me to say other than, it works. No tricks to it, just install and you’re off.

That said, I do have some recommendations and tips to help you if you want to get the most out of this hardware. 2 months ago, since I swapped out the RAID card for the standard backplane/interconnect and upgraded the RAM. It hasn’t skipped a beat since.

My System

This is my home custom 13u rack with sound insulation and active air flow. Networking is in the back (Ubiquiti). It sits in a lofted storage area with an A/C and vents into the bathroom.

Here you see an Xserve 3,1 on top of an Xserve 2,1. There’s a blank unit because I just sold an Xserve 2,1 on eBay, and the other 2,1 will be for sale soon as well to make room for a 4 node 2U Supermicro Server. The NAS is comprised of a 4u head unit and a 4U JBOD running Centos. Last of course, is a 2U CyberPower UPS which is really just to condition the power and keep brownouts from taking down the system.

I have about a dozen VM’s running between the 2 Xserves. I have each Mac OS X service separated on it’s own installation. This way I can run updates per service. It’s especially nice to have Open Directory separate from less important services. I also have Debian, Centos and OpenBSD VMs running various services. Getting the Xserve 3,1 running ESXi 6 is possible, but more problematic. Now that I have it working, I’m dropping the 2,1’s simply because the processors aren’t multithreaded. I am currently working on a companion article to this one, detailing my experience with the Xserve 3,1, so that information will be available soon.

Storage

1. Don’t use the RAID backplane/interconnect or count on using it. ESXi 6 does not recognize it, and RDM appears to work at first and then will crash your VM and never show up again. You can have it installed in the Xserve without any issue, but you’ll get a lot more mileage out of the hardware if you have the standard backplane/interconnect.

The backplane you want appears periodically on eBay. The part number is: 661-4648

2. If/once you have the non-RAID backplane/interconnect, keep in mind that it is SATAII and will only support 3Gb/s. I am using 3 old 500Gb WD RE3’s, but I’d recommend using some older SSDs that will max out the SATAII interface without paying for speed you can’t use. Be sure you consult the Apple Drive Module compatibility page to make sure you have the right drive caddies. They all fit, but they don’t all work.

Apple Drive Module Compatibility Chart: https://support.apple.com/en-us/HT1219

3. PCIE Flash is a good idea, whether you use it for cache, VM’s or as the ESXi boot disk, it is by far the fastest storage option. I have not invested in it, but the good people at Mushkin have told me their Scorpion PCIE flash will work in ESXi 6. Please contact them yourself though before investing in one of their cards. I have not tested them. While this will give you the best performance out of your Xserve 2,1, it seems like overkill to me, but hey, if you want to push this thing, you might find some meaningful performance gains here.

Mushkin Scorpion PCIE Flash: http://www.poweredbymushkin.com/index.php/products/solid-state-drives.html

4. It might occur to you that replacing the optical drive with an ssd might be a good idea. While MCE Technologies makes “Optibay” for Xserve 2,1, the connection is IDE, so this is not recommended. I also don’t know if ESXi would recognize it. My gut says probably, but again, it’s too slow to be useful. It isn’t that cheap either.

MCE Technologies Optibay: http://store.mcetech.com/mm/merchant.mvc?Store_code=MTOS&Screen=PROD&Product_Code=OBSXGB-XS

Network

There are a lot of options here. You can plug any network card that you want really as long as you can at least find an anecdotal account of it working on ESXi 6.

I have one such anecdote for you, and it is also a strong suggestion. The company Silicom makes/made several models of gigabit NICs which are all incredibly inexpensive, are all over eBay and work in ESXi 6. Buy the 6 port model. It’s cheap and you’ll get great scaling with ESXi load balancing across 8 gigabit ports.

The Xserve 2,1 has the added advantage here of an optional PCIX riser. If your model has one, or you find one for cheap, you can save even more on the Silicom NIC. The PCIE models go for $60-$80 on eBay, while the PCIX models go for $40.

Memory

ECC DDR2 is pretty cheap and easy to find used. I recommend memory4less though. I had to return some RAM that was mislabeled from a random eBay distributor. Memory4less will get it right. http://www.memory4less.com/

Processors

One great perk of the Xserve 2,1, is that you can upgrade the single processor hardware to dual processor. You can pick up an additional processor on eBay for $40 or so, but you’ll need to get a heat sink as well. The single processor units come with a fake aluminum heat sink, but do not use it. You want a copper one. I believe the heat sinks in the Xserve 1,1 are the same. Don’t forget the thermal paste.

Minor Issues

1. The Performance tab throws an error.

2. Not sure about the hardware sensors, but it looks like not everything is working even if it’s showing up. I did not do any testing here.

Stay tune for Part II of John's guest blog post on running ESXi 6.0 on Xserve 3,1.

Heads up: ESXi 5.x & 6.0 unable to detect newer Apple Mac Pro 6,1 local SSD Device

07/27/2015 by William Lam 12 Comments

Over the last couple of weeks there have been several reports coming in from customers that the local SSD device found in newer Apple Mac Pro 6,1 were no longer being detected by ESXi. Starting with ESXi 5.5 Patch03 and ESXi 6.0, the Apple Mac Pro 6,1 was officially supported but it looks like the latest versions of the Mac Pro 6,1 that are being shipped contain a slightly different local SSD device which is not recognized by ESXi.

This was not the first time that this has happened, when the 2014 Mac Mini were first released, they too had a similar issue in which a custom VIB was required to get the internal device to get recognized by ESXi. There is an internal bug (PR 1487494) that is currently tracking the issue and if you are also experiencing this problem, please file an SR and have the GSS Engineer attach your case to this bug.

In the meantime, there is an unofficial workaround which was discovered by one of my readers (Mr. Spock) that by installing the community SATA-XACHI VIB over on Andreas Peetz VIB Depot site, both ESXi 5.5 and ESXi 6.0 will then recognize the local SSD Device. You will need to either use VMware Image Builder or Andreas ESXi Customizer tool to create a custom image if you decide to install ESXi directly on the local SSD device. I personally would recommend installing ESXi on USB device, this would allow you to install the VIB as a post-installation and not requiring a custom ESXi image.