apple

ESXi 6.5 support for Apple Mac Pro 6,1

01/09/2017 by William Lam 56 Comments

I know several of you have reached out asking about the support for ESXi 6.5 on the Apple Mac Pro 6,1 but as of right now, the Mac Pro 6,1 is currently not supported with ESXi 6.5. I know this is not ideal especially for customers who wish to take advantage of the latest vSphere release. The good news is that VMware is in the process of testing the Apple Mac Pro 6,1 for ESXi 6.5, however there is not an ETA on when this will be completed by.

Some of you might be wondering why this did not happen earlier? The primary reason is that hardware certification for ESXi is actually performed by the hardware vendors. Once a vendor completes the certification for a particular hardware platform or component, they submit the results to VMware and the VMware HCL is updated. If there is a piece of hardware that is not on the VMware HCL today, it is definitely worth reaching out to your hardware vendor to inquire about its status.

In Apple's case, it unfortunate as they do not participate in VMware's Hardware Certification program for ESXi which makes certification challenging. VMware intends to continue to support customers who require the use of Mac OS X Virtualization and will work towards getting the Mac Pro's certified for latest version of vSphere as mentioned earlier. Historically, testing and certifying ESXi for Apple hardware does take an additional amount of time and in some cases, code changes may even be required due to unexpected hardware changes from Apple.

I hope this gives customers some additional insights into how Apple hardware is certified for ESXi. If you would like to see this improved in the future, you may want to reach out to Apple and provide them with your feedback.

Now ... before you close this blog post thinking it is going to take awhile before there is going to be an update regarding ESXi 6.5 and Mac Pro 6,1, please continue reading further 🙂

UPDATE (07/28/2017) - ESXi 6.5 Update 1 just GA'ed yesterday and is fully supported with all current Apple Mac Pro 6,1 (as you can see on the HCL here) and the workaround mentioned below is no longer required. This means you can install ESXi without any modification to the image.

UPDATE (03/25/2017) - VMware has just published the following VMware KB 2149537 which outlines the officially recommended workaround to install ESXi 6.5 onto the Apple Mac Pro 6,1. The VMware HCL has also been updated to include the Apple Mac Pro 6,1 4-Core, 6-Core, 8-Core & 12-Core systems. In a future release of ESXi, the workaround will not be required and ESXi will just install out of the box. This temporarily workaround is to enable customers who wish to run the current version of ESXi 6.5 which includes GA release, 6.5a and 6.5p01.

VMXNET3 driver now included in Mac OS X 10.11 (El Capitan)+

10/01/2016 by William Lam 12 Comments

Yesterday I received a pretty interesting comment from one of my Twitter followers @NTmatter who wrote:

@lamw Just noticed that OSX has a VMXNET3 driver. Have to edit the vmx file to actually use it, but it's there! AppleVmxnet3Ethernet.kext

— Thomas Johnson (@NTmatter) September 30, 2016

This is a pretty neat find because currently today, the only network adapter that is functional with an Apple Mac OS X guest running on either VMware vSphere or Fusion is the e1000{e} driver. This update was definitely news to me and after sharing it internally to see if I could find some more details, it turns out this news also came as surprise to the folks internally. Darius, one of the Engineers who I frequently reach out to on Apple related topics did some digging and found out that Apple started to bundle this VMXNET3 driver starting with Mac OS X 10.11 (El Capitan) release. You can find the driver located in /System/Library/Extensions/IONetworkingFamily.kext/Contents/PlugIns/AppleVmxnet3Ethernet.kext

Disclaimer: Given that this VMXNET3 Mac OS X driver was not developed by VMware nor has it been tested by VMware, it currently would not be officially supported by VMware.

If you wish to try out the VMXNET3 driver, you will need to install Mac OS X 10.11 or newer on a VM running on vSphere or Fusion. By default, the only available network adapter type is e1000{e}. To add a VMXNET3 network adapter, you can either manually tweak the .VMX file or you can easily add it by using either the vSphere Web/C# Client or ESXi Embedded Host Client. Below are the instructions on configuring the VMXNET3 network adapter for your Mac OS X guests.

Step 1 - Remove the existing network adapter and then temporarily change the GuestOS type to "Other" (no need to save setting, just update it in VM reconfigure wizard) so that you will be allowed to add a VMXNET3 network adapter. Once you have added it to the VM reconfigure wizard, go ahead and toggle back the GuestOS type to Mac OS X 10.10 and then save the settings as shown in the screenshots below.

Step 2 - Open a terminal inside of the Mac OS X guest and run the following command to load the VMXNET3 driver:

sudo kextload /System/Library/Extensions/IONetworkingFamily.kext/Contents/PlugIns/AppleVmxnet3Ethernet.kext

Step 3 - You can verify that the VMXNET3 driver was successfully loaded by running the following command:

kextstat | grep -i vmxnet3

Once the driver has been loaded, you should now have networking connectivity to your Mac OS X VM using the VMXNET3 network adapter. Below is a screenshot of the system info showing the VMXNET3 network adapter.

In addition to having an optimized networking when using the VMXNET3 driver, the other benefit is being able to get a link speed of 10GbE which is something customers have been inquiring about when virtualizing Mac OS X guests. Below is a screenshot of the media link shown in this Mac OS X 10.11 guests.

Although this a great development for Apple customers who uses VMware vSphere and Fusion, it also does raise an interesting question on whether Apple would be officially supporting this VMXNET3 driver going forward? If I do receive any more details on this, I will update the article. Until then, you can play with this new capability if you are running Mac OS X 10.11 or greater on VMware. Big thanks to Thomas for this great find and sharing it with the VMware Community!

ESXi Thunderbolt Driver to Fibre Channel Storage from ATTO

09/12/2016 by William Lam 6 Comments

One of the things I always enjoy doing at VMworld, when I am not running around and I have a few minutes to myself, is to check out the VMware Solutions Exchange. This is where you can learn and interact with hundreds of our VMware Certified Partners showcasing their new solutions and innovations that they have built on top of VMware's products.

UPDATE (08/22/17) - ATTO's ESXi Thunderbolt Driver is now officially on the VMware HCL, please see this blog post here for more details.

While walking through the show floor, I had stopped by the ATTO Technology booth who has been a long time partner of VMware in the storage and networking connectivity space. What caught my eye was that they had just released a Beta of an ESXi Thunderbolt Driver in the form of an ESXi VIB that would allow customers to connect their Apple Mac Pro 6,1 using the Thunderbolt 2 interface to an external Fibre Channel storage array. I believe ATTO might be the first vendor ever to produce a Thunderbolt Driver for ESXi. This is really exciting news if you ask me, especially as more and more of our customers are looking to virtualize Mac OS X guests in their Datacenters using vSphere.

Historically, the only option to connect a Mac Pro 6,1 to an external Fibre Channel array was to use something like a Sonnet Chassis. Now, you can potentially connect up to 6 of the built-in Thunderbolt 2 interfaces on the Mac Pro's to your external storage array using this new solution from ATTO. Before I go into some of the details, ATTO did want me to mention that this solution is currently not officially supported by VMware nor is it on VMware's HCL. ATTO will be providing full support on their software as well as VMware's software stack during the duration of the beta program. In terms of official certification on VMware's HCL, I suspect that it will most likely depend on customer demand which would influence whether ATTO applies for an official certification, which again, would be the first of its kind for Thunderbolt.

The way in which this solution works is that you install the ATTO Thunderbolt Driver on your ESXi host and this will allow it to communicate with an ATTO ThunderLink device which provides the Thunderbolt 2 to Fibre Channel connectivity. You have the option of using either the FC2082 which provides 20Gb/s Thunderbolt 2 (2-port) to 8Gb/s FC (2-Port) Device or the FC2182 which provides 20Gb/s Thunderbolt 2 (2-port) to 16Gb/s FC (2-Port) Device. Below is a diagram from the ATTO digital solution brief on Thunderbolt Driver for ESXi which outlines the configuration.

If you are interested in taking part in ATTO's ESXi Thunderbolt Driver Beta program or would like to learn more about the solution, you can reach out directly to Carllene Mowry (cmowry@attotech.com) who is running the program. For more information be sure to check out the ATTO digital brief on Thunderbolt Driver for ESXi.

Lastly, I was also fortunate to have a quick chat with Carllene and team to get a few additional exclusive tidbits on some of the things the ATTO team is working on next. The first of which is support for the Thunderbolt 3 (aka USB-C) interface to Fibre Channel which will be quite nice for newer platforms that include that interface, including home lab setups such as the Intel NUC. Speaking of Intel NUC, this is just one of the many other platforms that include either Thunderbolt 2 or 3 interfaces. Although the solution today is specifically supporting the Mac Pro, I know ATTO folks are interested to hear from customers on other systems with Thunderbolt interface and providing similiar capabilities.

The other really exciting development that is currently being investigated is support for Thunderbolt 2 or 3 to 10GbE connectivity on ESXi. As you can imagine, this is really going to open up some really cool new use cases, especially around things like VSAN which can easily benefit from this. It is still in early development but from my understanding, ATTO is already seeing a lot of interest in this area as well as how this might work with VSAN. I am hoping I will be able to share more details as this further develops. If any of these updates sounds interesting, do leave a comment to let the ATTO folks know and I will make sure they monitor the thread.

Apple Mac Pro 6,1 PCIe SSD issue resolved w/ESXi 6.0 Update 2

03/15/2016 by William Lam 6 Comments

Early last year, the new Apple Mac Pro 6,1 (aka black can design) was certified and fully supported on vSphere 6.0 which I had blogged about here. Several months later, customers discovered that some of the newer Mac Pro 6,1 units were shipping with different model of their PCIe SSD device than what was originally released at GA. This was problematic because ESXi was not aware of this newer device and could not detect during or after installation. Although a work around was identified for customers looking to install either ESXi 5.x or 6.x on the newer Apple Mac Pros, it definitely was not ideal.

It has taken a bit longer than expected, but the issue has now been resolved with the latest release of ESXi 6.0 Update 2. A similar fix will be available for customers running ESXi 5.5 in a future update. You can find the direct download for ESXi 6.0 Update 2 in link below which includes a pointer to the release notes in case you are interested in other fixes included in this release.

vSphere ESXi 6.0u2 - https://my.vmware.com/web/vmware/details?downloadGroup=ESXI60U2&productId=491&rPId=10348

ESXi 6.0 on Apple Xserve 3,1

11/17/2015 by William Lam 76 Comments

A couple of months ago, I shared a guest blog post from one of my readers John Clendenen who was able to get ESXi 6.0 running on an Apple Xserve 2,1. At the end of that article, it was hinted that John was also looking into getting ESXi 6.0 running on an Apple XServe 3,1 and you can the details below after several months of investigation.

Disclaimer: This is not officially supported by VMware, please use at your own risk.

*** This is a guest blog post from John Clendenen ***

First an update on my Xserve 2,1’s. I had them running for over 100 days without any issue! However, now that I have the 3,1 working reliably, it is time that I part ways with my Xserve 2,1’s. I currently have them up on eBay. Here is the link: http://www.ebay.com/itm/231752771080?ssPageName=STRK:MESELX:IT&_trksid=p3984.m1555.l2649

Anyway, onto the Xserve 3,1.

I came across an Xserve 3,1 on eBay about a year ago. It was badly photographed, and the seller didn’t really know what he/she had. It wasn’t getting much attention, so I thought I might get it cheap. I ended up paying $500 for it which I felt ok about, but not great.

When it arrived, it had no processors, heatsinks or airflow duct. I immediately messaged the seller, and was able to get $350 refunded to me. I found the missing parts for under $100 over the next few weeks, and developed an intimate understanding of the Xserve 3,1 hardware.

At this point, I had no familiarity with vSphere at all. I was running OS X server and virtualizing a few services in Fusion. It was only through researching the Xserve 3,1 to find the missing hardware that I discovered that VMware had supported once as an ESXi 5 host. This made me wonder if it might still be possible to run ESXi on it, despite it no longer being supported.

I have found, after a considerable time investment, that the Xserve 3,1 can run ESXi 6, just as I found the Xserve 2,1 can run ESXi 6. However, unlike the Xserve 2,1, the Xserve 3,1 took months of troubleshooting before I had it running as a reliable ESXi host.

As it turns out, despite how much time it took me to get it working, there are only 2 serious issues with the Xserve 3,1 running ESXi 6. The first is somewhat specific to my configuration, but the second will be relevant to all configurations.

The first issue concerns booting into ESXi on a headless Xserve 3,1. The issue is limited to configurations where ESXi is booting from a drive installed in the optical bay (my original configuration). I have since changed my configuration and swapped the ESXi boot drive from the optical bay to the first hard drive bay. I have had no issue since I made this change.

For my configuration, I used an OWC bracket to replace the optical drive with an SSD. I installed ESXi onto it without issue. During installation, it was connected to monitor, keyboard, etc. I ran some VM’s on it to make sure it worked, and there were zero issues. I was relieved! So, I put it in the rack, wired it up and turned it on. Nothing. The Xserve lit up, and it was clear that it got through POST, but ESXi was clearly not booting.

Long story short, when no monitor is plugged into the Xserve 3,1, it will not automatically boot into ESXi if the boot drive is installed in the optical bay. The Xserve boot options can even be programmed through the front panel, but no configuration will make it reliably boot from the optical bay when a hard drive is installed. It is truly baffling, and if anyone has some insight here, or if it is a problem specific to my particular Xserve, I would love to know.

The solution, in my case, was to plug a keyboard into the Xserve, and hold down option for a few minutes while it boots (bringing up the boot options). Once all LED activity has normalized and the fan has settled down, I released the option key and pushed the arrow buttons. I think you only need to push the up button, but I always just pressed all of them to be sure. Then I pressed enter, and ESXi will boot. I have since simply swapped the boot drive to the first drive bay. Ideally, I’d have the other drives in the hot-swap bays, but I felt it was too much trouble to keep it in the optical bay.

The second issue concerns the onboard NIC. Once I had ESXi up and running, everything worked fine for anywhere between a few hours and 2 days, after which the Xserve 3,1 host would disappear from the VCSA and become completely unresponsive (no ping/ssh/etc). The length of time before failure made this issue especially difficult and time consuming to diagnose.

After nearly a month of frustration and disappointment, I determined that ESXi actually continued to run, but all network connectivity was ceasing. The only solution I have found is to install a 3rd party NIC and completely avoid using the onboard NIC. Even in standby, the onboard NIC can cause problems, but when it is completely unused, both for management and VM traffic, it no longer causes any problems.

This has been superficially improved with the last update, but use of the onboard NIC should still be completely avoided. The ESXi host will remain accessible via the VCSA, but the network management will become grayed out after a day or so. I suspect this is a driver issue in ESXi, but I really do not know.

Beyond these 2 issues, I have had no problems. Since the last update, even the performance and hardware status tabs are functional. RDM is not available, but not recommended in the first place. The Apple RAID backplane will not be recognized, but this was even the case in ESXI 5 when it was officially supported by VMware.

I hope that my efforts here will save others a lot of time and frustration. I think that for a lot of IT infrastructures, ESXi on an Xserve might make sense. It can run non-critical OS X services (which are hopefully the only kind of services you’re trying to run in OS X).

Summary

Completely avoid using the on-board NIC. Silicom NIC’s are recommended.
Find a standard backplane. The RAID backplane is useless in ESXi.
A 2.5” drive can be installed in the optical bay, but booting from it is problematic

The Xserve 3,1 with the Silicom NIC installed

The 6 ports are a tight squeeze, but they just fit. My other 2 EXSi hosts are Supermicro Nodes, also with Silicom NIC’s and I had to use a Dremel to grind off part of the chassis to make all the ports accessible. But the Xserve works out of the box.

The OWC SSD “Data Doubler” bracket in the optical bay. Booting from here is a pain, but putting an additional SSD here works great for host caching.

The standard backplane is difficult to find, but is a great asset for vSphere. It is easy to distinguish it from the RAID backplane which would have a heat sink here.

There are no complications during installation/initial configuration.

Apologies for not having a longer uptime. I updated to ESXi6.0U1a 12 days ago, but I’ve had the Xserve 3,1 up for months. If something changes, I will post an update here, but I am confident that the system is stable.

This is the final stage of my home lab. The Xserve 3,1 is 1 of 3 ESXi hosts. These are accompanied by a primary domain controller (Samba4), a media server (Emby) and a home-grown NAS (Centos7). Networking in the back is Ubiquiti. I use this lab to prototype production environments for clients, and of course to run my home media services 🙂

Heads Up – Workaround for changing Mac OS X VM display resolution in vSphere & Fusion

10/22/2015 by William Lam 50 Comments

For customers who are running Mac OS X 10.9 (Mavericks) or newer in a Virtual Machine, you may have noticed that you can no longer set a custom display resolution beyond the default 1024x768 in either VMware Fusion and vSphere, regardless of the amount of video memory that has been allocated. The reason for this behavior is that Apple has changed the way in which it remembers previously used modes and would automatically fall back to this versus retaining the custom mode using the Display Preferences. Given this is a non-trivial fix, VMware Engineering has been working hard on a providing a workaround which would still allow users to set a custom resolution from within the GuestOS.

The workaround that has been developed is a tiny standalone command-line utility called vmware-resolutionSet which runs within the Mac OS X Guest and allows you to configure a custom display resolution. You will need to ensure you have VMware Tools installed and running before you can use this utility. As of right now, customers can get a hold of this utility by filing an SR with VMware Support and referencing PR 1385761. Although this tool has not been officially released and must go through the standard release process, the plan is to include it in a future update of VMware Tools and will available for use with both VMware Fusion and vSphere.

UPDATE (12/11/15) - Thanks to reader @elvisizer, it looks like the latest VMware Fusion 8.1 release now includes an updated version of VMware Tools (10.0.5) which includes the vmware-resolutionSet utility. You can find it under '/Library/Application Support/VMware Tools'. One thing to note is that there is a known issue right now for VMware Fusion 8.1 related to NAT and port forwarding, you may want to hold off on upgrading if you rely on this feature.

The syntax for the vmware-resolutionSet utility is pretty straight forward, it accepts a width and height argument. Make sure to use "sudo" if you want the display resolution to persist through a system reboot. For example, to set a 1920x1080 resolution, you would run the following command:

./vmware-resolutionSet 1920 1080

Note: Ensure you have sufficient video memory configured for your VM for larger display resolutions. In the example above, I have 16MB configured for my Mac OS X VM which would give you a max resolution of 2560x1600.

If everything was successful, you should see that both the "Requested resolution" and the "Effective resolution" match in the output. If output does not match, it most likely means you need to increase the video memory and you can refer to this VMware KB 1003 for more details. If we take a look at our Mac OS X VM, we should now see that our new custom display has taken effect. Below is a screenshot of a Mac OS X 10.11 (El Capitan) running on vSphere 6.0 Update 1 configured with a 1920x1080 resolution.

One other thing to note is if you plan on using higher display resolution than 2560x1600, you may need to configure some additional VM Advanced Settings due to use of framebuffers that are larger than 16MB. In this case, you would need to also add the following two advanced settings to the VM which can be done using the vSphere Web/C# Client or the vSphere API. For example, if I want a 2880x1800 display resolution, I would add the following:

svga.maxWidth = "2880"
svga.maxHeight = "1800"

Lastly, I would like to give a big thanks to Michael Udaltsov, the Engineer who is responsible for creating the workaround and providing me with some additional context to this change in behavior. I know our customers will greatly appreciate this workaround!