A killer custom Apple Mac Mini setup running VSAN

*** This is a guest blog post from Peter Bjork ***

The first time I was briefed on VMware VSAN, I fell in love. I finally knew how I would build my Home Lab.

Let me first introduce myself, my name is Peter Björk and I work at VMware as Lead Specialist within the EMEA EUC Practice. I fortunately have the opportunity to limit my focus on a very few products and truly specialize in these. I cover two products; VMware ThinApp and VMware Workspace Portal and one feature; the Application Publishing feature of VMware Horizon 6. I’m an End-User application kind of guy. That said, you should understand that I’m far from your ESXi and vSphere expert. If you want to keep up with the latest news in the VMware End-User Computing space make sure to follow me on Twitter, my handle is @thepeb. When I’m not a guest blogger, I frequently blog on the official ThinApp and Horizon Tech blogs.

In my role I produce a lot of blog posts and internal enablement material. I perform many tests using early code drops and on a daily basis I run my home lab to deliver live demos. I need a Home Lab that I can trust and that supports all my work tasks. I started building my lab many years ago. It all started with a single mid tower white box, but pretty soon I ran into resource constraints. I started to investigate what my next upgrade would look like.

I had a few requirements:

  • Keep the noise down
  • Shouldn’t occupy that much space
  • Should be affordable
  • Modular, I do not have money to buy everything upfront so it should be something I could build on top of.
  • Should be able to run VMware ESXi/vSphere
  • Should be cool

Being an Apple junky for many years, my eyes soon fell on the Apple Mac Minis and I stumbled over this great blog by William Lam that you are reading right now. At the same time I started to hear about VSAN and my design was pretty much decided. I was going to build a Mac Mini cluster using VSAN as storage. While I have Synology NAS I only use it for my private files. It is not used in my home lab and for reasons I can not really explain I wanted to keep it separate and use a separate storage solution for my home lab.

Now that I have decided to build my home lab, I went and bought my first Mac Mini. To keep cost down I found two used late 2012 models with i7 CPUs. Since VSAN requires one SSD and one HDD I had to upgrade them using the OWC Data Doubler Mounting Kit. I also upgraded the memory to 16GB RAM in each Mac Mini. This setup gave me some extra resources and together with my old Tower Server I could start building my VSAN Cluster. I started with the VSAN beta. I quickly realized that VSAN didn’t support my setup. I waited for the GA release of VSAN and on the release date I decided to go for a pure Mac Mini VSAN setup so I stole my families HTPC which was a late 2012 Mac Mini model with a i5 CPU. (I managed to get away with it because I replaced it with an Apple-TV.) I took one HDD and the SSD from my old Tower Server and put it into the i5 Mac Mini. While I managed to get VSAN up and running it was only running for an hour or so before I lost one disk in my VSAN setup. I recovered the disk back up through a simple reboot but then the next disk went down. The reason for the instability is that the GA release of VSAN did not support the AHCI controller. Hugely disappointed I had to run my home lab on local attached storage and my dreams of VSAN was just that, dreams. In all my eagerness I’ve already migrated the majority of my VMs onto the VSAN Datastore so I pretty much lost my entire home lab.

After complaining to my colleagues, I found out that AHCI controller support for VSAN was coming in vSphere 5.5 U2. I heard it was likely to solve my problems. So the 9th October came and vSphere 5.5u2 was finally here. To my joy, my three Mac Minis were finally able to run VSAN and it was completely stable.

Lets take a closer look at my setup. Below is an overview of the setup and how things are tied together.

Home Lab Picture
My VSAN Datastore houses most of my VMs. My old Tower Server is connected to the VSAN Datastore but does not currently contribute any storage. On the Tower Server I host my management VMs. Since I got burned loosing all my VMs, I made sure I keep my management VMs on a local disk in the Tower Server. Since my environment has been running quite stable for nearly two weeks now I’m considering migrating all of my VMs onto the VSAN Datastore.

I have noticed one issue so far which is with my i5 based Mac Mini. One day it was reporting not connected in the vCenter Server. The machine was running but I got a lot of timeouts when I pinged it. While I was thinking about rebooting the host it showed up as connected again and since then I’ve not noticed any other issues. I suspect the i5 CPU isn’t powerful enough to host a couple VMs and being a part of the VSAN. When I saw it disappear it might have been running some heavy workloads. So with this in mind I would recommend running i7 Mac Minis and leave the i5 models for HTPC work loads :).

Another thing I’ve noticed is that the Mac Minis are running quite hot. There is no power saving functionality that is active and my small server room doesn’t have cooling. The room is constantly around 30-35 degrees Celsius (86-95F) but the gear just keeps on running. The only time I got a little bit worried was when the room’s temperature peaked at 45 degrees Celsius (113F), for Sweden, that is an exceptionally warm summer day. Leaving the door to the room opened for a while helps cool things down. I’m quite impressed by the Mac Minis and how durable they are. My first two Mac Minis have been running like this for well over a year now.

IMG_2810
Here’s a picture of my server room. While I do have UPS there is no cooling or windows so the room tends to be quite warm. Stacking the Mac Minis on top of each other doesn’t really help cooling either. When I started stacking my Mac Minis on top of each other I realized how stupid it would be to have three separate power cords. So I ended up creating a custom Y-Y-Y-cable (last Y is for future expansion).

IMG_2807

IMG_2809

Y-cable inside

Y connector
The power cord is a simple lamp cable (0.75mm2) that has the three original Apple power cables butchered together. The Y-connector was found in a local Swedish hardware store. Since the Mac Mini’s maximum continuous power consumption is 85W, a 0.75mm2 cable would work perfectly. A 2 meter (6.56 feet) long 0.75mm2 cable is able to support at least 3A. My three Mac Minis only consume 1.1A (3x85W / 230V = 1.1A). In 120V countries you would have 2.5-3A running through the cable but this wouldn’t be a problem.

Since the Mac Minis only have a single onboard NIC and I wanted to have two physically separated networks I had to get an Ethernet Thunderbolt NIC. As shown in the overview picture I’m am running both VSAN traffic and VM traffic over the same NIC. This is probably not ideal from a performance point of view but for my EUC related workloads I’ve not noticed any performance bottlenecks. On the other hand, I’m very pleased with the performance and with the benefits of having shared storage, so things like DRS and vMotion can deal with the balance between my hosts and I’m super happy with this setup.

I found that the easiest way was to use VMware Fusion to install ESXi onto a USB key. Then I simply plug in the USB key in my Mac Mini and I’m up and running. I need to use external monitor and keyboard to configure the ESXi initially.

As for the next steps I’m planning on getting an SSD and an extra HDD for my Tower Server. This would allow my Tower Server to participate in the VSAN Cluster and contribute additional capacity. If the opportunity arises and I can find another Mac Mini with an i7 CPU for a decent price I would also like to replace the i5. Other than that, I don’t think I need much else. Well, I could always use a little bit more RAM of course (who doesn’t) but disk and CPU runs very low all the time.

VC
Technical details:

  • All Mac Minis are late 2012 models
    • All SSD disks are different models and vendors. Their capacity ranges from 120 to 250GB. Since I’ve had a couple SSD crashes I made sure to purchase the more heavy-duty models offered from the vendors. But none of them are designed for constant use in servers.
    • All Mac Minis have 16GB RAM (2x8GB)
    • I have 1TB HDD in my two i7 Mac Minis and 500GB HDD in the i5 one.
  • ESXi installed on USB key
  • My Tower Server specs are:
    • Supermicro Xeon E3 motherboard, uATX (X9SCM-F-B)
    • Intel Xeon E3-1230 3.2GHz QuadCore HT, 8MB
    • 4x8GB 1333MHz DDR3 ECC
    • Barracuda 500GB, 7200rpm, 16MB, SATA 6Gb/s

To wrap up, I’m very pleased with the setup I‘ve built. It works perfectly for my needs. Lastly, I do recommend having a separate management host, as I found it extremely useful when I had to move VMs back and forth to test earlier releases of VSAN. I also recommend going for the i7 CPU models of Mac Mini for better performance.

Download the VMware ESXi 5.5u2 Mac Mini ISO from virtuallyGhetto:

Apple Thunderbolt Ethernet Adapter VIB:

Apple Mac Pro 6,1 (black) officially supported on ESXi 5.5 Patch03

mac-pro-vsphere-certified-1
The much anticipated support for running vSphere on the latest generation of the Apple Apple Mac Pro 6,1 (black) is finally here with the release of ESXi 5.5 Patch03. Due to unforeseen issues, it has taken a bit longer than expected to get the Apple Mac Pro certified, but VMware Engineering has been working hard to get all the bugs fixed and triaged with Apple and you can now run the latest release of vSphere on the Apple Mac Pro 4-core, 6-core, 8-core & 12-core configuration. I also would like to point out that when the next release of vSphere (.NEXT) is available, the Apple Mac Pro will also be certified and supported.

You can find the ESXi 5.5 Patch03 (ESXi550-201410001) download here and you should download the latest ESXi 5.5 Update 2 ISO and using Image Builder or ESXi Customizer to author a new image which contains the patch required to install ESXi on the new Mac Pro.

The VMware HCL has also been updated to reflect this new update:

Screen Shot 2014-10-16 at 8.36.33 AM
Note: The VMware HCL currently lists 5.5 U2 as the supported release, but you will specifically need ESXi 5.5 Patch03 for this new hardware support. I am hoping to get this further clarified on the HCL.

Here is a screenshot of the latest ESXi 5.5 Patch03 running on an Apple Mac Pro 8-Core system courtesy from VMware Engineering:

esxi-mac-pro-6.1-1

How to evaluate the vSphere VCSA Beta running on VMware Fusion & Workstation?

If you are taking part in the vSphere Beta (available to public to sign up but still under NDA), you may have recently noticed a new milestone release (RC) that has been made available for download. Having been a long time Beta participant when I was customer and still continuing to do so in my current role, the best way to evaluate and test new VMware software is to of course run them on top of vSphere! I know this may not be an option for everyone and the next best thing would be to use VMware Fusion or Workstation.

For those of you who have tried to run the vSphere Beta of VCSA on VMware Fusion or Workstation, you may have found that it does not work as there are some input parameters that are required as part of the new VCSA deployment. These parameters leverages OVF properties which are currently not supported in VMware Fusion and Workstation and therefore the new injectOvfEnv option in ovftool can not be used.

Having said that, VMware Engineering is quite aware that this can be challenging for many customers as well as VMware Employees who make use of Fusion and Workstation on a daily basis. That is why they have built the VCSA to be quite flexible to support both vSphere as well as Fusion and Workstation, however the process may not be completely obvious for the latter. If you inspect the latest VCSA Beta OVA, which you will need to extract from the ISO, you will notice a series of "keys" that begin with guestinfo which is just leveraging custom key/value pairs for the OVF environment.

evaulate-vsphere-beta-vcsa-on-fusion-and-workstation-0
Ideally, these are passed in from the OVF Properties using either the vSphere Web Client or the new VCSA deployment tool. However, due to the lack of OVF Property support, it can also be passed in through the VMX file of the Virtual Machine.

Here are the steps to deploy the VCSA Beta using either VMware Fusion or Workstation:

Step 1 - Download the VCSA Beta which is available as an ISO

Step 2 - Extract the contents of the ISO and add the .ova extension to following file located in vcsa/vmware-vcsa (this is the VCSA OVA)

Step 3 - Upload the OVA using either VMare Fusion or Workstation (you can either double click or just go to File->Open) but make sure you do not power it on after deployment. (this is very important)

Step 4 - Locate the directory in which the VCSA was deployed to and open up the VMX file and append the following (make sure to change the IP information and passwords based on your environment):

Note: The example above is a very basic VCSA deployment which should suffice for the majority of you. If you wish to deploy a more complex scenario, you can inspect the VCSA OVA for additional parameters and see their expected values.

Step 5 - Once you have saved your changes, go ahead and power on the VCSA. At this point, the guestinfo properties that you just added will be read in by VMware Tools as the VCSA is booting up and the configuration will begin. Depending on the speed of your hardware, hopefully in a very short amount of time you will have a fully configured VCSA that is ready for your evaluation and testing.

Here is a screenshot of running the VCSA Beta on both VMware Fusion and Workstation:

evaulate-vsphere-beta-vcsa-on-fusion-and-workstation-1
evaulate-vsphere-beta-vcsa-on-fusion-and-workstation-2
If you wanted to take this one step further and automate the entire deployment, you can leverage the ovftool to deploy the OVA as shown with the Fusion example below:

'/Applications/VMware Fusion.app/Contents/Library/VMware OVF Tool/ovftool' --name=vmware-vcsa --acceptAllEulas --allowExtraConfig /PATH/TO/VCSA/OVA '/Users/lamw/Documents/Virtual Machines.localized'

and then append the specific configuration using either an echo or here-statement. You can also do the same on Windows leveraging either plain Windows Bat script or PowerShell.

Hopefully for those of you who only have access to Fusion or Workstation, you can now also take part in the vSphere Beta if you do not have a vSphere lab that can be used. I would also recommend checking out the vSphere Beta Community as there is a new contest that launched today for finding bugs in the latest RC release. Not only can you help improve the product through your feedback, you can also win some some $$$ in doing so!

Standalone VMRC (VM Remote Console) re-introduced in vSphere 5.5 Update 2b

The VMRC (VM Remote Console) has gone through several transitions from initially being available as a standalone Windows application to an integrated browser based plugin with the release of the vSphere Web Client. In the latest vSphere 5.5 Update 2b release, a new standalone VMRC has been re-introduced to provide an alternative way to launch a VM console. The reason for this is due to the deprecated and eventual removal of NPAPI (Netscape Plugin Application Programming Interface) based plugin support from all modern web browsers which the current VMRC implementation leverages. Here is a quick excerpt from the vSphere 5.5 Update 2b release notes:

Inability to open virtual machine console using Google Chrome browser when NPAPI support is deprecated
When the NPAPI support in Google Chrome is deprecated, the virtual machine console provided in the vSphere Client Integration Plugin might no longer function when the Chrome browser is updated. As a result, you might be unable to open the virtual machine console using the Google Chrome browser and you might not be able to connect to devices.

UPDATE (10/21) - Looks like the standalone VMRC has just been made available and you can now download it by either following the link in the vSphere Web Client if you are on vSphere 5.5 Update 2b OR simply by going to http://www.vmware.com/go/download-vmrc

UPDATE (10/12) - It looks like the standalone VMRC is currently not available for download just yet. You can continue using the existing methods to connect to your VM Console, the new Standalone VMRC is NOT required but the links have been put in place to proactively get ready for NPAPI deprecation (more details below). You can subscribe to VMware KB 2091284 which will be updated when the download is available.

The deprecation of NPAPI support is nothing new and has actually been communicated by all major web browsers for quite some time now. To ensure that VMware customers are not affected when this change goes into effect, a new standalone VMRC is being introduced to preempt the upcoming change and provides a new way of  launching a VM console using the vSphere Web Client as seen in the screenshot below.

vmrc
To be able to open a VM Console using the new standalone VMRC, you will of course need to have it installed first. You can find the link to the download on VMware.com but there is also a direct link provided on the VM Summary page in the vSphere Web Client. In addition to the new standalone VMRC, you will still be able to use the existing method as well as the HTML5 based VM console. The HTML5 console continues to work if you do not have CIP (Client Integration Package) installed on your Windows system or if you are running on a Mac OS X system. I am sure many of you are probably asking when will there be Mac OS X version of VMRC? I know I definitely am :) The good news is that this is being worked on and hopefully we will see a Mac OS X version in the very near future.

Furthermore, the new standalone VMRC also includes some nice enhancements that I know some of you have been asking for, especially those that have used the previous standalone VMRC application. The new VMRC can now be directly launched using the following two URI methods:

vmrc://[USERNAME]@[VC]/?moid=[VM-MOREF-ID]
vmrc://clone:[VC-TICKET]@[VC]/?moid=[VM-MOREF-ID]

Here is a screenshot of the standalone VMRC application:

vmrc-0
The first method accepts basic authentication using username/password, the vCenter Server address and the VM MoRef Id. Here is an example of what that would look like:

C:\Program Files (x86)\VMware\VMware Remote Console\vmrc.exe vmrc://[email protected]/?moid=vm-37

The second method accepts a vCenter Server session ticket which you can generate by using vSphere API acquireCloneTicket() method. A quick way to test this example is by using the vSphere MOB and making a call to acquireCloneTicket using the following URL https://[VCENTER-SERVER]/mob/?moid=SessionManager&method=acquireCloneTicket and then specifying the ticket as seen in the example below.

C:\Program Files (x86)\VMware\VMware Remote Console\vmrc.exe vmrc://clone:cst-VCT-5244c230-f59b-75c4-f4e9-4205d1918497--tp-DF-A9-72-7B-32-1[email protected]es.com/?moid=vm-37

With the new URI handler, you can automatically associate it with the standalone VMRC application which means you can type this into a browser or into a Windows explorer and it will automatically launch VMRC. The other nice thing about the new standalone VMRC is if you would like to reduce the complexity of getting a regular use connected to their desktop, you can easily use the standalone VMRC and dynamically generating a link for your end users to access their VMs without ever exposing them to the underlying vSphere infrastructure. I suspect there will be some really interesting use cases for the new standalone VMRC and the VMRC team will continue to iterate to make it better based on customer feedback.

Automate VSAN Observer offline mode configurations

I was recently reading two of Rawlinson Rivera's articles (here and here) on configuring the VSAN Observer to be able to run in an "offline mode". The VSAN Observer currently leverages several open source libraries including Javascript, CSS and Font files to render the UI which assumes you have direct internet access to load these library files. In a traditional Enterprise environment, direct Internet access is usually not available and thought it could be provided either through a white list of proxy addresses, but in most cases it is just blocked.

Rawlinson provided a nice writeup on the specific library files that needs to be downloaded, the directory structure that needs to be created and the modifications required for each file. Unfortunately, the process is quite manual and potentially very error prone which usually screams for some Automation :) I figure I could help my buddy Rawlinson out by creating two scripts which would download all the necessary files and the other script which will go ahead and update all the appropriate VSAN Observer files.

The first script is called download_vsan_observer_offline_files.sh which will download the necessary library files and put them in the expected directory structure externallibs{js,css,font} using cURL. This shell script is meant to run on a system which has Internet access and uses cURL to perform the download. If you do not have cURL, you can update the script to use wget instead. At the end of the script, you should see a directory created called externallibs which will need to be SCP'ed to the VCSA running the VSAN Observer (this is a prerequisite to the second script).

Here is an example of running the shell script:
automate-vsan-observer-offline-mode-0
The second script is called update_vsan_observer_offline_files.sh which runs on the VCSA that will be used for the VSAN Observer. This shell scripts expects the externallibs directory to be present before updating the VSAN Observer files and will error out if it does not detect it.

Here is an example of running the shell script:
automate-vsan-observer-offline-mode-1
At this point you are ready to run your VSAN Observer in an "offline mode" as Rawlinson has documented on his blog. Please refer his article for more details on using the VSAN Observer.

One thing I was pleasantly surprise to see in the latest vSphere 5.5 Update 2 release of the VCSA is that VSAN Observer now supports HTTPS as well as authentication when logging into the VSAN Observer UI. This is a very nice update and I recommend you download the latest release of VCSA to benefit from these new features.