Intel NUC

Exclusive vGhetto discount on homelab hardware from MITXPC

04/12/2017 by William Lam 3 Comments

On a regular basis I already receive a number of inquires from both internal VMware folks as well as external partners and customers about VMware homelabs and the type of hardware that can be used. After demo'ing our recent USB to SDDC project, the requests have literally tripled! Most folks are generally inquiring BOM details and/or where to purchase the Intel NUC or the SuperMicro E200-8D.

In particular, the SuperMicro E200-8D has probably received the most amount of interest lately. In fact, I am also interested in one after having an opportunity to play with one during the Melbourne VMUG. One thing I had noticed while talking to several colleagues who have purchased this system both locally within the Bay Area as well as overseas such as Australia was that one particular reseller kept coming up over and over again. That vendor was MITXPC which is a local bay area company located over in Fremont which specializes in Mini-ITX systems.

The reason MITXPC was being used by the majority of these folks was simple, they had the best price for the SuperMicro E200-8D which was significantly cheaper than other vendors including Amazon.

Vendor	Price
E200-8D on MITXPC	$799 USD ($783.02 w/discount code)
E200-8D on Amazon	$849 USD

Having heard good things about MITXPC, I decided to reach out to them and see if there was anything special they could do for the VMware Community. I was able to get a special discount code that would offer folks an additional 2% off their entire purchase at MITXPC. For those of you who have been holding off on a refresh your home lab or itching to build your own, this is a great time! If you would like to take advantage of this offer, simply use the discount code VIRTUALLYGHETTO2OFF when you check out. I would like to give a huge thanks to Eric Yui of MITXPC for working with me on this and helping out the VMware Community.

Disclaimer: I am not affiliated with MITXPC.

Update on Intel NUC 7th Gen (Kaby Lake) & ESXi 6.x

02/15/2017 by William Lam 27 Comments

Intel just started shipping their i3 7th Gen (Kaby Lake) Intel NUCs and no surprise, Florian of virten.net has already gotten his hands on a unit for testing. The Intel NUCs is a very popular platform for running vSphere/vSAN-based home labs, especially for their price point and footprint. Last week, Florian discovered from his testing that the built-in network adapter on the 7th Gen NUC was not being detected by any of the ESXi installers and had published his findings here.

Since the Intel NUC is not an officially supported platform for ESXi, it does not surprise me that these sort of things happen, even if the NUC had a pretty good track record going back to the 5th Gen releases. Nonetheless, I reached out to Florian to see if he can provide me with a vm-support bundle and see if there was anything I could do to help.

A couple of Engineers took a look and quickly identified the issue with the Kabylake NIC (8086:15d8) but before getting to the solution, I did want to clarify something. The Kabylake NIC is actually NOT an officially supported NIC for ESXi and although it currently shows up in the VMware HCL, it is a mistake. I have been told the VMware HCL will be updated shortly to reflect this, apologies for any confusion that this may have caused.

Ok, so the great news is that we do have a solution for getting ESXi to recognize the built-in NIC on the 7th Gen Intel NUCs. The semi-bad news is that we currently do not have a solution in the short term for any released version of ESXi, as the fix will require an updated version of the e1000e Native Driver which will only be available in a future update of ESXi. I can not provide any timelines, but keep an eye on this blog and I will publish more details once they are available.

In the meantime, if you already own a 7th Gen NUC, there is a workaround which Florian has already blogged about here which uses the USB Ethernet Adapter VIB for the initial ESXi installation. If you are planning to purchase the 7th Gen NUC and would like to wait for folks to confirm the fix, then I would recommend holding off or potentially looking at the 6th Gen if you can not wait. Thanks to Florian and others who shared their experiences with the 7th Gen NUC and also to the VMware Engineers who found a quick resolution to the problem.

UPDATE (07/27/17) - The updated e1000e Native Driver that was included in ESXi 6.0 Update 3 is now included in ESXi 6.5 Update 1 which just GA'ed. You should be able to install ESXi without require any additional modifications to the latest Intel NUCs.

UPDATE (02/24/17) - An updated e1000e driver which contains a fix for 7th Gen NUC is now available as part of ESXi 6.0 Update 3, below are several options in how you can consume the driver.

ESXi on the new Intel NUC Skull Canyon

05/21/2016 by William Lam 59 Comments

Earlier this week I found out the new Intel NUC "Skull Canyon" (NUC6i7KYK) has been released and have been shipping for a couple of weeks now. Although this platform is mainly targeted at gaming enthusiast, there have also been a lot of anticipation from the VMware community on leveraging the NUC for a vSphere based home lab. Similiar to the 6th Gen Intel NUC system which is a great platform to run vSphere as well as VSAN, the new NUC includes a several new enhancements beyond the new aesthetics. In addition to the Core i7 CPU, it also includes a dual M.2 slots (no SATA support), Thunderbolt 3 and most importantly, ~~an Intel Iris Pro GPU~~ a Thunderbolt 3 Controller. I will get to why this is important ...

UPDATE (05/26/16) - With some further investigation from folks like Erik and Florian, it turns out the *only* device that needs to be disabled for ESXi to successfully boot and install is the Thunderbolt Controller. Once ESXi has been installed, you can re-enable the Thunderbolt Controller and Florian has also written a nice blog post here which has instructions as well as screenshots for those not familiar with the Intel NUC BIOs.

UPDATE (05/23/16) - Shortly after sharing this article internally, Jason Joy, a VMware employee shared the great news that he has figured out how to get ESXi to properly boot and install. Jason found that by disabling unnecessary hardware devices like the Consumer IR/etc in the BIOS, it allowed the ESXi installer to properly boot up. Jason was going to dig a bit further to see if he can identify the minimal list of devices that needed to be disabled to boot ESXi. In the meantime, community blogger Erik Bussink has shared the list of settings he has applied to his Skull Canyon to successfully boot and install latest ESXi 6.0 Update 2 based on the feedback from Jason. Huge thanks to Jason for quickly identifying the workaround and sharing it with the VMware community and thanks to Erik for publishing his list. For all those that were considering the new Intel NUC Skull Canyon for a vSphere-based home lab, you can now get your ordering on! 😀

Below is an except from his blog post Intel NUC Skull Canyon (NUC6I7KYK) and ESXi 6.0 on the settings he has disabled:

BIOS\Devices\USB

disabled - USB Legacy (Default: On)
disabled - Portable Device Charging Mode (Default: Charging Only)
not change - USB Ports (Port 01-08 enabled)

BIOS\Devices\SATA

disabled - Chipset SATA (Default AHCI & SMART Enabled)
M.2 Slot 1 NVMe SSD: Samsung MZVPV256HDGL-00000
M.2 Slot 2 NVMe SSD: Samsung MZVPV512HDGL-00000
disabled - HDD Activity LED (Default: On)
disabled - M.2 PCIe SSD LEG (Default: On)

BIOS\Devices\Video

IGD Minimum Memory - 64GB (Default)
IGD Aperture Size - 256 (Default)
IGD Primary Video Port - Auto (Default)

BIOS\Devices\Onboard Devices

disabled - Audio (Default: On)
LAN (Default)
disabled - Thunderbolt Controller (Default is Enabled)
disabled - WLAN (Default: On)
disabled - Bluetooth (Default: On)
Near Field Communication - Disabled (Default is Disabled)
SD Card - Read/Write (Default was Read)
Legacy Device Configuration
disabled - Enhanced Consumer IR (Default: On)
disabled - High Precision Event Timers (Default: On)
disabled - Num Lock (Default: On)

BIOS\PCI

M.2 Slot 1 - Enabled
M.2 Slot 2 - Enabled
M.2 Slot 1 NVMe SSD: Samsung MZVPV256HDGL-00000
M.2 Slot 2 NVMe SSD: Samsung MZVPV512HDGL-00000

Cooling

CPU Fan HEader
Fan Control Mode : Cool (I toyed with Full fan, but it does make a lot of noise)

Performance\Processor

disabled Real-Time Performance Tuning (Default: On)

Power

Select Max Performance Enabled (Default: Balanced Enabled)
Secondary Power Settings
disabled - Intel Ready Mode Technology (Default: On)
disabled - Power Sense (Default: On)
After Power Failure: Power On (Default was stay off)

Over the weekend, I had received several emails from folks including Olli from the nucblog.net (highly recommend a follow if you do not), Florian from virten.net (another awesome blog which I follow & recommend) and few others who have gotten their hands on the "Skull Canyon" system. They had all tried to install the latest release of ESXi 6.0 Update 2 including earlier versions but all ran into a problem while booting up the ESXi installer.

~~The following error message was encountered:~~

Error loading /tools.t00
Compressed MD5: 39916ab4eb3b835daec309b235fcbc3b
Decompressed MD5: 000000000000000000000000000000
Fatal error: 10 (Out of resources)

Raymond Huh was the first individual who had reach out to me regarding this issue and then shortly after, I started to get the same confirmations from others as well. Raymond's suspicion was that this was related to the amount of Memory-Mapped I/O resources being consumed by the Intel Iris Pro GPU and does not leave enough resources for the ESXi installer to boot up. Even a quick Google search on this particular error message leads to several solutions here and here where the recommendation was to either disable or reduce the amount of memory for MMIO within the system BIOS.

Unfortunately, it does not look like the Intel NUC BIOS provides any options of disabling or modifying the MMIO settings after Raymond had looked which including tweaking some of the video settings. He currently has a support case filed with Intel to see if there is another option. In the mean time, I had also reached out to some folks internally to see if they had any thoughts and they too came to the same conclusion that without being able to modify or disable MMIO, there is not much more that can be done. There may be a chance that I might be able to get access to a unit from another VMware employee and perhaps we can see if there is any workaround from our side, but there are no guarantees, especially as this is not an officially supported platform for ESXi. I want to thank Raymond, Olli & Florian for going through the early testing and sharing their findings thus far. I know many folks are anxiously waiting and I know they really appreciate it!

For now, if you are considering purchasing or have purchased the latest Intel NUC Skull Canyon with the intention to run ESXi, I would recommend holding off or not opening up the system. I will provide any new updates as they become available. I am still hopeful that we will find a solution for the VMware community, so crossing fingers.

Adding custom VSAN BIOS splash screen to the Intel NUC

03/06/2016 by William Lam 5 Comments

One of the last things I wanted to look into after setting up my new VSAN 6.2 home lab on the new 6th Gen Intel NUC was to add a custom BIOS splash screen giving my system a personal touch. Updating the BIOS splash screen would require flashing the BIOS itself which gave me some concerns after hearing about the BIOS v33 issue in which the M.2 slot would no longer be detected after the update. Although there was a simple workaround after the update, I still wanted to be cautious. Over the weekend I had noticed that Intel had released BIOS v36 for the Intel NUC which resolved the M.2 issue among a few others. I decided to give it a shot and hope that I that I do not brick my NUC.

I am happy to say that I was successful in updating to the latest Intel NUC BIOS and as you can see from the screenshot below, I was also able to replace the default Intel BIOS splash screen with a Captain VSAN BIOS splash screen (TV is 46" for those wondering) 🙂

The process for building and customizing your Intel NUC BIOS is relatively straight forward but because I waited until after I had everything installed, it ended up being a bit more work than I had hoped. To customize your BIOS, Intel provides a Microsoft Windows only utility called Intel Integrator Toolkit. The easiest way to build and update your BIOS is to initially start off by installing Microsoft Windows on the Intel NUC itself which then allows you to easily flash the BIOS using the executable that is generated from the toolkit. Since I had already consumed both of my SSDs for VMware VSAN and Microsoft Windows does not allow you to install its OS directly onto a USB device, I had to use this method here to install a bootable version of Microsoft Windows onto the USB device since I did not want to blow away my VSAN setup.

OK, so now onto the cool stuff. Below are the instructions on how to build and customize your BIOS for the Intel NUC. If you would like to use the exact same BIOS splash screen as well as update to the latest BIOS v36 and do not want to go through the hassle, I have made my custom VSAN BIOS image available here. You just need to download the executable and run it on the Intel NUC itself which must be running Microsoft Windows (I used 8.1) and then follow the screens on flashing your BIOS.

Step 1 - Download the following two packages and transfer them to Microsoft Windows image running on your NUC:

Step 2 - Install the Intel Integrator Toolkit and then start the program

Step 3 - Select the "Customize a BIOS file" option and load either the custom VSAN BIOS image which I have made available here OR load the NUC BIOS v36 file you had downloaded earlier.

Step 4 - In the lower left hand corner, browse for the graphic image that you wish to use for your BIOS splash screen (images with black background works the best). For those interested, you can find the Captain VSAN image that I had used here. The tool actually supports several image formats in addition to the default BMP such as JPEG and PNG, you just need to change the extension type. There is a size limitation, but the nice thing about the tool is that there is an option to compress the image when it detects it is too large. Make sure to change the image for the four different options by clicking on the drop down wizard. I thought I only had to replace the first image but it looks like other versions of the splash screen is also used and it is best to just replace them all. You also have the option of changing other default settings in the BIOS, feel free to click on the tooltip for details on each of the options.

Step 5 - Once you are done customizing your BIOS, you will then save your changes and the tool will produce a single Windows executable (SY0036.exe) which you will run on the NUC itself to flash the BIOS. You will be prompted with a couple of questions and once the process begins, it will restart and you will need to confirm one more time before the imaging process starts. If everything was successful, you should now see a new BIOS splash screen replacing the default Intel image. There is a good chance you may go through this process a few times depending if you are happy with the splash screen display. I think it took me about three tries. Hope this helps anyone looking to add that personal touch to their home lab!

VSAN 6.2 (vSphere 6.0 Update 2) homelab on 6th Gen Intel NUC

03/03/2016 by William Lam 33 Comments

As many of you know, I have been happily using an Apple Mac Mini for my personal vSphere home lab for the past few years now. I absolutely love the simplicity and the versatility of the platform to easily run a basic vSphere lab to being able to consume advanced capabilities of the vSphere platform like VMware VSAN or NSX for example. The Mac Mini's also supports more complex networking configurations by allowing you to add an additional network adapter which leverages the built-in Thunderbolt adapter which many other similar form factors lack. Having said that all that, one major limitation with the Mac Mini platform has always been the limited amount of memory it can support which is a maximum of 16GB (same limitation as other form factors in this space). Although it is definitely possible to run a vSphere lab with only 16GB of memory, it does limit you some what on what you can deploy which is challenging if you want to explore other solutions like VSAN, NSX and vRealize.

I was really hoping that Apple would have released an update to their Mac Mini platform last year that would include support for 32GB of memory, but instead it was a very minor update and was mostly a let down which you can read more about here. Earlier this year, I found out from fellow blogger Florian Grehl that Intel has just released their 6th generation of the Intel NUC which officially adds support for 32GB of memory. I have been keeping an eye on the Intel NUC for some time now but due to the same memory limitation as the Mac Mini, I had never considered it as viable option, especially given that I own a Mac Mini already. With the added support for 32GB of memory and the ability to house two disk drives (M.2 and 2.5"), this was the update I was finally waiting for to pull the trigger and refresh my home lab given 16GB was just not cutting it for the work I was doing anymore.

There have been quite a bit of interest in what I ended up purchasing for running VSAN 6.2 (vSphere 6.0 Update 2) which has not GA'ed ... yet and so I figure I would together a post with all the details in case others were looking to build a similar lab. This article is broken down into the following sections:

Bill of Materials (BOM)
Installation
VSAN Configuration
Final Word

Disclaimer: The Intel NUC is not on VMware's official Hardware Compatibility List (HCL) and there for is not officially supported by VMware. Please use this platform at your own risk.

Bill of Materials (BOM)

Below are the components with links that I used for my configuration which is partially based on budget as well as recommendations from others who have a similar setup. If you think you will need more CPU horsepower, you can look at the Core i5 (NUC6i5SYH) model which is slightly more expensive than the i3. I opted for an all-flash configuration because I not only wanted the performance but I also wanted to take advantage of the much anticipated Deduplication and Compression feature in VSAN 6.2 which is only supported with an all-flash VSAN setup. I also did not have a need for large amount of storage capacity, but you could also pay a tiny bit more for the exact same drive giving you a full 1TB if needed. If you do not care for an all-flash setup, you can definitely look at spinning rust which can give you several TB's of storage at a very reasonable cost. The overall cost of the system for me was ~$700 USD (before taxes) and that was because some of the components were slightly discounted through the use of a preferred retailer that my employer provided. I would highly recommend you check with your employer to see if you have similiar HR benefits as that can help with the cost if that is important to you. The SSDs actually ended up being cheaper on Amazon and so I ended up purchasing them there.

1 x Intel NUC 6th Gen NUC6i3SYH (supports 2 drives: M.2 & 2.5)
2 x Crucial 16GB DDR4
~~1 x Samsung 850 EVO 250GB M.2 for “Caching” Tier~~ (Thanks to my readers, decided to upgrade to 1 x Samsung SM951 NVMe 128GB M.2 for "Caching" Tier)
1 x Samsung 850 EVO 500GB 2.5 SATA3 for “Capacity” Tier

Installation

The installation of the memory and the SSDs on NUC was super simple. You just need a regular phillips screwdriver and there were four screws at the bottom of the NUC that you will need to unscrew. Once loosen, you just need to flip the NUC unit back on top while holding the bottom and slowly taking the top off. The M.2 SSD requires a smaller phillips screwdriver which you will need to unscrew before you can plug in the device. The memory just plugs right in and you should hear a click to confirm its inserted all the way. The 2.5" SSD just plugs into the drive bay which is attached to the top of the NUC casing. If you are interested in more details, you can find various unboxing and installation videos online like this one.

UPDATE (05/25/16): Intel has just released BIOS v44 which fully enables unleashes the power of your NVMe devices. One thing to note from the article is that you do NOT need to unplug the security device, you can just update BIOS by simply download the BIOS file and loading it onto a USB key (FAT32).

UPDATE (03/06/16): Intel has just released BIOS v36 which resolves the M.2 SSD issue. If you have updated using earlier versions, to resolve the problem you just need to go into the BIOS and re-enable the M.2 device as mentioned in this blog here.

One very important thing to note which I was warned about by a fellow user was NOT to update/flash to a newer version of the BIOS. It turns out that if you do, the M.2 SSD will fail to be detected by the system which sounds like a serious bug if you ask me. The stock BIOS version that came with my Intel NUC is SYSKLi35.86A.0024.2015.1027.2142 in case anyone is interested. I am not sure if you can flash back the original version but another user just informed me that they had accidentally updated the BIOS and now he can no longer see the M.2 device 🙁

For the ESXi installation, I just used a regular USB key that I had lying around and used the unetbootin tool to create a bootable USB key. I am using the upcoming ESXi 6.0 Update 2 (which has not been released ... yet) and you will be able to use the out of the box ISO that is shipped from VMware. There are no additional custom drivers that are required. Once the ESXi installation loads up, you can then install ESXi back onto the same ESXi USB key which it initially boot it up. I know this is not always common knowledge and as some may think you need an additional USB device to install ESXi. Ensure you do not install anything on the two SSDs if you plan to use VSAN as it requires at least (2 x SSD) or (1 x SSD and 1 x MD).

If you are interested in adding a bit of personalization to your Intel NUC setup and replace the default Intel BIOS splash screen like I have, take a look at this article here for more details.

If you are interested in adding additional network adapters to your Intel NUC via USB Ethernet Adapter, have a look at this article here.

VSAN Configuration

Bootstrapping VSAN Datastore:

If you plan to run VSAN on the NUC and you do not have additional external storage to deploy and setup things like vCenter Server, you have the option to "bootstrap" VSAN using a single ESXi node to start with which I have written in more detail here and here. This option allows you to setup VSAN so that you can deploy vCenter Server and then help you configure the remainder nodes of your VSAN cluster which will require at least 3 nodes unless you plan on doing a 2-Node VSAN Cluster with the VSAN Witness Appliance. For more detailed instructions on bootstrapping an all-flash VSAN datastore, please take a look at my blog article here.
If you plan to *ONLY* run a single VSAN Node which is possible but NOT recommended given you need a minimum of 3 nodes for VSAN to properly function. After the vCenter Server is deployed, you will need to update the default VSAN VM Storage Policy to ether allow "Forced Provisioning" or changing the FTT from 1 to 0 (e.g. no protection given you only have a single node). This will be required else you will run into provisioning issues as VSAN will prevent you from deploying VMs as it is expecting two additional VSAN nodes. When logged into the home page of the vSphere Web Client, click on "VM Storage Policies" icon and edit the "Virtual SAN Default Storage Policy" and change the following values as show in the screenshot below:

Installing vCenter Server:

If you are new to deploying the vCenter Server, VMware has a deployment guide which you can follow here.

Optimizations:

In addition, because this is for a home lab, my buddy Cormac Hogan has a great tip on disabling device monitoring as the SSD devices may not be on the VMware's official HCL and can potentially negatively impact your lab environment. The following ESXCLI command needs to be run once on each of the ESXi hosts in the ESXi Shell or remotely:

esxcli system settings advanced set -o /LSOM/VSANDeviceMonitoring -i 0

I also recently learned from reading Cormac's blog that there is also new ESXi Advanced Setting in VSAN 6.2 which allows VSAN to provision a VM swap object as "thin" versus "thick" which has been the historically default. To disable the use of "thick" provisioning, you will need to run the following ESXCLI command on each ESXi host:

esxcli system settings advanced set -o /VSAN/SwapThickProvisionDisabled -i 1

Lastly, if you plan to run Nested ESXi VMs on top of your physical VSAN Cluster, be sure to add this configuration change outlined in this article here, else you may see some strangeness when trying to create VMFS volumes.

Final Word

I have only had the NUC for a couple of days but so far I have been pretty impressed with the ease of setup and the super tiny form factor. I thought the Mac Mini's were small and portable, but the NUC really blows it out of the water. I was super happy with the decision to go with an all-flash setup, the deployment of the VCSA was super fast as you would expect. If I compare this to my Mac Mini which had spinning rust, for a portion of the VCSA deployment, the fan would go a bit psycho and you can feel the heat if you put your face close to it. I could barely feel any heat from the NUC and it was dead silent which is great as it sits in our living room. Like the Mac Mini, the NUC has regular HDMI port which is great as I can connect it directly to our TV and has plenty of USB ports which could come in handy if you wanted to play with VSAN using USB-based disks 😉

One neat idea that Duncan Epping had brought up in a recent chat with him was to run a 2-Node VSAN Cluster and have the VSAN Witness appliance running on a desktop or laptop. This would make for a very simple and affordable VSAN home lab without requiring a 3rd physical ESXi node. I had also thought about doing the same but instead of 2 NUCs, I would be combining my Mac Mini and NUC to form the 2-Node VSAN Cluster and then run the VSAN Witness on my iMac desktop which has 16GB of memory. This is just another slick way you can leverage this new and powerful platform to run a full blow VSAN setup. For those of you following my blog, I am also looking to see if there is a way to add a secondary network adapter to the NUC by the way of a USB 3.0 based ethernet adapter. I have already shown that it is definitely possible with older releases of ESXi and if this works, could make the NUC even more viable.

Lastly, for those looking for a more beefier setup, there are rumors that Intel maybe close to releasing another update to the Intel NUC platform code named "Skull Canyon" which could include a Quad-Core i7 (Broadwell based) along with supporting the new USB-c interface which would be able to run Thunderbolt 3. If true, this could be another option for those looking for a bit more power for their home lab.

A few folks had been asking what I plan to do with my Mac Mini now that I have my NUC. I probably will be selling it, it is still a great platform and has Core i7 which definitely helps with any CPU intensive tasks. It also supports two drives, so it is quite inexpensive to purchase another SSD as it already comes with one to setup an all-flash VSAN 6.2 setup. Below are the the specs and If you interested in the setup, feel free to drop me an email at info.virtuallyghetto [at] gmail [dot] com.

Mac Mini 5,3 (Late 2011)
Quad-Core i7 (262635QM)
16GB memory
1 x SSD (120GB) Corsair Force GT
1 x MD (750 GB) Seagate Momentus XT
1 x built-in 1Gbe Ethernet port
1 x Thunderbolt port
4 x USB ports
1 x HDMI
Original packaging available
VSAN capable
ESXi will install OOTB w/o any issues

Additional Useful Resources: