virtuallyGhetto

Not Supported

vSphere 6.0 Update 2 hints at Nested ESXi support for Paravirtual SCSI (PVSCSI) in the future

by 6 Comments

Although Nested ESXi (running ESXi in a Virtual Machine) is not officially supported today, VMware Engineering continues to enhance this widely used feature by making it faster, more reliable and easier to consume for our customers. I still remember that it was not too long ago that if you wanted to run Nested ESXi, several non-trivial and manual tweaks to the VM's VMX file were required. This made the process of consuming Nested ESXi potentially very error prone and provide a less than ideal user experience.

Things have definitely been improved since the early days and here are just some of the visible improvements over the last few years:

  • Prior to vSphere 5.1, enabling Virtual Hardware Assisted Virtualization (VHV) required manual edits to the VMX file and even earlier versions required several VMX entries. VHV can now be easily enabled using either the vSphere Web Client or the vSphere API.
  • Prior to vSphere 5.1, only the e1000{e} networking driver was supported with Nested ESXi VMs and although it was functional, it also limited the types of use cases you might have for Nested ESXi. A Native Driver for VMXNET3 was added in vSphere 5.1 which not only increased the performance that came with using the optimized VMXNET3 driver but it also enabled new use cases such testing SMP-FT as it was now possible to get 10Gbe interface to Nested ESXi VM versus the traditional 1GBe with e1000{e} driver.
  • Prior to vSphere 6.0, selection of ESXi GuestOS was not available in the "Create VM" wizard which meant you had to resort to re-editing the VM after initial creation or using the vSphere API. You can now select the specific ESXi GuestOS type directly in the vSphere Web/C# Client.
  • Prior to vSphere 6.0, the only way to cleanly shutdown or power cycle a Nested ESXi VM was to perform the operation from within the system as there was no VMware Tools support. This changed with the development of a VMware Tools daemon specifically for Nested ESXi which started out as a VMware Fling. With vSphere 6.0, the VMware Tools for Nested ESXi was pre-installed by default and would automatically startup when it detected that it ran as a VM. In addition to power operations provided by VMware Tools, it also enabled the use of the Guest Operations API which was quite popular from an Automation standpoint.

Yesterday while working in my new vSphere 6.0 Update 2 home lab, I needed to create a new Nested ESXi VM and noticed something really interesting. I used the vSphere Web Client like I normally would and when I went to select the GuestOS type, I discovered an interesting new option which you can see from the screenshot below.

nested-esxi-changes-in-vsphere60u2-3
It is not uncommon to see VMware to add experimental support for potentially new Guest Operating Systems in vSphere. Of course, there are no guarantees that these OSes would ever be supported or even released for that matter.

What I found that was even more interesting was that when select this new ESXi GuestOS type (vmkernel65) is what was recommended as the default virtual hardware configuration for the VM. For the network adapter, it looks like the VMXNET3 driver is now recommended over the e1000e and for the storage adapter the VMware Paravirtual (PVSCSI) adapter is now recommended over the LSI Logic Parallel type. This is really interesting as it is currently not possible today to get the optimized and low overhead of the PVSCSI adapter working with Nested ESXi and this seems to indicate that PVSCSI might actually be possible in the future! 🙂

nested-esxi-changes-in-vsphere60u2-1
I of course tried to install the latest ESXi 6.0 Update 2 (not yet GA'ed) using this new ESXi GuestOS type and to no surprise, the ESXi installer was not able to detect any storage devices. I guess for now, we will just have to wait and see ...

VSAN 6.2 (vSphere 6.0 Update 2) homelab on 6th Gen Intel NUC

by 33 Comments

As many of you know, I have been happily using an Apple Mac Mini for my personal vSphere home lab for the past few years now. I absolutely love the simplicity and the versatility of the platform to easily run a basic vSphere lab to being able to consume advanced capabilities of the vSphere platform like VMware VSAN or NSX for example. The Mac Mini's also supports more complex networking configurations by allowing you to add an additional network adapter which leverages the built-in Thunderbolt adapter which many other similar form factors lack. Having said that all that, one major limitation with the Mac Mini platform has always been the limited amount of memory it can support which is a maximum of 16GB (same limitation as other form factors in this space). Although it is definitely possible to run a vSphere lab with only 16GB of memory, it does limit you some what on what you can deploy which is challenging if you want to explore other solutions like VSAN, NSX and vRealize.

I was really hoping that Apple would have released an update to their Mac Mini platform last year that would include support for 32GB of memory, but instead it was a very minor update and was mostly a let down which you can read more about here. Earlier this year, I found out from fellow blogger Florian Grehl that Intel has just released their 6th generation of the Intel NUC which officially adds support for 32GB of memory. I have been keeping an eye on the Intel NUC for some time now but due to the same memory limitation as the Mac Mini, I had never considered it as viable option, especially given that I own a Mac Mini already. With the added support for 32GB of memory and the ability to house two disk drives (M.2 and 2.5"), this was the update I was finally waiting for to pull the trigger and refresh my home lab given 16GB was just not cutting it for the work I was doing anymore.

There have been quite a bit of interest in what I ended up purchasing for running VSAN 6.2 (vSphere 6.0 Update 2) which has not GA'ed ... yet and so I figure I would together a post with all the details in case others were looking to build a similar lab. This article is broken down into the following sections:

  • Bill of Materials (BOM)
  • Installation
  • VSAN Configuration
  • Final Word

Disclaimer: The Intel NUC is not on VMware's official Hardware Compatibility List (HCL) and there for is not officially supported by VMware. Please use this platform at your own risk.

Bill of Materials (BOM)

vsan62-intel-nuc-bom
Below are the components with links that I used for my configuration which is partially based on budget as well as recommendations from others who have a similar setup. If you think you will need more CPU horsepower, you can look at the Core i5 (NUC6i5SYH) model which is slightly more expensive than the i3. I opted for an all-flash configuration because I not only wanted the performance but I also wanted to take advantage of the much anticipated Deduplication and Compression feature in VSAN 6.2 which is only supported with an all-flash VSAN setup. I also did not have a need for large amount of storage capacity, but you could also pay a tiny bit more for the exact same drive giving you a full 1TB if needed. If you do not care for an all-flash setup, you can definitely look at spinning rust which can give you several TB's of storage at a very reasonable cost. The overall cost of the system for me was ~$700 USD (before taxes) and that was because some of the components were slightly discounted through the use of a preferred retailer that my employer provided. I would highly recommend you check with your employer to see if you have similiar HR benefits as that can help with the cost if that is important to you. The SSDs actually ended up being cheaper on Amazon and so I ended up purchasing them there. 

Installation

vsan62-intel-nuc-1
The installation of the memory and the SSDs on NUC was super simple. You just need a regular phillips screwdriver and there were four screws at the bottom of the NUC that you will need to unscrew. Once loosen, you just need to flip the NUC unit back on top while holding the bottom and slowly taking the top off. The M.2 SSD requires a smaller phillips screwdriver which you will need to unscrew before you can plug in the device. The memory just plugs right in and you should hear a click to confirm its inserted all the way. The 2.5" SSD just plugs into the drive bay which is attached to the top of the NUC casing. If you are interested in more details, you can find various unboxing and installation videos online like this one. 

UPDATE (05/25/16): Intel has just released BIOS v44 which fully enables unleashes the power of your NVMe devices. One thing to note from the article is that you do NOT need to unplug the security device, you can just update BIOS by simply download the BIOS file and loading it onto a USB key (FAT32).

UPDATE (03/06/16): Intel has just released BIOS v36 which resolves the M.2 SSD issue. If you have updated using earlier versions, to resolve the problem you just need to go into the BIOS and re-enable the M.2 device as mentioned in this blog here.

One very important thing to note which I was warned about by a fellow user was NOT to update/flash to a newer version of the BIOS. It turns out that if you do, the M.2 SSD will fail to be detected by the system which sounds like a serious bug if you ask me. The stock BIOS version that came with my Intel NUC is SYSKLi35.86A.0024.2015.1027.2142 in case anyone is interested. I am not sure if you can flash back the original version but another user just informed me that they had accidentally updated the BIOS and now he can no longer see the M.2 device 🙁

For the ESXi installation, I just used a regular USB key that I had lying around and used the unetbootin tool to create a bootable USB key. I am using the upcoming ESXi 6.0 Update 2 (which has not been released ... yet) and you will be able to use the out of the box ISO that is shipped from VMware. There are no additional custom drivers that are required. Once the ESXi installation loads up, you can then install ESXi back onto the same ESXi USB key which it initially boot it up. I know this is not always common knowledge and as some may think you need an additional USB device to install ESXi. Ensure you do not install anything on the two SSDs if you plan to use VSAN as it requires at least (2 x SSD) or (1 x SSD and 1 x MD).

vsan62-intel-nuc-3
If you are interested in adding a bit of personalization to your Intel NUC setup and replace the default Intel BIOS splash screen like I have, take a look at this article here for more details.

custom-vsan-bios-splash-screen-for-intel-nuc-0
If you are interested in adding additional network adapters to your Intel NUC via USB Ethernet Adapter, have a look at this article here.

VSAN Configuration

Bootstrapping VSAN Datastore:

  • If you plan to run VSAN on the NUC and you do not have additional external storage to deploy and setup things like vCenter Server, you have the option to "bootstrap" VSAN using a single ESXi node to start with which I have written in more detail here and here. This option allows you to setup VSAN so that you can deploy vCenter Server and then help you configure the remainder nodes of your VSAN cluster which will require at least 3 nodes unless you plan on doing a 2-Node VSAN Cluster with the VSAN Witness Appliance. For more detailed instructions on bootstrapping an all-flash VSAN datastore, please take a look at my blog article here.
  • If you plan to *ONLY* run a single VSAN Node which is possible but NOT recommended given you need a minimum of 3 nodes for VSAN to properly function. After the vCenter Server is deployed, you will need to update the default VSAN VM Storage Policy to ether allow "Forced Provisioning" or changing the FTT from 1 to 0 (e.g. no protection given you only have a single node). This will be required else you will run into provisioning issues as VSAN will prevent you from deploying VMs as it is expecting two additional VSAN nodes. When logged into the home page of the vSphere Web Client, click on "VM Storage Policies" icon and edit the "Virtual SAN Default Storage Policy" and change the following values as show in the screenshot below:

Screen Shot 2016-03-03 at 6.08.16 AM

Installing vCenter Server:

  • If you are new to deploying the vCenter Server, VMware has a deployment guide which you can follow here.

Optimizations:

  • In addition, because this is for a home lab, my buddy Cormac Hogan has a great tip on disabling device monitoring as the SSD devices may not be on the VMware's official HCL and can potentially negatively impact your lab environment. The following ESXCLI command needs to be run once on each of the ESXi hosts in the ESXi Shell or remotely:

esxcli system settings advanced set -o /LSOM/VSANDeviceMonitoring -i 0

  • I also recently learned from reading Cormac's blog that there is also new ESXi Advanced Setting in VSAN 6.2 which allows VSAN to provision a VM swap object as "thin" versus "thick" which has been the historically default. To disable the use of "thick" provisioning, you will need to run the following ESXCLI command on each ESXi host:

esxcli system settings advanced set -o /VSAN/SwapThickProvisionDisabled -i 1

  • Lastly, if you plan to run Nested ESXi VMs on top of your physical VSAN Cluster, be sure to add this configuration change outlined in this article here, else you may see some strangeness when trying to create VMFS volumes.

vsan62-intel-nuc-2

Final Word

I have only had the NUC for a couple of days but so far I have been pretty impressed with the ease of setup and the super tiny form factor. I thought the Mac Mini's were small and portable, but the NUC really blows it out of the water. I was super happy with the decision to go with an all-flash setup, the deployment of the VCSA was super fast as you would expect. If I compare this to my Mac Mini which had spinning rust, for a portion of the VCSA deployment, the fan would go a bit psycho and you can feel the heat if you put your face close to it. I could barely feel any heat from the NUC and it was dead silent which is great as it sits in our living room. Like the Mac Mini, the NUC has regular HDMI port which is great as I can connect it directly to our TV and has plenty of USB ports which could come in handy if you wanted to play with VSAN using USB-based disks 😉

vsan62-intel-nuc-4
One neat idea that Duncan Epping had brought up in a recent chat with him was to run a 2-Node VSAN Cluster and have the VSAN Witness appliance running on a desktop or laptop. This would make for a very simple and affordable VSAN home lab without requiring a 3rd physical ESXi node. I had also thought about doing the same but instead of 2 NUCs, I would be combining my Mac Mini and NUC to form the 2-Node VSAN Cluster and then run the VSAN Witness on my iMac desktop which has 16GB of memory. This is just another slick way you can leverage this new and powerful platform to run a full blow VSAN setup. For those of you following my blog, I am also looking to see if there is a way to add a secondary network adapter to the NUC by the way of a USB 3.0 based ethernet adapter. I have already shown that it is definitely possible with older releases of ESXi and if this works, could make the NUC even more viable.

Lastly, for those looking for a more beefier setup, there are rumors that Intel maybe close to releasing another update to the Intel NUC platform code named "Skull Canyon" which could include a Quad-Core i7 (Broadwell based) along with supporting the new USB-c interface which would be able to run Thunderbolt 3. If true, this could be another option for those looking for a bit more power for their home lab.

A few folks had been asking what I plan to do with my Mac Mini now that I have my NUC. I probably will be selling it, it is still a great platform and has Core i7 which definitely helps with any CPU intensive tasks. It also supports two drives, so it is quite inexpensive to purchase another SSD as it already comes with one to setup an all-flash VSAN 6.2 setup. Below are the the specs and If you interested in the setup, feel free to drop me an email at info.virtuallyghetto [at] gmail [dot] com.

  • Mac Mini 5,3 (Late 2011)
  • Quad-Core i7 (262635QM)
  • 16GB memory
  • 1 x SSD (120GB) Corsair Force GT
  • 1 x MD (750 GB) Seagate Momentus XT
  • 1 x built-in 1Gbe Ethernet port
  • 1 x Thunderbolt port
  • 4 x USB ports
  • 1 x HDMI
  • Original packaging available
  • VSAN capable
  • ESXi will install OOTB w/o any issues

Additional Useful Resources:

Working USB Ethernet Adapter (NIC) for ESXi

by

As part of upgrading my personal vSphere home lab from an Apple Mac Mini to an Intel NUC (more on this in a future blog), I have been researching to see if there are other alternatives for adding an additional network adapter. The Intel NUC only includes a single built-in ethernet adapter which is similar to the Mac Mini. However, the NUC also lacks additional IO connectors like a Thunderbolt port which the Mac Mini includes and can support a Thunderbolt to Ethernet adapter. I think this is probably the only downside to the Intel NUC platform which has been similar feedback that I have seen from other vSphere home labbers who currently use or would like to use the NUC. Perhaps, with the next update of the NUC platform code named "Skull Canyon", the rumored Thunderbolt 3 / USB-c connector may make things easier as some of the existing vendors who produce Thunderbolt to ethernet adapter also use common drivers like the Broadcom tg3 which have historically worked with ESXi.

One option that has been suggested by many folks over the years was to see if a USB based ethernet adapter could be used to provide additional networking to ESXi? Historically, the answer had been no because there were no known device drivers that would work with ESXi. I had even looked into this a few years ago and although I ran into some folks who seemed to have made it work, I was never able to find the right USB ethernet adapter to personally confirm myself. It was only until last week, I decided to start fresh again and after a bit of Googling I came across an old VMTN thread here where VMTN user AK_____28 mentioned he had success with the StarTech USB 3.0 to Gigabit Ethernet NIC Network Adapter and using a custom compiled driver that was posted over here by another user named Trickstarter.

UPDATE (02/12/19) - A new VMware Native Driver for USB-based NICs has just been released for ESXi 6.5/6.7, please use this driver going forward. If you are still on ESXi 5.5/6.0, you can continue using the existing driver but please note there will be no additional development in the existing vmklinux-based driver.

UPDATE (03/29/16) - Please have a look at this updated article here which includes ESXi 5.5 and 6.0 driver.

Disclaimer: In case it is not clear and apparent, you should never install any unknown 3rd party software on your ESXi host as it can potentially lead to instability issues or worse open yourself to a security hole. The following solution is not officially supported by VMware, please use at your own risk.

I decided to bite the bullet and give this solution a try and purchased the USB ethernet adapter from Amazon here.

usb-ethernet-adapter
There are two modules that needs to be downloaded, extracted and loaded onto ESXi. I have included the links below for your convenience:

As the VMTN thread mentioned, you can load using either the vmkload_mod or ESXCLI. Here are the two commands that I used in the following order:

vmkload_mod /vmfs/volumes/mini-local-datastore-1/usbnetvz026
vmkload_mod /vmfs/volumes/mini-local-datastore-1/ax88179vz026

When I tried to initially load either of the modules, I would always get the following error:

vmkwarning.log:2016-02-28T21:54:54.531Z cpu6:374787)WARNING: Elf: 2041: Load of <usbnetvz026> failed : missing required namespace <com.vmware.usb#9.2.1.0>

As you can imagine, I was pretty bummed to see this since I was afraid that something like this would happen. I was not sure if the device I had purchased no longer worked or if was the drivers? I saw that these modules were initially compiled for ESXi 5.1 (at the time, that was the latest version) and the only difference was that I was using a much newer version of ESXi, specifically 6.0 Update 1. I decided to install the latest version of ESXi 5.1 Update 3 and tried the process again and to my surprise, the modules loaded without errors. I suspect that this was a hard dependency on the namespace version which was version 9.2.1.0 and the latest version is now 9.2.3.0.

usb-network-adapter-esxi-1
After successfully loading the two modules, I ran the following command:

esxcfg-nics -l

to verify that ESXi did in fact did claim the USB ethernet device and as you can see from the screenshot below, it did indeed!

usb-network-adapter-esxi-2
Next up, I needed to verify basic connectivity and added the new uplink to my existing vSwitch. You must use the following ESXCLI command (esxcfg-vswitch command does not work apparently for non vmnicX devices)

esxcli network vswitch standard uplink add -u vusb0 -v vSwitch0

Once added, I hopped over to the vSphere C# Client to see if the device is now showing up under the Network Adapters tab, which it is.

usb-network-adapter-esxi-4
Finally, the last test was to make the vsb0 (this is how ESXi names the device) device the active connection while moving my existing vmnic0 to stand-by. Networking connectivity continued to function and I was even able to transfer an ISO image over the USB ethernet adapter without any issues.

usb-network-adapter-esxi-5
So it looks like it is possible to get a USB based ethernet adapter to function with ESXi, at least with the specific model listed above (PCI ID 0b95:1790). The challenge now is to see if there is a way to build an updated version of the drivers targeted at the latest ESXi 6.0 release. From what I have been able to follow on the forum here, it looks like there was also some amount of non-trivial code changes that were required to get the driver to function. If true, without those changes, it can difficult to re-compile the driver. I have reached out to the original author to see if he might be able to share the changes he made to the driver code. In the mean time, if folks are interested in giving the build process a try, Trickstarter did a great two part write up on how to setup your build environment and compile an example driver.

Although the write up is targeted at ESXi 5.x, you can download the equilvenet packages for ESXi 6.0 which includes the ESXi Open Source Disclosure Package as well as the VMware Toolchain which is required and used to compile the source code. I have provided the direct download links below.

You can also find the latest version of the USB ethernet adapter ax88179 ASIX driver here. I have also attempted to compile just the driver but have already ran into some issues. I have not had time to dig further, so not sure how far I will be able to get. Any tips or tricks others may have for compiling against ESXi 6.0, feel free to share them and I will give them a shot when I get some time!

Deploying Nested ESXi is even easier now with the ESXi Virtual Appliance

by 90 Comments

Several months back I had built an ESXi Virtual Appliance that allows anyone to quickly stand up a fully functional Nested ESXi VM which includes guest customization such as networking, NTP, syslog, passwords, etc. The virtual appliance was initially built for my own personal use as I found myself constantly rebuilding my lab environment for evaluating and breaking new VMware software. I figured if this was useful for myself, it probably could benefit others at VMware and I posted the details internally on our Socialcast forum. Since then, I have received numerous stories on how helpful the ESXi Virtual Appliance has been for both our Field and Engineering for setting up demos, POCs, evaluations, etc.

Most recently, I was contacted by Massimo Re Ferre' (crazy Mainframe guy ;)) who works over in our Cloud Native Apps team and was working on a pretty cool project with Photon Controller which was recently open sourced. He was interested in leveraging the ESXi Virtual Appliance along with using VMware AppCatalyst to make it super simple for anyone to try out the Photon Controller in their own environment. Over the last couple of weeks, I have been working closely with Massimo on incorporating on his requirements for the Photon Controller POC back into my ESXi Virtual Appliance. My original goal for the appliance was to keep it generic so that it could cater to multiple use cases and Photon Controller POC was just another neat solution that could be built on top of.

I just found out today that new Photon Controller POC has just been released and you can find more details in the links below:

As part of the release, the ESXi Virtual Appliance is also made avialable which I thought was pretty cool! 😀 I highly recommend you check out the awesome work done by Massimo if you want to play with Photon Controller. This is a really easy way of getting started with Photon Controller and giving it a spin in your own environment.

Since the ESXi Virtual Appliance is now available externally, I wanted to share a few details about the appliance for those who might be interested in checking it out. As I mentioned earlier, the goal of the ESXi Virtual Appliance was to be generic and to be used as a build block that could enable different use cases such as spinning up a quick vSphere lab using it and the VCSA or putting together a fully functional VSAN lab in literally a couple of minutes (at the very bottom, I have a couple of PowerCLI scripts to demonstrate this). You could deploy 3 instances of the appliance to get a basic 3 Node VSAN Cluster or you could scale up to 64 Node VSAN Cluster all within just minutes. The limit is truly your imagination.

The appliance contains a pre-installed GA release of ESXi 6.0 Update 1. There are 11 OVF properties that are available for customizing the Nested ESXi VM which are shown in the table below. Once powered on, the default 60 day evaluation will start counting down as if you had manually installed ESXi yourself. In addition, the OVA also contains several optimizations for running Nested ESXi including the Mac Learn dvFilter params as well as other configurations for quickly setting up a VSAN environment which are also described below. I have also built the appliance to be easily consume in all VMware based environments including vSphere, vCloud Air, Fusion, Workstation, Player & AppCatalyst.

UPDATE (05/10/17) - Updated VA to latest ESXi 6.0u3 & 6.5d (vSAN 6.6)

UPDATE (05/09/17) - The ESXi 5.5u3 has been decommissioned due to its limited use.

  • ESXi 5.5 Virtual Appliance download link (Decommissioned)

UPDATE (11/18/16) - ESXi 6.5 Virtual Appliance has been released and you can find the details here.

UPDATE (04/07/16) - Minor bug fix with refreshing auto-generated SSL Certificates, latest version is v5 for 6.0u2 VA and v2 for 5.5u3 VA

UPDATE (03/18/16) - I updated the ESXi 6.0 VA using vSphere 6.0 Update 2. It is now back to vHW10 for backwards compat and includes an All-Flash configuration + 2 VMXNET3 adapters and ready for VSAN 6.2 😀

UPDATE (03/01/16) - I have also created a new ESXi 5.5 VA using vSphere 5.5 Update 3b

OVF Property Description Type
guestinfo.hostname FQDN of the ESXi host string
guestinfo.ipaddress IP Address string
guestinfo.vlan VLAN ID string
guestinfo.netmask Netmask string
guestinfo.gateway Gateway string
guestinfo.dns DNS Server string
guestinfo.domain DNS Domain string
guestinfo.ntp NTP Server string
guestinfo.ssh  Whether or not SSH is enabled boolean
guestinfo.syslog Syslog Server string
guestinfo.password Root password for ESXi host string
guestinfo.createvmfs Whether to automatically creates a VMFS datastore (datastore1) on largest VMDK boolean

The ESXi 6.x Virtual Appliance includes the following configuration:

  • ESXi 6.0 Update 2
  • GuestType: ESXi 5.x (backwards compat)
  • vHW 10
  • 2 vCPU
  • 6GB vMEM
  • 2 x vmxnet3 vNIC
  • 1 x 2GB HDD (ESXi Installation)
  • 1 x 4GB SSD (for use w/VSAN, empty by default)
  • 1 x 8GB SSD (for use w/VSAN, empty by default)
  • VHV added (more info here)
  • dvFilter Mac Learn VMX params added (more info here)
  • disk.enableUUID VMX param added
  • VSAN traffic tagged on vmk0
  • Disabled VSAN device monitoring for home labs (more info here)

The ESXi 5.x Virtual Appliance includes the following configuration:

  • ESXi 5.5 Update 3b
  • GuestType: ESXi 5.x
  • vHW 10
  • 2 vCPU
  • 6GB vMEM
  • 2 x vmxnet3 vNIC
  • 1 x 2GB HDD (ESXi Installation)
  • 1 x 4GB SSD (for use w/VSAN, empty by default)
  • 1 x 8GB HDD (for use w/VSAN, empty by default)
  • VHV added (more info here)
  • dvFilter Mac Learn VMX params added (more info here)
  • disk.enableUUID VMX param added
  • VSAN traffic tagged on vmk0
  • Disabled VSAN device monitoring for home labs (more info here)
  • ESXi Embedded Host Client (more info here)

If you do not wish to use VSAN, there is an OVF property that allows you to specify whether or not a default VMFS datastore is created. You can increase the capacity of any of the disks after deployment (if you wish for the automatically VMFS creation, you will need to expand the disk prior to powering on the VM). Below are the different methods in which you can deploy the ESXi Virtual Appliance which includes vSphere, vCloud Air, Fusion, Workstation, Player & AppCatalyst. The idea is that you can easily setup Nested ESXi on any VMware based hypervisor and be up and running in just minutes!

Option 1 - Deploy to vSphere environment w/vCenter Server or vCloud Air

Download the OVA (or you can just paste the URL link) and import that into your vCenter Server using the vSphere Web Client. Make sure to accept the "Extra Configuration" when prompted and then fill out the 11 OVF properties which will allow you to customize the Nested ESXi VM to your environment. If you wish to increase the VMDK capacity and automatically have a VMFS datastore created for you automatically, be sure to expand the VMDK prior to powering on the VM. Once powered on, the customization process will start and in a few minutes you should have a fully functional Nested ESXi VM. Below are a couple of screenshots

Screen Shot 2015-12-10 at 1.35.02 PM
Screen Shot 2015-12-10 at 1.36.04 PM
Screen Shot 2015-12-10 at 1.38.51 PM

Option 2 - Deploy to ESXi

Download the OVA and import it into ESXi using either the vSphere C# Client but do not power it on. Since ESXi does not support OVF properties, you will need to add the following guestinfo*. properties as shown in "Option 3" below by using the VM Advanced Settings UI.

Screen Shot 2015-12-11 at 6.48.51 AM
If you prefer NOT to mess around with manually adding these VM Advanced Settings which can also be automated using the vSphere API or PowerCLI, one additional method which CAN make use of the OVF properties is by using ovftool which is a CLI to import the OVA and using the --injectOvfEnv option that was added in ovftool version 4.x You can find more details in this blog post here.

Option 3 - Deploy to VMware Workstation, Fusion or Player

Download the OVA and import it into Fusion/Workstation but do not power it on. You will then need to edit the VMX file and add the following guestinfo.* properties as shown below since Fusion/Workstation do not support OVF properties. If you wish to increase the VMDK capacity and automatically have a VMFS datastore created for you automatically, be sure to expand the VMDK prior to powering on the VM. Once you have saved the changes, you can then power on the Nested ESXi VM and the customization process will start and in a few minutes you should have a fully functional Nested ESXi VM.

Option 4 - Deploy to VMware AppCatalyst

Download the the OVA and import that into AppCatalyst using ovftool but do not power it on. You will then need to edit the VMX file and add the following guestinfo.* properties as in the above example for Workstation/Fusion in addition to the following params listed below which are required for Nested ESXi to run in AppCatalyst. If you wish to increase the VMDK capacity and automatically have a VMFS datastore created for you automatically, be sure to expand the VMDK prior to powering on the VM.

You will also need to run the following command to allow promiscuous mode in AppCatalyst since there's no UI to prompt (this is only required once):

touch "/Library/Preferences/VMware AppCatalyst/promiscAuthorized"

Once you have saved the changes, you can then power on the Nested ESXi VM and the customization process will start and in a few minutes you should have a fully functional Nested ESXi VM.

Option 5 - Deploy to vSphere using PowerCLI

I have also created two very simple PowerCLI scripts which demonstrates how you can easily deploy N-number of these Nested ESXi VMs and in fact, you can setup a fully functional VSAN Cluster in just under 5 minutes! You can find the two scripts below:

The first script will perform the deployment of the Nested ESXi VM, but you will first need to convert the OVA to an OVF because the Get-OvfConfiguration cmdlet does not support OVA. To properly convert the OVA, you will need ovftool and then run the following command:

ovftool.exe --allowAllExtraConfig --skipManifestCheck Nested_ESXi_Appliance.ova Nested_ESXi_Appliance.ovf

The second script will go ahead and add the deployed Nested ESXi VMs to a vSphere Cluster and then enable VSAN on the vSphere Cluster.

Migrating ESXi to a Distributed Virtual Switch with a single NIC running vCenter Server

by 21 Comments

Earlier this week I needed test something which required a VMware Distributed Virtual Switch (VDS) and this had to be a physical setup, so Nested ESXi was out of the question. I could have used my remote lab, but given what I was testing was a bit "experimental", I prefered using my home lab in the event I need direct console access. At home, I run ESXi on a single Apple Mac Mini and one of the challenges with this and other similar platforms (e.g. Intel NUC) is that they only have a single network interface. As you might have guessed, this is a problem when looking to migrate from a Virtual Standard Switch (VSS) to VDS, as it requires at least two NICS.

Unfortunately, I had no other choice and needed to find a solution. After a couple minutes of searching around the web, I stumbled across this serverfault thread here which provided a partial solution to my problem. In vSphere 5.1, we introduced a new feature which would automatically roll back a network configuration change if it negatively impacted network connectivity to your vCenter Server. This feature could be disabled temporarily by editing the vCenter Server Advanced Setting (config.vpxd.network.rollback) which would allow us to by-pass the single NIC issue, however this does not solve the problem entirely. What ends up happening is that the single pNIC is now associated with the VDS, but the VM portgroups are not migrated and the reason that this is problematic is that the vCenter Server is also running on the ESXi host which it is managing and has now lost network connectivity 🙂

I lost access to my vCenter Server and even though I could connect directly to the ESXi host, I was not able to change the VM Network to the Distributed Virtual Portgroup (DVPG). This is actually an expected behavior and there is an easy work around, let me explain. When you create a DVPG, there are three different bindings: Static, Dynamic, and Ephemeral that can be configured and by default, Static binding is used. Both Static and Dynamic DVPGs can only be managed through vCenter Server and because of this, you can not change the VM network to a non-Ephemeral DVPG and in fact, it is not even listed  when connecting to the vSphere C# Client. The simple work around is to create a DVPG using the Ephemeral binding and this will allow you to then change the VM network of your vCenter Server and is the last piece to solving this puzzle.

Disclaimer: This is not officially supported by VMware, please use at your own risk.

Here are the exact steps to take if you wish to migrate an ESXi host with a single NIC from a VSS to VDS and running vCenter Server:

Step 1 - Change the following vCenter Server Advanced Setting config.vpxd.network.rollback to false:

migrating-from-vss-to-vds-with-single-nic-1
Note: Remember to re-enable this feature once you have completed the migration

Step 2 - Create a new VDS and the associated Portgroups for both your VMkernel interfaces and VM Networks. For the DVPG which will be used for the vCenter Server's VM network, be sure to change the binding to Ephemeral before proceeding with the VDS migration.

migrating-from-vss-to-vds-with-single-nic-0
Step 3 - Proceed with the normal VDS Migration wizard using the vSphere Web/C# Client and ensure that you perform the correct mappings. Once completed, you should now be able connect directly to the ESXi host using either the vSphere C# Client or ESXi Embedded Host Client to confirm that the VDS migration was successful as seen in the screenshot below.

migrating-from-vss-to-vds-with-single-nic-2
Note: If you forgot to perform Step 2 (which I initially did), you will need to login to the DCUI of your ESXi host and restore the networking configurations.

Step 4 - The last and final step is to change the VM network for your vCenter Server. In my case, I am using the VCSA and due to a bug I found in the Embedded Host Client, you will need to use the vSphere C# Client to perform this change if you are running VCSA 6.x. If you are running Windows VC or VCSA 5.x, then you can use the Embedded Host Client to modify the VM network to use the new DVPG.

migrating-from-vss-to-vds-with-single-nic-3
Once you have completed the VM reconfiguration you should now be able to login to your vCenter Server which is now connected to a DVPG running on a VDS which is backed by a single NIC on your ESXi host 😀

There is probably no good use case for this outside of home labs, but I was happy that I found a solution and hopefully this might come in handy for others who might be in a similar situation and would like to use and learn more about VMware VDS.

ESXi 6.0 on Apple Xserve 2,1

by 7 Comments

I really enjoy hearing from my readers, especially when they share some of the unique challenges they have come across or boundaries they have pushed with our VMware software. Several weeks back I received an interesting email from a reader named John Clendenen who was able to get ESXi 6.0 running on both his Apple Xserve 3,1 as well as Xserve 2,1! To be honest, I was pretty surprised to hear that this worked and not only that, there was not a whole lot that John had to do to get it working. I thought this was a pretty cool setup and asked if John was interested in sharing more details with the VMware/Mac OS X community in case others were interested.

Disclaimer: This is not officially supported by VMware, please use at your own risk.

*** This is a guest blog post from John Clendenen ***

For the past 5 years, I have lived in New York where I work at various print and post-production studios. The IT situations in many of these locations is often home-grown and sub-adequate, if not barely functional, so I took it upon myself to learn OS X Server administration. I have a background in computer science and networking, so it wasn’t a huge leap to step into administration. I was able to gradually build one studio’s IT infrastructure form a single AFP share with incessant permissions problems, to an Open Directory driven, single sign-on infrastructure with mobile homes, messaging etc.. However, with each OS X Server update, things broke, and I was spending a lot of time putting out fires.

This led me to pursue virtualization in hopes of isolating services on separate machines, without telling the studio to go buy 8 Apple Mac Minis. However, I needed to learn how to do this, before pitching it to them, so I looked on Craigslist for some cheap hardware. I found an Xserve 2,1, which I was able to talk down to $100. I also found a brief post in a thread that said that the Xserve 2,1 ran ESXi 5.5 without issue. I figured I’d take the plunge and just resell it on eBay if it didn’t work out.

Since then, my home lab has grown considerably, and I have learned that the best way to provide Mac services, is simply to use Linux (I’ve had a great experience with Netatalk). That said, Open Directory, Software Update, Caching and a few other services still need to be run on a Mac, so it’s still necessary to have the hardware. For a while, I had 3 Xserves, all purchased very cheaply, running VMs. I just sold one, and will sell another in the next month or two in favor of some Supermicro hardware (I really am mostly running Linux VM’s at this point). I’ll keep the one to run a few Mac OS X VMs. I’m still working on getting Samba 4 to work as the PDC, but once that is running smoothly, I’ll have a functional environment that I can take to the studios where I work (with approved hardware of course). Then I’ll have an improved work experience, while also pulling some extra income on the installation/maintenance.

Anyway, you’ve come here to read about running ESXi 6.0 on an Xserve 2,1. Well, it works. It’s 7 years old, and not officially supported, but you already know that. So, if you were considering it, I doubt anything here will dissuade you. On top of that, there’s not much for me to say other than, it works. No tricks to it, just install and you’re off.

pic1
That said, I do have some recommendations and tips to help you if you want to get the most out of this hardware. 2 months ago, since I swapped out the RAID card for the standard backplane/interconnect and upgraded the RAM. It hasn’t skipped a beat since.

My System

This is my home custom 13u rack with sound insulation and active air flow. Networking is in the back (Ubiquiti). It sits in a lofted storage area with an A/C and vents into the bathroom.

pic8
Here you see an Xserve 3,1 on top of an Xserve 2,1. There’s a blank unit because I just sold an Xserve 2,1 on eBay, and the other 2,1 will be for sale soon as well to make room for a 4 node 2U Supermicro Server. The NAS is comprised of a 4u head unit and a 4U JBOD running Centos. Last of course, is a 2U CyberPower UPS which is really just to condition the power and keep brownouts from taking down the system.

I have about a dozen VM’s running between the 2 Xserves. I have each Mac OS X service separated on it’s own installation. This way I can run updates per service. It’s especially nice to have Open Directory separate from less important services. I also have Debian, Centos and OpenBSD VMs running various services. Getting the Xserve 3,1 running ESXi 6 is possible, but more problematic. Now that I have it working, I’m dropping the 2,1’s simply because the processors aren’t multithreaded. I am currently working on a companion article to this one, detailing my experience with the Xserve 3,1, so that information will be available soon.

Storage

1. Don’t use the RAID backplane/interconnect or count on using it. ESXi 6 does not recognize it, and RDM appears to work at first and then will crash your VM and never show up again. You can have it installed in the Xserve without any issue, but you’ll get a lot more mileage out of the hardware if you have the standard backplane/interconnect.

The backplane you want appears periodically on eBay. The part number is: 661-4648

2. If/once you have the non-RAID backplane/interconnect, keep in mind that it is SATAII and will only support 3Gb/s. I am using 3 old 500Gb WD RE3’s, but I’d recommend using some older SSDs that will max out the SATAII interface without paying for speed you can’t use. Be sure you consult the Apple Drive Module compatibility page to make sure you have the right drive caddies. They all fit, but they don’t all work.

Apple Drive Module Compatibility Chart: https://support.apple.com/en-us/HT1219

pic2
3. PCIE Flash is a good idea, whether you use it for cache, VM’s or as the ESXi boot disk, it is by far the fastest storage option. I have not invested in it, but the good people at Mushkin have told me their Scorpion PCIE flash will work in ESXi 6. Please contact them yourself though before investing in one of their cards. I have not tested them. While this will give you the best performance out of your Xserve 2,1, it seems like overkill to me, but hey, if you want to push this thing, you might find some meaningful performance gains here.

Mushkin Scorpion PCIE Flash: http://www.poweredbymushkin.com/index.php/products/solid-state-drives.html

4. It might occur to you that replacing the optical drive with an ssd might be a good idea. While MCE Technologies makes “Optibay” for Xserve 2,1, the connection is IDE, so this is not recommended. I also don’t know if ESXi would recognize it. My gut says probably, but again, it’s too slow to be useful. It isn’t that cheap either.

MCE Technologies Optibay: http://store.mcetech.com/mm/merchant.mvc?Store_code=MTOS&Screen=PROD&Product_Code=OBSXGB-XS

Network

There are a lot of options here. You can plug any network card that you want really as long as you can at least find an anecdotal account of it working on ESXi 6.

I have one such anecdote for you, and it is also a strong suggestion. The company Silicom makes/made several models of gigabit NICs which are all incredibly inexpensive, are all over eBay and work in ESXi 6. Buy the 6 port model. It’s cheap and you’ll get great scaling with ESXi load balancing across 8 gigabit ports.

The Xserve 2,1 has the added advantage here of an optional PCIX riser. If your model has one, or you find one for cheap, you can save even more on the Silicom NIC. The PCIE models go for $60-$80 on eBay, while the PCIX models go for $40.

pic3

Memory

ECC DDR2 is pretty cheap and easy to find used. I recommend memory4less though. I had to return some RAM that was mislabeled from a random eBay distributor. Memory4less will get it right. http://www.memory4less.com/

pic4

Processors

One great perk of the Xserve 2,1, is that you can upgrade the single processor hardware to dual processor. You can pick up an additional processor on eBay for $40 or so, but you’ll need to get a heat sink as well. The single processor units come with a fake aluminum heat sink, but do not use it. You want a copper one. I believe the heat sinks in the Xserve 1,1 are the same. Don’t forget the thermal paste.

pic5

Minor Issues

1. The Performance tab throws an error.

pic6
2. Not sure about the hardware sensors, but it looks like not everything is working even if it’s showing up. I did not do any testing here.

pic7
Stay tune for Part II of John's guest blog post on running ESXi 6.0 on Xserve 3,1.