virtuallyGhetto

esxi5

Installing ESXi 5.1 Update 1 on Mac Mini is Now a Breeze! (No Custom ISO/patches Needed!)

by 68 Comments

ESXi 5.1 Update 1 was just released by VMware and similar to the ESXi 5.0 Update 2 release last year, the tg3 (Broadcom) driver has now been updated to 3.123b.v50.1 which is required to support network connectivity on the Apple Mac Mini's. Prior to this, to install ESXi on an Apple Mac Mini, users were required to build a custom ISO that included the updated tg3 driver and I am happy to say this is no long necessary! In addition, having the latest driver also provides out of the box support for the Thunderbolt ethernet adapter which is great if you are looking to add an additional ethernet connection to the Apple Mac Mini.

Disclaimer: The Apple Mac Mini is not officially supported by VMware.

Here is a quick screenshot of the networking details including the Thunderbolt ethernet adapter on my Apple Mac Mini 5,3 running ESXi 5.1 Update 1:

Apple Mac Mini 6,2 Users

If you have an Apple Mac Mini 6,2 the workarounds described in this article are still required as well as an additional issue regarding the on-board NIC for the Apple Mac Mini 6,2. The issue is that the on-board NIC is not recognized and claimed by the tg3 driver because there is not a PCI ID entry in the tg3 map file. Note, BCM57766 driver is supported, just that it was missing the mapping (I believe this was the case in 5.0 as well). Working with VMware engineering, I was able to obtain the SMC fix and along with the workarounds that I built earlier for ESXi 5.0 and I have created a custom ESXi 5.1 Update 1 ISO that includes all the fixes and you will be able to install it on your Apple Mac Mini 6,2 without any further modifications.

Download: ESXi-5.1u1-MacMini-SMC-BOOT-FIX-6-2.iso

Note: From my understanding, some of these issues have been fixed but just did not make it in time for the Update 1 release. I'm hoping that we will be able to get most of these fixed in a future patch or update. This is actually pretty awesome, given that this is NOT SUPPORTED by VMware, but engineers are still willing to help the community out!

UPDATE: When trying to enable vSphere HA, the following error "Could not find a trusted signer" is thrown. To get around this problem, you will need to manually install the vSphere HA VIB and then reconfigure vSphere HA. The easiest method is to SCP the VIB from the VCSA and then using ESXCLI and the --no-sig-check flag to install the VIB.

Here are the commands to run:

scp /etc/vmware-vpx/docRoot/vSphere-HA-depot/vib20/vmware-fdm/VMware_bootbank_vmware-fdm_5.1.0-1064983.vib [email protected]:/tmp
esxcli software vib install -v /tmp/VMware_bootbank_vmware-fdm_5.1.0-1064983.vib --no-sig-check

Additional Resources:

Configure Apple Mac Mini to Default Boot ESXi

by 13 Comments

If you are running ESXi on an Apple Mac Mini and it is installed on a USB key, you probably have noticed that the Mac Mini tries to boot from disk by default and instead of using the USB device. This means when you reboot your ESXi host each time, you will need to hold down the "ALT/OPTION" key which will present you with a boot menu to select the device you wish to boot from.

This can be quite annoying if you have a headless setup for your Mac Mini and you just want it to automatically boot off of the right device containing your ESXi installation. To fix this, you can configure the default boot device which can be done by first selecting the device you wish to boot off of as shown in the screenshot above. Next, hold down on the "CONTROL" key which will turn the straight arrow into circular arrow icon as shown in the screenshot below.

Now you just need to either hit enter or if you have a mouse, click on the circular arrow icon and this will configure the default boot device the Apple Mac Mini will use going forward. It is that simple! If you want to boot off of another device after configuring the default boot device, you can still do so by holding down "ALT/OPTION" key while the Mac Mini is still booting up.

Credit goes to this site for solution. 

Installing ESXi 5.0 Update 2 on Mac Mini is Now a Breeze! (No Custom ISO/patches Needed!)

by 15 Comments

VMware has just released ESXi 5.0 Update 2 which includes many bug fixes, but along with these fixes, these updates usually also include new inbox drivers as part of the default ISO image for ESXi. One important driver that I had noticed while going through the release notes is the inclusion of the tg3 (Broadcom) inbox driver:

  • Updates the tg3 driver to version 3.123b.v50.1
    The tg3 inbox driver version shipped with the ESXi 5.0 Update 2 is 3.123b.v50.1.

Disclaimer: The Apple Mac Mini is not officially supported by VMware. 

Why is this awesome!? Well, for those of you who own an Apple Mac Mini and would like to run ESXi, may recall an additional step is required to create a customized ESXi ISO to include an updated tg3 driver for the networking stack to function in an Apple Mac Mini. Though the steps have been documented here, it is great to see this working right out of the box using the new ESXi 5.0 Update 2 ISO from VMware. In addition to the networking stack functioning properly after installation, it also enables connectivity to an Apple Thunderbolt Ethernet adapter if you happen to have one connected to your Apple Mac Mini! You no longer have to create a custom ESXi ISO for the Thunderbolt Ethernet adapter as mentioned in an earlier article here.

Note: This article is only relevant to pre-2012 Apple Mac Mini, if you have a newer Apple Mac Mini 6,2 - Please refer to this article for installation.

Here are a few screenshots of running the latest ESXi 5.0 Update 2 on my Apple Mac Mini 5,3 as well as showing the Apple Thunderbolt Ethernet adapter active in ESXi:

If you want a tiny form factor for a vSphere home lab, you should definitely consider asking Santa for an Apple Mac Mini this Christmas 😉 Hope everyone has a Happy Holiday and Happy New Years! 

Running ESXi 5.0 & 5.1 on 2012 Mac Mini 6,2

by 59 Comments

If you recently purchased the new 2012 Apple Mac Mini 6,2 which was just released not too long ago and tried to install either ESXi 5.0 or 5.1, you probably noticed a PSOD (Pink/Purple Screen of Death) during the installation. This is currently a known issue and there is an extensive VMTN thread (9,300+ views) about this problem which also includes a fix through a collaboration between VMTN community user zer010gic and VMware Engineer dariusd. Even though the Apple Mac Mini is not an officially supported hardware platform for running ESXi, it is great to see VMware engineers going out their way and trying to help the VMware community find a solution as well as providing an "unofficial" fix in this case.

I would also like to point out that this issue only applies to the new 2012 Apple Mac Mini, for previous models such as the Apple Mac Mini 5,1 or 5,3 you can install ESXi 5.0 or 5.1 without any issues. For more details, please refer to the instructions in this blog post.

Disclaimer: The Apple Mac Mini is not officially supported by VMware. The only supported platform for ESXi 5.0 for Apple hardware is the Apple XServe 3,1 and for ESXi 5.1 is the Apple Mac Pro, which you can get more details here.

Before jumping into the solution, if you think VMware should support the Apple Mac Mini for running ESXi, please provide feedback to VMware by submitting a Feature Request. The more feedback that VMware receives from customers along with business justifications, the better our product management team can prioritize features that are most important to our customers.

Here are the current problem/solutions when trying to install on the new Mac Mini:

Problem: PSOD during ESXi 5.0 or 5.1 installation.
Solution: Add iovDisableIR=true to the kernel option before attempting installation. When you are asked to reboot, be prepared to enter iovDisableIR=true again (SHIFT+O) which is required to get ESXi to boot after installation. Once the system has booted up, go ahead and run "esxcli system settings kernel set -s iovDisableIR -v true" in the ESXi Shell to persist the kernel setting. This is a "temp" workaround while PSOD is being investigated.

Problem: Unable to install new OSX Server on a VM or power on existing OSX Server VMs.
Solution: There appears to be a significant change in Apple's SMC (System Management Controller) device in the newer models that prevents the Apple SMC VMkernel driver from properly loading. A tempoary fix was provided to zer010gic to create a custom ISO until the fix is integrated into a future release.

Note: There may be other minor/unconfirmed issues listed on the VMTN thread, but for basic ESXi installation/usage + OSX Server VM creation/installation, the above solutions should be sufficent.

Instead of having everyone walk through the process of creating a custom ESXi ISO which includes the two fixes mentioned above as well as the bundling the updated tg3 Broadcom network drivers for network connectivity, zer010gic has generously created and is hosting ESXi 5.1 ISOs for users to download and use. It contains some work that I have been doing with zer010gic to create an ESXi 5.1 ISO that does not require any manual intervention outside of the normal ESXi installation. I recently completed the rest of this work which is based off of the oriignal ISO that zer010gic has shared on the VMTN community (unfortunately I have not been able to get a hold of him to provide him with the necessary bits and I have decided to post a modified ISO).

Here is a step by step instruction for zer010gic ESXi 5.1 ISO

Step 1 - Download zer010gic ESXi-5.1-MacMini-SMC-6-2.iso.

Step 2 - Transfer ISO to either USB key or CD-ROM

Step 3 - Perform ESXi installation as you would, but when you get to the very last step prior to rebooting, be ready for some typing when the host boots back up (this is important else you will get a PSOD)

Step 4 - When ESXi starts to boot up, hit SHIFT+O which will allow you to add additional kernel boot option. Add the following text the bootUUID (remember to add a space first)

iovDisableIR=true

This step is required to ensure your ESXi boots up properly for the first time so you can permanently enable this kernel option using ESXCLI which will then persist this upon sub-sequent reboots.

Step 5 - Login to ESXi Shell (you may need to enable it first) and run the following ESXCLI command:

esxcli system settings kernel set -s iovDisableIR -v true

Once this is set, you no longer have to do this again. If you prefer not to go through these manual steps, please refer to the section below for a modified ESXi 5.1 ISO which automates all this for you.

Here is my modified ESXi 5.1 ISO which does not require any additional user intervention

Step 1 - Download my ESXi-5.1-MacMini-SMC-BOOT-FIX-6-2.iso

Step 2 - Transfer ISO to either USB key or CD-ROM

Step 3 - Go through normal ESXi install and enjoy

Note: For details on how I automated the kernel setting setting, take a look at the very end.

So if you are looking to refresh your home lab, you just may want to consider using the new Apple Mac Minis, especially with small form factor footprint 🙂

Note: A couple of users mentioned it took a bit of time to boot up, specifically when usbarbitrator module is being loaded. I noticed this too and it took quite a bit of time, probably 5-6 minutes. If you do not plan on any USB pass-through from the Mac Mini to your guestOSes, you can actually disable this service which should help speed the bootup. If you wish to disable usbarbitrator, run the following command:

chkconfig usbarbitrator stop

ESXi ISO Customization Details

If you take a look at the steps required to install the ISO provided by zer010gic, most of the heavy work has already been done for you. The only "manual" part that is required from the user is to enter a kernel option during the first boot and then run an ESXCLI command to persist this kernel setting which will prevent Mac Mini from PSODing. Removing these these manual steps is actually harder than it looks because of when you need to actually perform the changes. After much trial and error, I came up with the following script below (it's not the cleanest, but it works).

Basically the script is loaded from custom.tgz and executed before the installation begins and it generates a script stored in /tmp/customboot.sh which will look for the boot.cfg configuration file stored in the primary bootbank. This is where we insert the iovDisableIR=true parameter so the user is not required to do this after the first boot up. The challenge with this is the boot.cfg does not exists until after the installation has completed, so what I ended up doing was insert a command into /usr/lib/vmware/weasel/process_end.py which is part of the weasel installer for ESXi and is the very last script that is called when a user hits reboot. The command points back to the /tmp/customboot.sh which will perform the insert into boot.cfg right before rebooting. To automatically take care of the ESXCLI configuration, I added the ESXCLI command to /etc/rc.local.d/local.sh which will automatically run after all init scripts have executed. Then finally, I need to clean up local.sh since I only need that that run once which is handled by another script that is also created and stored in /etc/init.d/customcleanup which will just clean up local.sh file as well as delete itself. Simple right? 😉

Note: There is probably a more optimal way of doing this, probably using one of the weasel installer scripts and just set the boot.cfg option and then clean up with an init script, but I decided to leverage some of my earlier work for Disabling LUN Duringn ESXi Installation

Here is the script within the custom.tgz file:

Creating Custom VIBs For ESXi 5.0 & 5.1 with VIB Author Fling

by 37 Comments

VMware Labs just released a really cool new Fling called VIB Author which is a tool that allows you to easily create custom VIBs for your ESXi 5.x hosts. If you have tried to create custom ESXi firewall rules or add custom scripts to your ESXi host, you may have noticed they are not persisted after a system reboot and you had to play all sorts of games to get the files to persist. The VIB Author tool now solves that problem and you can even take your custom VIB and integrate them into an Auto Deploy Image Profile using Image Builder. Before you jump right in, be sure to read over the important note in the documentation before getting started.

So how does the VIB Author tool work?

You will need to provide two pieces of input: payload which is set of files you wish to include in your VIB and the descriptor.xml which contains the metadata for your files. From that, VIB Author can produce either a VIB and/or an offline bundle (can be used with Image Builder).

VIB Author is distributed only as an RPM and you will need to install the VIB Author tool on a 32-bit Linux system (sorry, no 64-bit support). In my home setup, I went with CentOS 6.2 i386 as it was free to download & easy to setup or you may choose go with SUSE Linux Enterprise 11 SP2 which is the recommended platform per the documentation.

To install the RPM, run the following command:

rpm -ivh vmware-esx-vib-author-5.0.0-0.0.844296.i386.rpm

In the example below, I will show you how to create a custom VIB that contains several different configurations:

  • Custom Firewall Rule
  • Custom Startup script (adds a static route)
  • Custom Files (ghettoVCB)

Disclaimer: The example below is not officially supported by VMware, please thoroughly test this in a development environment before using in production.

Here is the directory structure for the example that we will be going through:

Step 1 - Create your stage directory structure which we will then populate with your payload files as well as the descriptor.xml file.

mkdir -p stage/payloads/payload1 

Step 2 - Create your descriptor.xml file which should be placed in the stage directory. For more details on the parameters within the descriptor.xml, please take a look at the documentation.

Here is an example of my descriptor.xml file:

Step 3 - Create the directory structure and store the files you wish to include under payload1. Ensure the the directory structure matches the absolute path of how you want the files to appear on the ESXi host. For example, if you wish to create a file call foo in /etc/vmware/foo then your directory structure should look like stage/payloads/payload1/etc/vmware/foo

Note: In the documentation, there is a list of default supported paths, if you venture off of this supported list, then you will need to issue the -f flag when creating your VIB as well as installing your VIB on your ESXi host

So for our examples we have the following files:

stage/payloads/payload1/etc/vmware/firewall/virtuallyghetto.xml
This one should be pretty straight forward, we are just creating a custom ESXi firewall rule and you will need to place your configuration file under /etc/vmware/firewall, please take a look at this article for more details on creating your own firewall rules.

stage/payloads/payload1/etc/rc.local.d/999.addStaticRoute.sh
This is a custom shell script that adds a static route to an ESXi host upon bootup under /etc/rc.local.d. There maybe other startup scripts that could be executed and you do not want to conflict with any system defaults. I recommend you label yours with a high number such as 999 to ensure it is one of the last scripts to execute.

stage/payloads/payload1/opt/ghettoVCB/{ghettoVCB.conf,ghettoCB-restore.sh,ghettoVCB.sh}
This is a custom set of files that I would like to store in ESXi under /opt directory and the files are my free ghettoVCB backup script.

Here is a copy of my directory structure (stage.zip) which can be used as a reference.

Step 4 - Now we ready to create our VIB and/or offline bundle by specifying our stage directory as input. In this example, we will generate both a VIB as well as an offline bundle containing the same contents. Run the following command:

vibauthor -C -t stage -v virtuallyghetto.vib -O virtuallyghetto-offline-bundle.zip -f

Note: Since we added some files outside of the default supported paths, we also need to specify the -f flag to force the creation.

We can also extract information about our VIB by using the -i option in VIB Author, to do so, run the following command:

vibauthor -i -v virtuallyghetto.vib

Finally, we are now ready to copy the VIB over to our ESXi host and install our custom VIB.

To install VIB run the following command:

esxcli software vib install -v /vmfs/volumes/[datastore-name]/virtuallyghetto.vib -f

To install the offline bundle run the following command:

esxcli software vib install -d /vmfs/volumes/[datastore-name]/virtuallyghetto-offline-bundle.zip -f

Note: You need to specify the -f flag to force the installation since we created files in an unsupported path. I have been able to test the VIB and offline bundle installation on both ESXi 5.0 as well as ESXi 5.1

To confirm we have succesfully installed our custom VIB, we can query it by running the following command:

esxcli software vib list | grep virtuallyghetto

So there you have it, in just a few steps, you can create your own custom VIBs!

That’s so cool! Running ESXi 5.0 & 5.1 on Apple Mac Mini

by 38 Comments

Those of you who follow me on twitter should know that I recently got my hands on an Apple Mac Mini Server (Thanks Randy K.) and are probably thinking I would install Apple OSX on the Mini. Nope! I am actually running vSphere ESXi 5.0 or 5.1 on the Mac Mini!

UPDATE (12/21): Yo no longer need to create a customized ESXi ISO for Apple Mac Mini, the necessary drivers are now included in the new ESXi 5.0 Update 2 and will work out of the box. Please take a look at this article for more details.

UPDATE (01/18): To run ESXi 5.1 on Mac Mini 5,1 or 5,3 please follow the same steps outlined below but for an ESXi 5.1 ISO image. If you are trying to run ESXi 5.0 or 5.1 on the new 2012 Apple Mac Mini 6,2 please refer to this blog post for the instructions.

Disclaimer: This is not officially supported by VMware. Use at your own risk.

Note: I did not have a spare monitor at home and luckily the Mac Mini has a DVI output which I was able to connect to my 46" TV. Nothing like ESXi on the big screen 🙂

Even though this is not officially supported by VMware, it is still a very cool solution and the Mac Mini is great form factor for a vSphere home lab. I also want to mention that this was only possible with the research from the folks over at Paraguin Consulting who initially blogged about the process needed to get ESXi 5 running on a Mac Mini. I would highly recommend you check out their blog post which provides additional details as well as a step by step installation guide including screenshots for each step.

I did not have access to an Apple Super Drive which is what the Paraguin folks used in their installation guide, as they thought formatting a USB key would have taken too long ... and who has a CD burner these days? 😉 There is also an additional step that is needed to get network connectivity which requires the user to manually install a network driver on the ESXi host.

I decided to go down the route of using a USB key to perform the installation and using a spare 1GB USB key, I created a custom ESXi installation that included the network driver which allows for network connectivity during and after the installation.

What you will need:

Step 1 - Extract the offline bundle "tg3-3.120h.v50.2-offline_bundle-547149.zip" from Broadcom zip file

Step 2 - Use Image Builder to add the Broadcom driver and create a custom ESXi 5 ISO (steps taken from this VMware KB article)

# Add the ESXi 5.0 Update 1 Offline Bundle

Add-EsxSoftwareDepot "C:\VMware-ESXi-5.0.0-623860-depot.zip"

# Add the Broadcom Offline VIB

Add-EsxSoftwareDepot "C:\tg3-3.120h.v50.2-offline_bundle-547149.zip"

# Create a new Image Profile (in example, I'm using the full version w/tools)

New-EsxImageProfile -CloneProfile "ESXi-5.0.0-623860-standard" -name "ESXi50u1-Custom"

# Add the broadcom driver to our Image Profile

Add-EsxSoftwarePackage -ImageProfile "ESXi50u1-Custom" -SoftwarePackage "net-tg3"

# Create an ISO from our custom Image Profile

Export-EsxImageProfile -ImageProfile "ESXi50u1-Custom" -ExportToISO -filepath C:\VMware-ESXi-5.0u1-Custom.iso

Step 3 - Plug in a USB key into your system and run UNetbootin which will take the custom ISO we just created and make it bootable on the USB key

Step 4 - Finally, power on your Mac Mini and plug in the USB key. You can either hold the "Alt/Option" key while the system is booting and select the EFI volume OR just plug the USB key and the ESXi installer should automatically start up.

After a few minutes, you should now have ESXi 5 running on your Mac Mini, here is my setup:

Note: If you wish to add a custom kickstart file so it automatically installs and configures the host, take a look at this blog post for the details. If you want the Mac Mini to automatically boot from USB without having to press the "Alt/Option" key after installation, you can set the default boot device by following this article here.

Since you are running ESXi on Apple hardware, you can also create Mac OSX Virtual Machines (10.5 Server and 10.6 Server) and with the latest release of vSphere 5.0 Update 1, OSX 10.7 (Lion) is now officially supported. To install OSX 10.7 as a VM, make sure you follow the instructions here.

Here is a screenshot of running OSX Lion 64bit as a VM running on ESXi 5 running on a Mac Mini 😀

If you are looking to refresh your vSphere home lab, definitely consider looking at the Mac Mini, which can get up to a max of 16GB of memory. Also if you are wondering if other people are doing this, I would recommend you check out Christopher Well's vSamarai blog here which details his experience using the Mac Mini as well as voting for his VMworld CFP which will also include topic about running ESXi on Mac Mini.

I want to thank Randy K. again for lending me the Mac Mini, you rock dude! Hopefully this will not be the last article about the Mac Mini 😉

Additional Resources: