Installing ESXi 5.0 Update 2 on Mac Mini is Now a Breeze! (No Custom ISO/patches Needed!)

VMware has just released ESXi 5.0 Update 2 which includes many bug fixes, but along with these fixes, these updates usually also include new inbox drivers as part of the default ISO image for ESXi. One important driver that I had noticed while going through the release notes is the inclusion of the tg3 (Broadcom) inbox driver:

• Updates the tg3 driver to version 3.123b.v50.1
  The tg3 inbox driver version shipped with the ESXi 5.0 Update 2 is 3.123b.v50.1.

Disclaimer: The Apple Mac Mini is not officially supported by VMware. 

Why is this awesome!? Well, for those of you who own an Apple Mac Mini and would like to run ESXi, may recall an additional step is required to create a customized ESXi ISO to include an updated tg3 driver for the networking stack to function in an Apple Mac Mini. Though the steps have been documented here, it is great to see this working right out of the box using the new ESXi 5.0 Update 2 ISO from VMware. In addition to the networking stack functioning properly after installation, it also enables connectivity to an Apple Thunderbolt Ethernet adapter if you happen to have one connected to your Apple Mac Mini! You no longer have to create a custom ESXi ISO for the Thunderbolt Ethernet adapter as mentioned in an earlier article here.

Note: This article is only relevant to pre-2012 Apple Mac Mini, if you have a newer Apple Mac Mini 6,2 - Please refer to this article for installation.

Here are a few screenshots of running the latest ESXi 5.0 Update 2 on my Apple Mac Mini 5,3 as well as showing the Apple Thunderbolt Ethernet adapter active in ESXi:

If you want a tiny form factor for a vSphere home lab, you should definitely consider asking Santa for an Apple Mac Mini this Christmas ;) Hope everyone has a Happy Holiday and Happy New Years! 

vCenter Server Simulator

I recently spent some time experimenting with a really cool tool found only in the VCSA (vCenter Server Appliance) called vcsim, short for vCenter Simulator. I initially noticed vcsim during some of my early beta testing of vSphere 5.1 and during this period it is not uncommon to find various utilities and debugging tools used by developers and QE for testing. At the time, I was more focused on usability issues and reporting bugs with the product and I did not think much of this vcsim. It was only until recently, after talking to Mr. Not Supported aka Randy Keener, did I look into vcsim again as it appears to have been included in the GA (generally available) build of the VCSA 5.1 which I had not expected.

Disclaimer: This is not officially supported by VMware, use at your own risk.

vCenter Simulator is an internal tool developed by two VMware engineers: Zhelong Pan and Kinshuk Govil which allows you to quickly simulate thousands of virtual machines all running in memory while requiring only a minimal amount resources within the VCSA (2vcpu & 8GB memory - default configuration). Building such a tool is definitely not a trivia task, but it is also not the first time we have seen something like this. Awhile back there was project called simDK created by Andrew Kutz that did something similar but only supported reading information from vCenter Server, it did not support any actual operations. vcsim is much more advanced and the really cool thing about vcsim is that even though the inventory is simulated, it actually supports some basic vSphere inventory operations such as create/destroy and power operations. It also supports a hybrid configuration where you can mix both simulated and actual ESXi hosts and virtual machines since it is an actual vCenter Server.

Before we dive into using vcsim, I wanted to go through a few use cases where a tool such as this would be useful:

• Exploring and learning about the vSphere API and the basic inventory hierarchy of vSphere objects
• Environment to develop and create various inventory reporting scripts (vCLI, PowerCLI, etc)
• Developing performance metric gathering tools
• Developing vSphere Web Client plugins and being able visualize large inventory of objects

As you can see, once you start to think about the potential of a such tool, the possibilities can be endless. Having said all of this, no amount of simulation can ever replace actual testing of a real system and any development made using vcsim should be validated against an actual vSphere environment.

To enable vcsim, you will need to add some configuration entries into the vpxd.cfg (vCenter Server configuration file) an example template of the configuration is provided in:

/etc/vmware-vpx/vcsim/model/vcsim.cfg.template

To setup a basic vCenter Simulator, you should deploy a brand new VCSA (you can use an existing VCSA, but the VCDB could potentially get wiped) and go through the basic setup as you would normally. Next, you need to add the following lines at the end of /etc/vmware-vpx/vpxd.cfg between </vpxd> and </config>

Note: Notice the cleardb parameter is false in my example where as the template is set to true. This is very important because if you use the default of "true", you will not be able to view your vSphere inventory using the vSphere Web Client but only the vSphere C# Client as the Inventory Service DB is wiped.

Once you have added the configurations and saved the vpxd.cfg, you will need to restart the vCenter service by running the following command:

service vmware-vpxd restart

Note: A restart of the vmware-vpxd service ONLY works the very FIRST time you add in the vcsim configurations. For any additional changes to the vcsim configuration files, a different method is required to reload the changes, else the vCenter service will fail to start. This is shown in detail further in the article.

Once the vCenter service has restarted, you should now be able to login using either the vSphere Web Client or the vSphere C# Client and you should see a default vSphere inventory that contains a Datacenter, Cluster, several ESXi hosts with Resource Pools along with some powered on and off virtual machines.

Here is a screenshot of logging into the vSphere Web Client:

 

Here is a screenshot logging into the vSphere C# Client:

You might notice that your inventory may not be as large as mines ... oh about 10,000 VM large :) Another cool thing about vcsim is that it has a configurable inventory that you can customize to fit whatever design you wish to have and this can be modified in /etc/vmware-vpx/vcsim/model/initInventory.cfg file. You can tweak the following in the configuration files:

• Datacenter
• Hosts per Datacenter
• VM per Host
• Powered On VM per Host
• Cluster per Datacenter
• Host Per Cluster
• Resource Pool per Cluster
• VM per Resource Pool
• Powered On VM
• vCPU for a VM
• vMEM for a VM

Once you have saved your changes, to reload the new configurations into vcsim, you will need stop the vCenter service and run vpxd -b command to recreate the database and then start the vCenter service. To do so, run the following 3 commands (this is required each time for any changes):

service vmware-vpxd stop
vpxd -b
service vmware-vpxd start

When you log back into your vCenter Server, you now should see the new inventory based on your configurations. In addition to inventory configuration, the vcsim template also points to three other configuration files which I encourage to explore further:

• vcsim/model/metricMetadata.cfg (simulated Performance Metrics, none by default)
• vcsim/model/vcModules.cfg (simulated VC modules such as DRS)
• vcsim/model/OperationDelay.cfg (operations latencies)

Note: You should only be modifying the *.cfg files and not the XML configuration files else you could potentially run into issues.

At this point, you are probably ready to start playing with vcsim and even though this is an internal tool, if you think this is something that is useful to have or have other use cases for, please leave a comment. You never know, this could be a VMware Fling one day if there is enough interest from the community.

Seperating Out the vCenter SSO, vSphere Web Client and vCenter Server Services Using the VCSA

The VCSA 5.1 (vCenter Server Appliance) is provided as single virtual appliance that is pre-installed with all the components needed to run a vCenter Server. These components include vCenter SSO (Single Sign-on), Lookup Service, Inventory Service, vSphere Web Client and the vCenter Server itself. In the Windows installer for vCenter Server 5.1, there is an option to install each individual component on a separate machine. How would you go about doing that for the VCSA as all the components are installed on a single machine?

The answer is actually quite simple, you just need to deploy additional VCSA systems and enable the specific component service on each of the VCSA's. I have already written articles covering some of these use cases such as deploying additional vCenter Servers leveraging a common vCenter SSO Server as well as deploying additional vSphere Web Client Servers. The one particular use case that I have not covered is running just the vCenter SSO Server on the VCSA and with this configuration, there is a minor tweak that is required to get things working correctly.

Disclaimer: This may not be officially supported by VMware, please use at your own risk. 

If you have attempted to configure the VCSA to run just the vCenter SSO service, then you may have seen the following error message "Could not connect to one or more vCenter Server systems" when logging into the vSphere Web Client.

The reason you are seeing this error is due to an invalid configuration found in the vCenter SSO Server and specifically with something called the Lookup Service. The Lookup Service is installed with the vCenter SSO service which can be thought of as a DNS lookup for vSphere components so they can securely find and communicate with each other. Since each VCSA component is registered with the Lookup Service as part of their initial installation and when you only enable the vCenter SSO service, the remainder services will become invalid as they are not running on the same VCSA system. 

Un-Registering Services from Lookup Service:

To fix this problem, we just need to identify the services that should not be registered to the Lookup Service in the vCenter SSO Server and unregister them. To view the list of registered services to a particular Lookup Service endpoint, you can use the /usr/lib/vmware-sso/bin/vi_regtool utility with the listServices option found on the VCSA.

To use the utility, you will need to specify either the IP Address and/or Hostname of the vCenter SSO Server which runs the Lookup Service. Here is an example:

/usr/lib/vmware-sso/bin/vi_regtool listServices https://172.30.0.186:7444/lookupservice/sdk

If the command is successful, you should see a list of service endpoints such as the following:

Service 1
-----------
serviceId=local:7
serviceName=vsphere-client-localhost.localdom-eed72307-2dd2-4069-9650-e78a60b549c7
type=urn:com.vmware.vsphere.client
endpoints={[url=https://172.30.0.185:9443/vsphere-client,protocol=vmomi]}
version=5.1
description=vSphere Web Client at 172.30.0.185
ownerId=vsphere-client-localhost.localdom-eed72307-2dd2-4069-9650-e78a60b549c7@System-Domain
productId=<null>
viSite=local

A default VCSA installation contains the following 6 services:

• vSphere Web Client
• Security Token Service
• VMware Log Browser
• SSO Group Check Service
• vpxd (vCenter Server)
• SSO Administration Service

We will need to identify the serviceId which starts with local:# and unregister the vSphere Web Client, VMware Log Browser and the vpxd service which is not running locally on our vCenter SSO Server. To unregister a service, you will need to create a temporarily file which contains the serviceId and use the unregisterService option with the vi_regtool.

Note: Please make sure you identify the correct serviceId before unregistering, else you may potentially run into issues with your VCSA.

Let's say we want to unregister the service that we showed earlier local:7, we would need to run the following two commands:

echo "local:7" > /tmp/serviceid
/usr/lib/vmware-sso/bin/vi_regtool unregisterService -d https://172.30.0.185:7444/lookupservice/sdk -u root -p vmware -si /tmp/serviceid

The first command will "echo" the serviceId into a temporarily file called /tmp/serviceid and the second command will perform the actual un-registration and you will need to specify the root credentials. You will need to repeat this for the other two services and once you have finished un-registering the three services, you can now log back into the vSphere Web Client and the error message should go away (a service restart is not necessary).

Now that you have some background on how to run a standalone vCenter SSO on the VCSA and the minor tweak that is required, how do we go about automating all of this during deployment? For those of you who know me, know that I would not leave my readers hanging without some scripts to assist with this manual work.

Automating Deployment of vCenter SSO, vSphere Web Client & vCenter Server Component:

The following section will describe how to completely automate the deployment of 3 separate VCSA running vCenter SSO + Lookup Service, vSphere Web Client and vCenter Server + Inventory Service as seen in the diagram above. 

Step 1 - Deploy 3 VCSA 5.1 and configure basic network connectivity. In my example, I have the following setup:

Component	Hostname	IP Address
vCenter SSO + LS	sso.primp-industries.com	172.30.0.185
vSphere Web Client	webclient.primp-industries.com	172.30.0.186
vCenter Server + IS	vcenter.primp-industries.com	172.30.0.187

Step 2 - Configure the vCenter SSO by creating the following shell script called configureVCSASSOStandalone.sh

The only user configuration that is required is to update the SSO_IP_ADDRESS variable in the script to the IP Address of the vCenter SSO Server. You can execute the script via SSH without having to copy the script to the VCSA system, here is an example execution:

We can see from the screenshot above, we automatically look for the 3 services mentioned earlier and unregister it from the vCenter SSO Server running the Lookup Service. You can easily confirm this by re-running the listServices operation with the vi_regtool.

Step 3 - Configure the vSphere Web Client Server and you can use the configureVCSAvSphereWebClientStandalone.sh script noted in this article. The only user configuration that is required is to update the VCENTER_SSO_IPADDRESS variable in the script to point to the IP Address of your vCenter SSO Server. Here is an example execution:

Step 4 - Finally, the last step is to configure the vCenter Server and you can use the configureVCSAExtra.sh script noted in this article. The only user configuration that is required is to update the PRIMARY_VC variable in the script to point to the IP Address of your vCenter SSO Server. Here is an example execution:

Once the vCenter Server has successfully started, then you are now done with seperating out the three components of the vCenter Server using the VCSA. You can confirm additionally by logging back into the vCenter SSO Server and run the listServices and you should now see the IP Address or Hostname of your vSphere Web Client Server and vCenter Server being registered to the Lookup Service from the separate VCSA's. You can now login to the vSphere Web Client server and make sure you specify the full URL which should be https://[hostname-or-ipaddress]:9443/vsphere-client and you should be able to see your vCenter Server.

Note: Steps 3 and 4 can be interchange as the order does not matter, as long as vCenter SSO system is setup first.

Blocking vSphere C# Client Logins

I recently picked up on this neat little tidbit from Mr. Not Supported aka Randy Keener, where you can block a user from logging into the vCenter Server using the vSphere C# Client. Other than playing a prank on your co-workers, you might be wondering is there a use case for this? Surprisingly, this is a request I have heard from a few customers in the past where they would like to block their users from using the vSphere C# Client in favor of leveraging only the vSphere APIs for routine tasks.

Since the vSphere C# Client also uses the vSphere API itself, a user with proper credentials to the vSphere environment can easily download the client from an alternative source and still login. Of course, there are ways of preventing this such as restricting application installation on end users desktop but there is some amount of management overhead of identifying those existing and new users, especially if access is delegated out to other teams.

There is a very simple solution if you choose to block ALL users from using the vSphere C# Client which requires a tiny modification on the vCenter Server itself and it takes effect immediately with no service restarts.

Disclaimer: This is probably not officially supported by VMware, use at your own risk.

Login to your vCenter Server and locate a file called version.txt

Windows: C:\ProgramData\VMware\VMware VirtualCenter\docRoot\client
VCSA: /etc/vmware-vpx/docRoot/client

There is parameter called exactVersion which will be set to current supported version of the vSphere C# Client which should also match the version of your vCenter Server. You just need to change this to some other value that you know will not exist in your environment such as 9.0.0. Once you have made this change, now when a user tries to connect and there is a miss-match in the version, the vCenter Server will provide you with a download to the vSphere C# Client located on the server as it normally would if you did not have the latest client.

What the user will find out shortly, is that this will continue in an infinite loop even after installing the proper vSphere C# Client. The reason for this is that the number in version.txt will never match the vSphere C# Client and vCenter Server will just continue serving the installer in an infinite loop. I also looked into this trick for a standalone ESXi host and you can do the same by editing a file called clients.xml which is located in /usr/lib/vmware/hostd/docroot/client and users will not be able to login to the ESXi host using the vSphere C# Client.

Now, even though this prevents users from logging into the vSphere C# Client, users will still be able to connect using the vSphere API which includes the use of vCLI/ESXCLI, PowerCLI, vCO, SDKs, etc. and the use of the vSphere Web Client for either vSphere 5.0 or 5.1 will continue to work. Ideally, it would be nice to be able to control this access on a per user/group basis and perhaps even specify how a user can connect whether that is through the use of the APIs or UI only. Is this even useful to have at all? Would love to hear your comments.

For now, if you want users to get familiar with the new vSphere Web Client 5.1 ... this is one way of "encouraging" them ;)

Running ESXi 5.0 & 5.1 on 2012 Mac Mini 6,2

If you recently purchased the new 2012 Apple Mac Mini 6,2 which was just released not too long ago and tried to install either ESXi 5.0 or 5.1, you probably noticed a PSOD (Pink/Purple Screen of Death) during the installation. This is currently a known issue and there is an extensive VMTN thread (9,300+ views) about this problem which also includes a fix through a collaboration between VMTN community user zer010gic and VMware Engineer dariusd. Even though the Apple Mac Mini is not an officially supported hardware platform for running ESXi, it is great to see VMware engineers going out their way and trying to help the VMware community find a solution as well as providing an "unofficial" fix in this case.

I would also like to point out that this issue only applies to the new 2012 Apple Mac Mini, for previous models such as the Apple Mac Mini 5,1 or 5,3 you can install ESXi 5.0 or 5.1 without any issues. For more details, please refer to the instructions in this blog post.

Disclaimer: The Apple Mac Mini is not officially supported by VMware. The only supported platform for ESXi 5.0 for Apple hardware is the Apple XServe 3,1 and for ESXi 5.1 is the Apple Mac Pro, which you can get more details here. 

Before jumping into the solution, if you think VMware should support the Apple Mac Mini for running ESXi, please provide feedback to VMware by submitting a Feature Request. The more feedback that VMware receives from customers along with business justifications, the better our product management team can prioritize features that are most important to our customers.

Here are the current problem/solutions when trying to install on the new Mac Mini:

Problem: PSOD during ESXi 5.0 or 5.1 installation.
Solution: Add iovDisableIR=true to the kernel option before attempting installation. When you are asked to reboot, be prepared to enter iovDisableIR=true again (SHIFT+O) which is required to get ESXi to boot after installation. Once the system has booted up, go ahead and run "esxcli system settings kernel set -s iovDisableIR -v true" in the ESXi Shell to persist the kernel setting. This is a "temp" workaround while PSOD is being investigated.

Problem: Unable to install new OSX Server on a VM or power on existing OSX Server VMs.
Solution: There appears to be a significant change in Apple's SMC (System Management Controller) device in the newer models that prevents the Apple SMC VMkernel driver from properly loading. A tempoary fix was provided to zer010gic to create a custom ISO until the fix is integrated into a future release.

Note: There may be other minor/unconfirmed issues listed on the VMTN thread, but for basic ESXi installation/usage + OSX Server VM creation/installation, the above solutions should be sufficent.

Instead of having everyone walk through the process of creating a custom ESXi ISO which includes the two fixes mentioned above as well as the bundling the updated tg3 Broadcom network drivers for network connectivity, zer010gic has generously created and is hosting ESXi 5.1 ISOs for users to download and use. It contains some work that I have been doing with zer010gic to create an ESXi 5.1 ISO that does not require any manual intervention outside of the normal ESXi installation. I recently completed the rest of this work which is based off of the oriignal ISO that zer010gic has shared on the VMTN community (unfortunately I have not been able to get a hold of him to provide him with the necessary bits and I have decided to post a modified ISO).

Here is a step by step instruction for zer010gic ESXi 5.1 ISO

Step 1 - Download zer010gic ESXi-5.1-MacMini-SMC-6-2.iso.

Step 2 - Transfer ISO to either USB key or CD-ROM

Step 3 - Perform ESXi installation as you would, but when you get to the very last step prior to rebooting, be ready for some typing when the host boots back up (this is important else you will get a PSOD)

Step 4 - When ESXi starts to boot up, hit SHIFT+O which will allow you to add additional kernel boot option. Add the following text the bootUUID (remember to add a space first)

iovDisableIR=true

This step is required to ensure your ESXi boots up properly for the first time so you can permanently enable this kernel option using ESXCLI which will then persist this upon sub-sequent reboots.

Step 5 - Login to ESXi Shell (you may need to enable it first) and run the following ESXCLI command:

esxcli system settings kernel set -s iovDisableIR -v true

Once this is set, you no longer have to do this again. If you prefer not to go through these manual steps, please refer to the section below for a modified ESXi 5.1 ISO which automates all this for you.

Here is my modified ESXi 5.1 ISO which does not require any additional user intervention

Step 1 - Download my ESXi-5.1-MacMini-SMC-BOOT-FIX-6-2.iso

Step 2 - Transfer ISO to either USB key or CD-ROM

Step 3 - Go through normal ESXi install and enjoy

Note: For details on how I automated the kernel setting setting, take a look at the very end.

So if you are looking to refresh your home lab, you just may want to consider using the new Apple Mac Minis, especially with small form factor footprint :)

Note: A couple of users mentioned it took a bit of time to boot up, specifically when usbarbitrator module is being loaded. I noticed this too and it took quite a bit of time, probably 5-6 minutes. If you do not plan on any USB pass-through from the Mac Mini to your guestOSes, you can actually disable this service which should help speed the bootup. If you wish to disable usbarbitrator, run the following command:

chkconfig usbarbitrator stop

ESXi ISO Customization Details

 

If you take a look at the steps required to install the ISO provided by zer010gic, most of the heavy work has already been done for you. The only "manual" part that is required from the user is to enter a kernel option during the first boot and then run an ESXCLI command to persist this kernel setting which will prevent Mac Mini from PSODing. Removing these these manual steps is actually harder than it looks because of when you need to actually perform the changes. After much trial and error, I came up with the following script below (it's not the cleanest, but it works). 

 

Basically the script is loaded from custom.tgz and executed before the installation begins and it generates a script stored in /tmp/customboot.sh which will look for the boot.cfg configuration file stored in the primary bootbank. This is where we insert the iovDisableIR=true parameter so the user is not required to do this after the first boot up. The challenge with this is the boot.cfg does not exists until after the installation has completed, so what I ended up doing was insert a command into /usr/lib/vmware/weasel/process_end.py which is part of the weasel installer for ESXi and is the very last script that is called when a user hits reboot. The command points back to the /tmp/customboot.sh which will perform the insert into boot.cfg right before rebooting. To automatically take care of the ESXCLI configuration, I added the ESXCLI command to /etc/rc.local.d/local.sh which will automatically run after all init scripts have executed. Then finally, I need to clean up local.sh since I only need that that run once which is handled by another script that is also created and stored in /etc/init.d/customcleanup which will just clean up local.sh file as well as delete itself. Simple right? ;)

 

Note: There is probably a more optimal way of doing this, probably using one of the weasel installer scripts and just set the boot.cfg option and then clean up with an init script, but I decided to leverage some of my earlier work for Disabling LUN Duringn ESXi Installation

Here is the script within the custom.tgz file: