virtuallyGhetto

vSphere 5

w00t! VMware Tools for Nested ESXi!

by 40 Comments

I have been working with Nested ESXi since it original inception and this technology has greatly benefited me and the entire VMware community, especially when it comes to learning about VMware software and being able to easily prototype something before installing it on actual hardware. However, one thing that I felt that has been missing for awhile now is the ability to run an instance of VMware Tools within a Nested ESXi VM. I have personally been asking for this feature for a couple of years and I know many in the VMware community have expressed interests as well.

I am super excited to announce that VMware has just released a new Fling that provides you with a VIB that you can install VMware Tools inside a Nested ESXi host. I originally showed a demo of this at VMworld Barcelona in my vBrownBag Tech Talk and as I mentioned we would be releasing this as a VMware Fling very soon. So here it is!

UPDATE (08/20/15) - An updated version of VMware Tools for Nested ESXi was just published, make sure to download latest version and you can find more details here.

Requirements:

  • Nested ESXi running 5.0, 5.1 or 5.5 

Installation:

To install the VIB, you simply just need to download it and upload the VIB it to your Nested ESXi datastore and then run the following commands:

esxcli system maintenanceMode set -e true
esxcli software vib install -v /vmfs/volumes/[VMFS-VOLUME-NAME]/esx-tools-for-esxi-9.7.0-0.0.00000.i386.vib -f
esxcli system shutdown reboot -r "Installed VMware Tools"

You can also install the VIB directly from VMware.com if you have direct or proxy internet connectivity from your ESXi host by running the following commands:

esxcli network firewall ruleset set -e true -r httpClient
esxcli software vib install -v http://download3.vmware.com/software/vmw-tools/esxi_tools_for_guests/esx-tools-for-esxi-9.7.0-0.0.00000.i386.vib -f

Once the VIB has been successfully installed, you will need to reboot the host for the changes to take effect. To verify, you can now login to either your vSphere Web/C# Client and you should now see the status for VMware Tools for your Nested ESXi host showing green and the IP Address of the Nested ESXi host should be displayed.

So why would you want to do this? Well, there’s a couple of reasons actually. The first one is pretty basic, which is when I need to reboot or shutdown a Nested ESXi VM, instead of having to jump into the VM console or SSH into ESXi host, I could just right click in the vSphere Web/C# Client and just say shutdown or reboot. I also tend to do all sorts of craziness in my lab (I’m sure this is an understatement for folks that know me) and may often break networking connectivity to my Nested ESXi VM. In vSphere 5.0, we introduced the Guest Operations API (formally known as VIX API) which is now part of the vSphere API. This API is actually quite handy as it allows you to perform guest operations within the VM without needing network connectivity as it relies on the fact that VMware Tools is running (pretty cool stuff!).

Here is a screenshot demonstrating the executing of vmkfstools through the Guest Operations API to one of my Nested ESXi VM:

A couple of things to note:

  • If you install VMware Tools on Nested ESXi VM, you will NOT be able to just right click in the UI and say install/upgrade
  • If you wish to integrate this into you ESXi image, you can take a look at a community tool  called ESXi-Customizer created by Andreas Peetz which I have used in the past and works great. Image Builder does not support raw VIBs, only zip files which may need to contain additional metadata information. If you want to create an offline bundle instead to then use Image Builder to create your custom ISO, Andreas has a new tool you can take a look at here.

Finally, if you have any feedback (likes/dis-likes), thanks, comments please head over to the VMware's Fling page for VMware Tools for Nested ESXi and leave a comment. I am sure the Jim Mattson the engineer who built this Fling would greatly appreciate any feedback you may have.

Did you know that VMware Host Profile is extensible by 3rd Parties?

by 1 Comment

Managing ESXi host configurations can be challenging and the potential risk for configuration drift between the running environment and the set of configuration scripts or worse, manual configuration is quite high. On top of that, how do you ensure proper compliance of all your ESXi host configurations in your environment and easily prove that in an internal or security audit?

This is where VMware Host Profile can help which allows administrators to capture the running configurations of an ESXi host and automatically creating a template (Host Profile) that can then be applied across new or existing ESXi hosts. By leveraging Host Profile, administrators can ensure that all their ESXi host configurations are always consistent and configuration drifts can easily be prevented through automatic compliance checks.

Recently, while searching for something on VMware's HCL website, I accidentally stumbled onto what appears to be 3rd party Host Profiles? There were only two listed, one from Brocade for managing and configuring Brocade storage adapters and the other from Dell for managing and configuring Dell's EqualLogic MEM (Multipathing Extension Module). I was actually quite surprise to learn about these custom 3rd party Host Profiles. In doing a bit of digging and research it turns out that VMware Host Profile are in fact extensible by design, which was something new to me.

Note: For a technical overview of Host Profile, you can take a look at this whitepaper here. 

Host Profile Architecture

Host Profile was first introduced with the release of vSphere 4.1 and the brain of the system is known as the Host Profile Engine which was part of the vCenter Server. In vSphere 5.0, Host Profile was re-architected and the Host Profile Engine was moved into the ESXi host which allowed for Host Profile Plugins to be added to an ESXi Image and expose new Host Profiles through the Host Profile Engine.

A Host Profile is actually a hierarchical composition of multiple sub-profiles and policies. Each policy defines a set of parameters that a user can select from and apply to an ESXi host. For instance, the default VMware Host Profile is composed up of 12 individual sub-profiles: authentication, datetime, firewall, memory, network, option, security, service, storage, userAccount and userGroupAccount.

With this new re-architecutre, Host Profile can be extended by 3rd party partners/vendors to create custom Host Profile Plugins to expose vendor specific hardware or software configurations and made available through a common Host Profile API/UI for customers to consume.

Host Profile Extensibility Options

To build a Host Profile Plugin, you will need to use the Host Profile SDK which is only available as part of VMware TAP (Technology Alliance Partner) Program. A Host Profile Plugin basically wraps the actual configuration work and can be backed by one of three ways:

  1. CIM Provider using the CIM SDK
  2. ESXCLI plugins
  3. Userworld binaries

As you can see, creating a Host Profile Plugin is quite flexible and can be exposed through several mechanisms. The most shocking discovery that I found was the lack of 3rd party vendor Host Profiles that exists today, especially from the big server hardware vendors. Coming from a Systems Administrator background, I would loved to have been able to configure and manage my server's firmware, BIOS, out-of-band management (iLO/DRAC), etc. through either a custom ESXCLI plugin or Host Profile Plugin. This would really benefit customers from having to manage configurations using multiple tools and allowing them centralize their management including compliance capabilities all from a single interface.

Hopefully this was an educational post for everyone and if you are a customer and would like to see certain functionality exposed by our 3rd party partners, feel free to leave a message and perhaps one of them may consider adding a custom Host Profile Plugin 🙂

Whitepaper: Migrating From VIX API to the vSphere Guest Operations API

by 7 Comments

The VMware VIX API in my opinion is still one of the most powerful and undervalued API's that is available to customers and partners for Virtual Machine guest operating system Automation. The VIX API allows you to perform guest operations such as starting/stopping an application, file/directory manipulation, uploading/downloading files all within the guest operating system without requiring any network connectivity to the Virtual Machine. This is all made possible through the use of VMware Tools that is running inside of the Virtual Machine and operations are only performed after a user of the guestOS is properly authenticated.

Guest Operations using vSphere API

The use cases for such an API are endless:

  • Network reconfiguration (Re-IP for DR or miss-configuration)
  • Operating System configurations
  • Application configurations or deployments (example of this)
  • Backup/Restore for individual files
  • Downloading log files for troubleshooting
  • The list goes on ....

The VIX API was first introduced as a separate client API supporting VMware's hosted products such as VMware Fusion, Workstation and Player and later supported VMware vSphere. The API was quite popular for the hosted products and with the release of vSphere 5.0, the VIX API was finally integrated into the vSphere API to provide a single API that could manage all aspects of vSphere as well as these new guest operations APIs for your Virtual Machines. With this integration, these new APIs are now known as the vSphere Guest Operations API.

If you are familiar with the VIX API and would like to move or migrate to using the new Guest Operations API within vSphere, there is a really useful whitepaper that I recently came across called Transporting VIX Guest Operations to the vSphere API that provides a nice mapping of the API methods between the VIX and new vSphere Guest Operations API. The whitepaper also includes various code samples using Java, PowerCLI cmdlets and vSphere SDK for Perl to demonstrate the new Guest Operations APIs.

I think every vSphere administrator or developer should be familiar with the capabilities of the VIX and Guest Operations API and how it can help them further automate and manage your guestOSes and the applications that run inside of them.

Additional Resources:

VMware Tools For Apple Mac OS X Guests?

by 3 Comments

With the release of vSphere 5, virtualizing Apple Mac OS X as a guest OS was possible and fully supported from VMware. To do so, you would need to be run ESXi on Apple hardware either the now deprecated Apple XServe 3.1 or an Apple Mac Pro. A comment that came up yesterday on Twitter was that VMware Tools did not exists for Mac OS X guests and this would make it difficult to manage Mac OS X guests on vSphere. I guess it may not be that well known or just an assumption, but VMware Tools does in fact exists for Mac OS X guests and it is also documented in the VMware Tools installation guide.

It is still amazing to me to see the number of guest OSes the vSphere platform supports and perhaps virtualizing Mac OS X is still relatively new for folks and hence the initial assumption about VMware Tools not being available. In any case, I thought I take you through a few screenshots of installing VMware Tools for a Mac OS X 10.7 guests running on my Apple Mac Mini.

In the screenshot below, we can see that VMware Tools is not detected in the guest OS and we have a option to install VMware Tools, so we go ahead and click on that.

This will mount the darwin.iso to the VM from the vmimages directory of the ESXi host and you can proceed with the VMware Tools installation.

Upon finishing the installation, you will be asked to reboot the guest OS and now when we take a look at the VM summary view, we can see VMware Tools is now running in our Mac OS X guests.

Note: For instructions on installing Apple Mac OS X as a guest OS on vSphere, please refer to this tech note.

Running ESXi 5.0 & 5.1 on 2012 Mac Mini 6,2

by 59 Comments

If you recently purchased the new 2012 Apple Mac Mini 6,2 which was just released not too long ago and tried to install either ESXi 5.0 or 5.1, you probably noticed a PSOD (Pink/Purple Screen of Death) during the installation. This is currently a known issue and there is an extensive VMTN thread (9,300+ views) about this problem which also includes a fix through a collaboration between VMTN community user zer010gic and VMware Engineer dariusd. Even though the Apple Mac Mini is not an officially supported hardware platform for running ESXi, it is great to see VMware engineers going out their way and trying to help the VMware community find a solution as well as providing an "unofficial" fix in this case.

I would also like to point out that this issue only applies to the new 2012 Apple Mac Mini, for previous models such as the Apple Mac Mini 5,1 or 5,3 you can install ESXi 5.0 or 5.1 without any issues. For more details, please refer to the instructions in this blog post.

Disclaimer: The Apple Mac Mini is not officially supported by VMware. The only supported platform for ESXi 5.0 for Apple hardware is the Apple XServe 3,1 and for ESXi 5.1 is the Apple Mac Pro, which you can get more details here.

Before jumping into the solution, if you think VMware should support the Apple Mac Mini for running ESXi, please provide feedback to VMware by submitting a Feature Request. The more feedback that VMware receives from customers along with business justifications, the better our product management team can prioritize features that are most important to our customers.

Here are the current problem/solutions when trying to install on the new Mac Mini:

Problem: PSOD during ESXi 5.0 or 5.1 installation.
Solution: Add iovDisableIR=true to the kernel option before attempting installation. When you are asked to reboot, be prepared to enter iovDisableIR=true again (SHIFT+O) which is required to get ESXi to boot after installation. Once the system has booted up, go ahead and run "esxcli system settings kernel set -s iovDisableIR -v true" in the ESXi Shell to persist the kernel setting. This is a "temp" workaround while PSOD is being investigated.

Problem: Unable to install new OSX Server on a VM or power on existing OSX Server VMs.
Solution: There appears to be a significant change in Apple's SMC (System Management Controller) device in the newer models that prevents the Apple SMC VMkernel driver from properly loading. A tempoary fix was provided to zer010gic to create a custom ISO until the fix is integrated into a future release.

Note: There may be other minor/unconfirmed issues listed on the VMTN thread, but for basic ESXi installation/usage + OSX Server VM creation/installation, the above solutions should be sufficent.

Instead of having everyone walk through the process of creating a custom ESXi ISO which includes the two fixes mentioned above as well as the bundling the updated tg3 Broadcom network drivers for network connectivity, zer010gic has generously created and is hosting ESXi 5.1 ISOs for users to download and use. It contains some work that I have been doing with zer010gic to create an ESXi 5.1 ISO that does not require any manual intervention outside of the normal ESXi installation. I recently completed the rest of this work which is based off of the oriignal ISO that zer010gic has shared on the VMTN community (unfortunately I have not been able to get a hold of him to provide him with the necessary bits and I have decided to post a modified ISO).

Here is a step by step instruction for zer010gic ESXi 5.1 ISO

Step 1 - Download zer010gic ESXi-5.1-MacMini-SMC-6-2.iso.

Step 2 - Transfer ISO to either USB key or CD-ROM

Step 3 - Perform ESXi installation as you would, but when you get to the very last step prior to rebooting, be ready for some typing when the host boots back up (this is important else you will get a PSOD)

Step 4 - When ESXi starts to boot up, hit SHIFT+O which will allow you to add additional kernel boot option. Add the following text the bootUUID (remember to add a space first)

iovDisableIR=true

This step is required to ensure your ESXi boots up properly for the first time so you can permanently enable this kernel option using ESXCLI which will then persist this upon sub-sequent reboots.

Step 5 - Login to ESXi Shell (you may need to enable it first) and run the following ESXCLI command:

esxcli system settings kernel set -s iovDisableIR -v true

Once this is set, you no longer have to do this again. If you prefer not to go through these manual steps, please refer to the section below for a modified ESXi 5.1 ISO which automates all this for you.

Here is my modified ESXi 5.1 ISO which does not require any additional user intervention

Step 1 - Download my ESXi-5.1-MacMini-SMC-BOOT-FIX-6-2.iso

Step 2 - Transfer ISO to either USB key or CD-ROM

Step 3 - Go through normal ESXi install and enjoy

Note: For details on how I automated the kernel setting setting, take a look at the very end.

So if you are looking to refresh your home lab, you just may want to consider using the new Apple Mac Minis, especially with small form factor footprint 🙂

Note: A couple of users mentioned it took a bit of time to boot up, specifically when usbarbitrator module is being loaded. I noticed this too and it took quite a bit of time, probably 5-6 minutes. If you do not plan on any USB pass-through from the Mac Mini to your guestOSes, you can actually disable this service which should help speed the bootup. If you wish to disable usbarbitrator, run the following command:

chkconfig usbarbitrator stop

ESXi ISO Customization Details

If you take a look at the steps required to install the ISO provided by zer010gic, most of the heavy work has already been done for you. The only "manual" part that is required from the user is to enter a kernel option during the first boot and then run an ESXCLI command to persist this kernel setting which will prevent Mac Mini from PSODing. Removing these these manual steps is actually harder than it looks because of when you need to actually perform the changes. After much trial and error, I came up with the following script below (it's not the cleanest, but it works).

Basically the script is loaded from custom.tgz and executed before the installation begins and it generates a script stored in /tmp/customboot.sh which will look for the boot.cfg configuration file stored in the primary bootbank. This is where we insert the iovDisableIR=true parameter so the user is not required to do this after the first boot up. The challenge with this is the boot.cfg does not exists until after the installation has completed, so what I ended up doing was insert a command into /usr/lib/vmware/weasel/process_end.py which is part of the weasel installer for ESXi and is the very last script that is called when a user hits reboot. The command points back to the /tmp/customboot.sh which will perform the insert into boot.cfg right before rebooting. To automatically take care of the ESXCLI configuration, I added the ESXCLI command to /etc/rc.local.d/local.sh which will automatically run after all init scripts have executed. Then finally, I need to clean up local.sh since I only need that that run once which is handled by another script that is also created and stored in /etc/init.d/customcleanup which will just clean up local.sh file as well as delete itself. Simple right? 😉

Note: There is probably a more optimal way of doing this, probably using one of the weasel installer scripts and just set the boot.cfg option and then clean up with an init script, but I decided to leverage some of my earlier work for Disabling LUN Duringn ESXi Installation

Here is the script within the custom.tgz file: