esxi

How To Backup & Restore Free ESXi Host Configuration

02/15/2013 by William Lam 22 Comments

ESXi host configurations can easily be backed up and restored using either the vCLI's vicfg-cfgbackup or PowerCLI's Get-VMHostFirmware cmdlet. These commands along with others that perform "write" operations are only supported when you have a (paid) licensed version of ESXi. If you are using free ESXi, the remote commands are only available for "read-only" operations. For more details, please refer to this article here.

Note: In my personal opinion, it is much quicker and more efficient to re-install ESXi and apply your configurations using either a scripted deployment such as kickstart or a combination along with post configuration scripts. Re-installs become extremely trivial when you centralize your ESXi host configurations, even for small setups.

Having said that, if you are running free ESXi in a small shop or in a home lab and wish to backup your ESXi host configurations, you can still do so by leveraging a neat little tool called vim-cmd found within the ESXi Shell. There is a section under hostsvc/firmware which manages the ESXi host configuration which also uses the same vSphere APIs that both vicfg-cfgbackup & Get-VMHostFirmeware command uses.

Under this section of vim-cmd, there are four commands:

backup_config
reset_config
restore_config
sync_config

Prior to actually backing up your ESXi host configuration, run the following command which will flush the ESXi configuration changes:

vim-cmd hostsvc/firmware/sync_config

To backup the ESXi host configurations, run the following command which will generate a file that will be automatically stored in /scratch/downloads and can also be downloaded from a web browser using the URL shown from the output:

vim-cmd hostsvc/firmware/backup_config

Before restoring your ESXi host configurations, you will need to ensure the file is renamed to configBundle.tgz and stored under /tmp directory. You will also need to ensure the ESXi host is placed in maintenance mode by running the following command:

vim-cmd hostsvc/maintenance_mode_enter

To restore the ESXi host configurations, run the following command and specify the backup configuration file which should reside in /tmp/configBundle.tgz:

vim-cmd hostsvc/firmware/restore_config /tmp/configBundle.tgz

Note: Upon completing the restore, it will automatically reboot your ESXi host.

Here is a screenshot using the above commands to backup and then restore ESXi host:

Note: You can not restore an ESXi host using a configuration file backed up from a different host.

Enable Auto Startup After Power Failure For Apple Mac Mini

02/11/2013 by William Lam 34 Comments

I recently came across a very useful tidbit after receiving several inquires asking how to configure an Apple Mac Mini to automatically startup after a power failure. This is extremely useful for situations where power is eventually restored and you are not physically around to press the power button. Automatically starting up after a power failure is not a new feature of the Mac Mini and it actually exists on most modern day systems and can be configured using a variety of tools.

The challenge arises when you are running ESXi, how do you go about enabling this functionality in ESXi itself? Well the answer is actually quite simple, you can enable this outside of ESXi. Normally to enable this feature you can either run a setpci command on UNIX/Linux system or configure the energy saver settings in OS X. Several folks from the VMTN community such as zippytiff and twuhabro have already confirmed having success using the latter option when booting OS X off a USB or SD card to modify the energy saving settings.

I finally got a chance to look into this a bit more for myself and with a bit of research, I found several other methods which also works and may potentially be easier.

Note: I have heard that historically the auto startup flag has not persisted in older Apple hardware, but for the new Mac Mini 5,3 and 6,2, they seem to be persisting without any issue from my testing. YMMV depending on your hardware and/or firmware.

Method 1 (Configure in OS X)

If you already have OS X install on your Mac Mini, then you just need open up the System Preferences and enable auto startup under the Energy Saver section. Once that has been enabled, you can then perform your ESXi installation.

If you already have ESXi installed, then you can use either Method 2 or 3.

Method 2 (Configure using bootable Ubuntu on USB)

We can create a bootable Ubuntu image running off of a USB device (minimal footprint) and run the following setpci command to enable auto startup:

setpci -s 0:1f.0 0xa4.b=0

If you are interested in the gory details on the above command, please refer to this great article which breaks it all down for you. After you have created your Ubuntu image using the instructions in the above link, you can boot off of the USB device (Make sure to hold down ALT/OPT key so you can select to boot off of the USB device). Once Ubuntu has booted up, you will right click on the purple icon on the upper left hand corner and type in terminal.

You will then launch the terminal application and type the following to change over to root user.

sudo su -

Finally, you will enter the above setpci command which will enable the auto startup. At this point, you can type reboot and remove the USB device.

Method 3 (Configure using bootable OS X on USB)

Another method is to create a bootable OS X image running off of a USB device and change the power management settings by using the pmset utility. To enable auto startup, run the following command in the terminal:

pmset autorestart 1

You will need the original OS X installer and if you are using either Lion or Mountain Lion, you can use Lion DiskMaker to help you create the bootable USB image (Make sure to hold down ALT/OPT key so you can select to boot off of the USB device). Once the OS X installer boots up, at the top select Utilities and click on the Terminal application.

Go ahead and run the above command to enable auto start and then type reboot and remove the USB device.

The last thing to do of course is to actually test this out on your Mac Mini. Go ahead and let the system boot up and then yank the power cord and then give it a few seconds before plugging it back in. You should see the Mac Mini automatically power back on after you plug the power back in.

References:

How to Send vCenter Alarm Notifications to SMS & Other Online Services Using IFTTT

02/02/2013 by William Lam 2 Comments

I just discovered a pretty cool free online service called IFTTT (If That Then This) which allows you to easily create what is known as a recipe that is composed of trigger and an action to send a notification. This can be thought of like a vCenter Alarm with a trigger and action, but in IFTTT's case, a trigger and action can be almost anything such as an email, facebook event, twitter, etc.

I stumbled onto IFTTT while going through some of my unread blog feeds this morning since I could not go back to sleep after randomly waking up at 4am in morning. I was reading an article from Matt Cowger about something called a Pebble which I had not heard of before and it peaked my interest. In reading about Pebble (pretty slick actually), I learned about IFTTT which is one of the notification systems Pebble supports. I decided to give IFTTT a try and to see how easy it would be to setup SMS notifications for a vCenter Server alarm.

Step 1 - Sign up for IFTTT account, you need to ensure the email account that you use to register is the same account that will be used to send the trigger.

Step 2 - Create a new Recipe using email as the trigger and SMS for the action. The process is very straight forward, just follow the wizard and at the end you will enter your phone number for SMS notification and confirm with a code.

Step 3 - Create a vCenter Alarm for the event you wish to trigger off of and set the email address to send to *protected email* as noted in the Recipe.

Step 4 - In my test, I created an alarm for a Powered Off event and went ahead and powered off a VM to generate the alarm.

Step 5 - IFTTT checks the triggers every 15minutes. If you do not wish to wait 15minutes you can force a check by clicking on Check Now button in the Recipe.

Step 6 - If everything was setup correctly, you should have received a text message with the details of the vCenter alarm. In my Recipe, I configured it to only send the Subject which contains everything you would need to know about the vCenter alarm, at least for you to decide whether or not you want to investigate the issue.

The possibilities are pretty much endless in terms of the triggers and actions that you could create with IFTTT, I even created one for my ghettoVCB backup notification. The only downside that I noticed while giving IFTTT a try is that the trigger check is every 15minutes which could be a bit long for things requiring immediate attention, but I also read that there are certain Recipes that supports a "Quick Trigger" which would then execute immediately upon receiving. I think for a free service this is very cool and much easier than setting up your own SMS system. I would recommend giving IFTTT a try and see what cool Recipes you can build and integrate with your VMware or other environments.

I hope my body will let me crash and get some sleep now ...

Detecting A Duplicate IP Address For Your ESXi Hosts Using a vCenter Alarm

01/28/2013 by William Lam 5 Comments

The motivation for this article was a tweet I noticed from Duncan Epping this morning. Per Duncan's tweet, it looks like he may have accidentally assigned an IP Address to one of his virtual machines which was already being used by an existing ESXi host causing a duplicate IP Address error. We probably have all experienced this once in our lives and it can be quite difficult and frustrating to troubleshoot. Similar to a Windows OS, ESXi can also detect a duplicate IP Addresses but instead of a notification window, it is just logged in the VMkernel logs which looks like the following:

2013-01-21T15:52:35.989Z cpu1:2049)Tcpip_Vmk: 112: arp: 00:50:56:bd:3b:2b is using my IP address 172.30.0.213 on vmk1!

The biggest challenge of course is to identify which ESXi host actually has a conflict and then taking a look at the logs to find the offending MAC Address and shutting them down yourself or with the help of a Network Administrator. Wouldn't it be great if we had an alarm to automatically notify us when a duplicate IP Address is detected? Well I am glad you asked and the answer is YES! 🙂

In addition to logging to the VMkernel logs, ESXi also logs this "observation" in /var/log/vobd.log which stands for the VMkernel Observation. These "observations" can provide critical identifying information in case of an error and is usually used during troubleshooting. In our case, we are seeing an intermittent network connectivity to our ESXi host which is in result of a duplicate IP Address. The really neat thing about these VOBs is that you can create vCenter Alarms when a specific VOB has been detected. I have shown an example of this before in my Detecting ESXi Remote Syslog Connection Error Using a vCenter Alarm article.

We can do exactly the same for detecting a duplicate IP Address for an ESXi host. The first thing we need to do is identify the VOB ID by looking in /var/log/vobd.log:

2013-01-21T15:02:07.513Z: [netCorrelator] 917174784727us: [esx.problem.net.vmknic.ip.duplicate] Duplicate IP address detected for 172.30.0.83 on interface vmk0, current owner being 00:50:56:bd:3b:2b

We can see the VOB ID for this is esx.problem.net.vmknic.ip.duplicate and this will be used in our vCenter Alarm trigger.

Step 1 - Create a new Alarm and specify a name, the Monitor type will be Hosts and Monitor For will be for a specific event:

Step 2 - Copy the VOB ID that we have identified from above and specify that as our alarm Trigger:

Step 3 - If you wish to receive an email notification or send an SNMP trap go ahead and configure additional actions, else just click next which will just display a vCenter Server alert in the UI.

Now that our alarm has been created, we will want to give this a test drive .... who can we ask? Well it just happens that I have a new user in my environment and I provisioned him a new VM which is already connected to the network. Let's hope he does not try to change the IP Address (because this never happens, right?)

After the user statically assigns the IP Address of an existing ESXi host in the VM, we should see our new alarm trigger in vCenter.

As you can see, we have quickly identified the ESXi host that is impacted and we can then login to DCUI via the console to take a look at the logs to find the offending MAC Address. Hopefully duplicate IP Addresses is not a common problem in your environment but it does happen from time to time and having an alarm to help you quickly narrow down the culprit can be quite useful.

What Are the Shadow of vmnic in ESXTOP?

01/04/2013 by William Lam 6 Comments

In ESXi 5.1, you might have noticed something new called Shadow of vmnic under the USED-BY column in the Network view of ESXTOP.

I initially heard about this from a few followers on twitter and I was curious myself since I could not find any documentation regarding this topic. It took a bit of digging but it turns out these shadow vmnics is actually related to the new VDS Health Check feature released in vSphere 5.1. A shadow port is created automatically for every uplink in your ESXi host and is used for transmitting and receiving health check related packets for each uplink. In ESXTOP, you can monitor the statistics for these shadow ports which can be used to debug/troubleshoot the network health check feature and this is why they are present.

One thing to note, these shadow ports are created regardless of whether or not your ESXi host is connected to a VDS or if you have the network health check features enabled. This is by design and there are four VMkernel modules that are responsible for the network health check feature:

vlanmtucheck
teamcheck
heartbeat
healthchk

Disclaimer: Do not modify or disable any of the above VMkernel modules else you can potentially disable network connectivity to your ESXi host (trust me, I know).

After some investigation and testing in my lab, I found that I could unload the above VMkernel modules and the shadow vmnics entries would no longer appear in ESXTOP. To do this, you will need an ESXi 5.1 host which is not running any virtual machines and you will need to run the following commands in this exact order (as there are module dependencies):

vmkload_mod -u vlanmtucheck
vmkload_mod -u teamcheck
vmkload_mod -u heartbeat
vmkload_mod -u healthchk

Once you have successfully unloaded all four VMkernel modules, if you open up ESXTOP, you should no longer see the shadow vmnics. To restore the shadow vmnics, you can either reboot (since the unload is not persistent) OR you can run the following commands in this exact order:

vmkload_mod heartbeat
vmkload_mod teamcheck
vmkload_mod vlanmtucheck

Note: By loading the heartbeat VMkernel module, it also loads the healthchk module.

Configure Apple Mac Mini to Default Boot ESXi

01/02/2013 by William Lam 13 Comments

If you are running ESXi on an Apple Mac Mini and it is installed on a USB key, you probably have noticed that the Mac Mini tries to boot from disk by default and instead of using the USB device. This means when you reboot your ESXi host each time, you will need to hold down the "ALT/OPTION" key which will present you with a boot menu to select the device you wish to boot from.

This can be quite annoying if you have a headless setup for your Mac Mini and you just want it to automatically boot off of the right device containing your ESXi installation. To fix this, you can configure the default boot device which can be done by first selecting the device you wish to boot off of as shown in the screenshot above. Next, hold down on the "CONTROL" key which will turn the straight arrow into circular arrow icon as shown in the screenshot below.

Now you just need to either hit enter or if you have a mouse, click on the circular arrow icon and this will configure the default boot device the Apple Mac Mini will use going forward. It is that simple! If you want to boot off of another device after configuring the default boot device, you can still do so by holding down "ALT/OPTION" key while the Mac Mini is still booting up.

Credit goes to this site for solution.