virtuallyGhetto

esxi

How to Send vCenter Alarm Notifications to SMS & Other Online Services Using IFTTT

by 2 Comments

I just discovered a pretty cool free online service called IFTTT (If That Then This) which allows you to easily create what is known as a recipe that is composed of trigger and an action to send a notification. This can be thought of like a vCenter Alarm with a trigger and action, but in IFTTT's case, a trigger and action can be almost anything such as an email, facebook event, twitter, etc.

I stumbled onto IFTTT while going through some of my unread blog feeds this morning since I could not go back to sleep after randomly waking up at 4am in morning. I was reading an article from Matt Cowger about something called a Pebble which I had not heard of before and it peaked my interest. In reading about Pebble (pretty slick actually), I learned about IFTTT which is one of the notification systems Pebble supports. I decided to give IFTTT a try and to see how easy it would be to setup SMS notifications for a vCenter Server alarm.

Step 1 - Sign up for IFTTT account, you need to ensure the email account that you use to register is the same account that will be used to send the trigger.

Step 2 - Create a new Recipe using email as the trigger and SMS for the action. The process is very straight forward, just follow the wizard and at the end you will enter your phone number for SMS notification and confirm with a code.

Step 3 - Create a vCenter Alarm for the event you wish to trigger off of and set the email address to send to *protected email* as noted in the Recipe.

Step 4 - In my test, I created an alarm for a Powered Off event and went ahead and powered off a VM to generate the alarm.

Step 5 - IFTTT checks the triggers every 15minutes. If you do not wish to wait 15minutes you can force a check by clicking on Check Now button in the Recipe.

Step 6 - If everything was setup correctly, you should have received a text message with the details of the vCenter alarm. In my Recipe, I configured it to only send the Subject which contains everything you would need to know about the vCenter alarm, at least for you to decide whether or not you want to investigate the issue.

The possibilities are pretty much endless in terms of the triggers and actions that you could create with IFTTT, I even created one for my ghettoVCB backup notification. The only downside that I noticed while giving IFTTT a try is that the trigger check is every 15minutes which could be a bit long for things requiring immediate attention, but I also read that there are certain Recipes that supports a "Quick Trigger" which would then execute immediately upon receiving. I think for a free service this is very cool and much easier than setting up your own SMS system. I would recommend giving IFTTT a try and see what cool Recipes you can build and integrate with your VMware or other environments.

I hope my body will let me crash and get some sleep now ...

Detecting A Duplicate IP Address For Your ESXi Hosts Using a vCenter Alarm

by 5 Comments

The motivation for this article was a tweet I noticed from Duncan Epping this morning. Per Duncan's tweet, it looks like he may have accidentally assigned an IP Address to one of his virtual machines which was already being used by an existing ESXi host causing a duplicate IP Address error. We probably have all experienced this once in our lives and it can be quite difficult and frustrating to troubleshoot. Similar to a Windows OS, ESXi can also detect a duplicate IP Addresses but instead of a notification window, it is just logged in the VMkernel logs which looks like the following:

2013-01-21T15:52:35.989Z cpu1:2049)Tcpip_Vmk: 112: arp: 00:50:56:bd:3b:2b is using my IP address 172.30.0.213 on vmk1! 

The biggest challenge of course is to identify which ESXi host actually has a conflict and then taking a look at the logs to find the offending MAC Address and shutting them down yourself or with the help of a Network Administrator. Wouldn't it be great if we had an alarm to automatically notify us when a duplicate IP Address is detected? Well I am glad you asked and the answer is YES! 🙂

In addition to logging to the VMkernel logs, ESXi also logs this "observation" in /var/log/vobd.log which stands for the VMkernel Observation. These "observations" can provide critical identifying information in case of an error and is usually used during troubleshooting. In our case, we are seeing an intermittent network connectivity to our ESXi host which is in result of a duplicate IP Address. The really neat thing about these VOBs is that you can create vCenter Alarms when a specific VOB has been detected. I have shown an example of this before in my Detecting ESXi Remote Syslog Connection Error Using a vCenter Alarm article.

We can do exactly the same for detecting a duplicate IP Address for an ESXi host. The first thing we need to do is identify the VOB ID by looking in /var/log/vobd.log:

2013-01-21T15:02:07.513Z: [netCorrelator] 917174784727us: [esx.problem.net.vmknic.ip.duplicate] Duplicate IP address detected for 172.30.0.83 on interface vmk0, current owner being 00:50:56:bd:3b:2b

We can see the VOB ID for this is esx.problem.net.vmknic.ip.duplicate and this will be used in our vCenter Alarm trigger.

Step 1 - Create a new Alarm and specify a name, the Monitor type will be Hosts and Monitor For will be for a specific event:

Step 2 - Copy the VOB ID that we have identified from above and specify that as our alarm Trigger:

Step 3 - If you wish to receive an email notification or send an SNMP trap go ahead and configure additional actions, else just click next which will just display a vCenter Server alert in the UI.

Now that our alarm has been created, we will want to give this a test drive .... who can we ask? Well it just happens that I have a new user in my environment and I provisioned him a new VM which is already connected to the network. Let's hope he does not try to change the IP Address (because this never happens, right?)

After the user statically assigns the IP Address of an existing ESXi host in the VM, we should see our new alarm trigger in vCenter.

As you can see, we have quickly identified the ESXi host that is impacted and we can then login to DCUI via the console to take a look at the logs to find the offending MAC Address. Hopefully duplicate IP Addresses is not a common problem in your environment but it does happen from time to time and having an alarm to help you quickly narrow down the culprit can be quite useful.

What Are the Shadow of vmnic in ESXTOP?

by 6 Comments

In ESXi 5.1, you might have noticed something new called Shadow of vmnic under the USED-BY column in the Network view of ESXTOP.

I initially heard about this from a few followers on twitter and I was curious myself since I could not find any documentation regarding this topic. It took a bit of digging but it turns out these shadow vmnics is actually related to the new VDS Health Check feature released in vSphere 5.1. A shadow port is created automatically for every uplink in your ESXi host and is used for transmitting and receiving health check related packets for each uplink. In ESXTOP, you can monitor the statistics for these shadow ports which can be used to debug/troubleshoot the network health check feature and this is why they are present.

One thing to note, these shadow ports are created regardless of whether or not your ESXi host is connected to a VDS or if you have the network health check features enabled. This is by design and there are four VMkernel modules that are responsible for the network health check feature:

  • vlanmtucheck
  • teamcheck
  • heartbeat
  • healthchk

Disclaimer: Do not modify or disable any of the above VMkernel modules else you can potentially disable network connectivity to your ESXi host (trust me, I know).

After some investigation and testing in my lab, I found that I could unload the above VMkernel modules and the shadow vmnics entries would no longer appear in ESXTOP. To do this, you will need an ESXi 5.1 host which is not running any virtual machines and you will need to run the following commands in this exact order (as there are module dependencies):

vmkload_mod -u vlanmtucheck
vmkload_mod -u teamcheck
vmkload_mod -u heartbeat
vmkload_mod -u healthchk

Once you have successfully unloaded all four VMkernel modules, if you open up ESXTOP, you should no longer see the shadow vmnics. To restore the shadow vmnics, you can either reboot (since the unload is not persistent) OR you can run the following commands in this exact order:

vmkload_mod heartbeat
vmkload_mod teamcheck
vmkload_mod vlanmtucheck

Note: By loading the heartbeat VMkernel module, it also loads the healthchk module.

Configure Apple Mac Mini to Default Boot ESXi

by 13 Comments

If you are running ESXi on an Apple Mac Mini and it is installed on a USB key, you probably have noticed that the Mac Mini tries to boot from disk by default and instead of using the USB device. This means when you reboot your ESXi host each time, you will need to hold down the "ALT/OPTION" key which will present you with a boot menu to select the device you wish to boot from.

This can be quite annoying if you have a headless setup for your Mac Mini and you just want it to automatically boot off of the right device containing your ESXi installation. To fix this, you can configure the default boot device which can be done by first selecting the device you wish to boot off of as shown in the screenshot above. Next, hold down on the "CONTROL" key which will turn the straight arrow into circular arrow icon as shown in the screenshot below.

Now you just need to either hit enter or if you have a mouse, click on the circular arrow icon and this will configure the default boot device the Apple Mac Mini will use going forward. It is that simple! If you want to boot off of another device after configuring the default boot device, you can still do so by holding down "ALT/OPTION" key while the Mac Mini is still booting up.

Credit goes to this site for solution. 

Blocking vSphere C# Client Logins

by 6 Comments

I recently picked up on this neat little tidbit from Mr. Not Supported aka Randy Keener, where you can block a user from logging into the vCenter Server using the vSphere C# Client. Other than playing a prank on your co-workers, you might be wondering is there a use case for this? Surprisingly, this is a request I have heard from a few customers in the past where they would like to block their users from using the vSphere C# Client in favor of leveraging only the vSphere APIs for routine tasks.

Since the vSphere C# Client also uses the vSphere API itself, a user with proper credentials to the vSphere environment can easily download the client from an alternative source and still login. Of course, there are ways of preventing this such as restricting application installation on end users desktop but there is some amount of management overhead of identifying those existing and new users, especially if access is delegated out to other teams.

There is a very simple solution if you choose to block ALL users from using the vSphere C# Client which requires a tiny modification on the vCenter Server itself and it takes effect immediately with no service restarts.

Disclaimer: This is probably not officially supported by VMware, use at your own risk.

Login to your vCenter Server and locate a file called version.txt

Windows: C:\ProgramData\VMware\VMware VirtualCenter\docRoot\client
VCSA: /etc/vmware-vpx/docRoot/client

There is parameter called exactVersion which will be set to current supported version of the vSphere C# Client which should also match the version of your vCenter Server. You just need to change this to some other value that you know will not exist in your environment such as 9.0.0. Once you have made this change, now when a user tries to connect and there is a miss-match in the version, the vCenter Server will provide you with a download to the vSphere C# Client located on the server as it normally would if you did not have the latest client.

What the user will find out shortly, is that this will continue in an infinite loop even after installing the proper vSphere C# Client. The reason for this is that the number in version.txt will never match the vSphere C# Client and vCenter Server will just continue serving the installer in an infinite loop. I also looked into this trick for a standalone ESXi host and you can do the same by editing a file called clients.xml which is located in /usr/lib/vmware/hostd/docroot/client and users will not be able to login to the ESXi host using the vSphere C# Client.

Now, even though this prevents users from logging into the vSphere C# Client, users will still be able to connect using the vSphere API which includes the use of vCLI/ESXCLI, PowerCLI, vCO, SDKs, etc. and the use of the vSphere Web Client for either vSphere 5.0 or 5.1 will continue to work. Ideally, it would be nice to be able to control this access on a per user/group basis and perhaps even specify how a user can connect whether that is through the use of the APIs or UI only. Is this even useful to have at all? Would love to hear your comments.

For now, if you want users to get familiar with the new vSphere Web Client 5.1 ... this is one way of "encouraging" them 😉

vGhetto Lab #NotSupported Slides Posted

by Leave a Comment

As promised, here are slides to my #NotSupported session at VMworld Europe which I continued the theme of home labs with my vGhetto Lab #NotSupported presentation.

The idea behind the vGhetto Lab is to easily setup a vSphere home lab without too much effort and most importantly, leveraging as little resources as possible. This is all accomplished with the following:

  • Physical host running ESXi 5.x
  • ESXi 5.x offline depot image
  • VCSA 5.x (vCenter Server Appliance)

In addition to the above, you will also need to download the vGhetto Lab scripts which are shown in the video.
Here are some additional details on how to quickly get setup with your own vGhetto Lab.

Step 1 - After installing ESXi 5.x on your physical host, you will need to deploy the VCSA. Make sure you add a second network interface to VCSA as shown in the presentation. In my example, I created another vSwitch with no uplinks and portgroup for Auto Deploy network

Step 2 - Once the VCSA is powered on, go ahead and SCP the scripts to virtual machine. The first script that we will need to execute is the setupNetwork.sh and you will need to edit the following variables:

VCENTER_IP_ADDRESS_1=192.168.1.150
VCENTER_NETMASK_1=255.255.255.0
VCENTER_GATEWAY=192.168.1.1
VCENTER_IP_ADDRESS_2=172.30.0.1
VCENTER_NETMASK_2=255.255.255.0
VCENTER_HOSTNAME=vcenter.primp-industries.com
DOMAIN_LIST=primp-industries.com
DNS_LIST=192.168.1.1

Note: To ensure that you do not accidentally run the script without changing out the variables, there is another variable called ACTUALLY_READ_SCRIPT that needs to be changed from "no" to "yes" else the script will not execute.

Step 3 - Next we need to configure the vCenter Server, we will need to execute the configureVCSA51.sh which will configure the embedded SSO Database as well as the database of the vCenter Server. You do not need to edit any variables in this script

Step 4 - Finally, we need to configure our DHCP, TFTP, Auto Deploy services as well extracting the ESXi offline depot image and preparing it for use with Auto Deploy. You will need to edit the following variables before running the setupvGhettoLab.sh script.

DHCP_SUBNET=172.30.0.0
DHCP_NETMASK=255.255.255.0
DHCP_START_RANGE=172.30.0.100
DHCP_END_RANGE=172.30.0.200
DHCP_INTEFACE=eth1
TFTP_SERVER=172.30.0.1
VCSA_SERVER=192.168.1.150
ESXI_OFFLINE_DEPOT=/root/VMware-ESXi-5.1.0-799733-depot.zip
ESXI_REPO_PATH=/etc/vmware-vpx/docRoot
ESXI_REPO_DIR=ESXi-5.1.0

Note: For the Image Profile and Auto Deploy rule creation, if you wish for the script to execute them automatically versus echoing to the screen, remove the "echo" statement as well as the double quotes from the following so the last three lines look like this:

pxe-profile-cmd create $(cat /tmp/VIBS) ${ESXI_REPO_DIR}
rule-cmd create -i pxe:${ESXI_REPO_DIR} ${AUTO_DEPLOY_RULE} vendor=='VMware, Inc.'
rule-set-cmd set ${AUTO_DEPLOY_RULE}

Step 5 - You are now ready to create your nested ESXi virtual machines. You can use RVC as shown in the presentation (there is a slide at the very end which lists the commands) or you can connect to vSphere Web Client and create the ESXi virtual machines the traditional way via the GUI.

After updating the DHCP configurations with the new MAC Addresses from your nested ESXi virtual machines, you should then see Auto Deploy automatically provision your ESXi hosts and join them to the VCSA you deployed earlier.

Additional Links: