Cloning vMA still not supported?

Cloning and/or moving vMA is not supported by VMware, this has been the case since VIMA 1.0. If you take look at the "Known Issues" in the vMA release notes, you will see there is no still workaround. Users may find it useful to be able to clone vMA for backup purposes or to deploy another vMA without having to re-download or re-uploading .OVF from your local desktop/VMware's website.

If you ever tried to clone VIMA 1.0 and boot the system up, you will notice a warning message during the init process when vMA verifies the system's UUID. If it detects that it has been cloned, it will actually disable itself and makes the vMA components unusable until you reinitialize vMA using vimaclean. This is still the case with vMA 4.1, except the vMA components are no longer disabled when you perform a clone.

You will just see the warning message during the init process, but all functionality continues to work. Before the vMA specific components starts up, it makes a call to /opt/vmware/vma/bin/verifyuuid. It verifies the system UUID which you can extract using dmidecode, this is the same UUID found within the .vmx entry called uuid.bios. When you clone a VM, a new UUID will be generated which causes an issue when it tries to verify.

Prior to vMA 4.x, VIMA 1.0 stored all it's configurations within a set of XML files located in /etc/vmware/viconfig, these files held the config for both the vi-fastpass targets and vilogger config and policies. With the release of vMA 4.0, these configurations were migrated to a sqlite3 database which is stored in /var/opt/vmware/vMA/vMA.db. Within this database, it also stored as the old flat file configuration, the system UUID of vMA. This is what verifyuuid was checking against to see if it was cloned or moved, you can actually query the vMA DB to extract this information using a simple SQL query.

To resolve the warning message, you just need to update the new UUID in the vMA DB using an update statement.

Instead of manually typing all that out, I actually created a very simple script that queries for the current system UUID and then updates the vMA DB with the correct UUID. Here is a sample execution:

Download: updatevMAUUID.sh

I suspect that VMware may have allowed cloning of vMA, but just did not update their release notes? The only thing you have to do is manually fix the UUID if you do not want to see the warning message. If you would like to fix this for VIMA 1.0, you will need to do the same thing except you need to modify the UUID found in /etc/vmware/viconfig/viconfig.xml and restart vMA for the changes to effect.

How to automate & cron vi-fastpass scripts on vMA 4.1

vMA's vi-fastpass and vifptarget utility is meant to be used interactively on the command line, this requires the user to set a particular target (ESX,ESXi or vCenter host) and then execute either a vCLI and/or vSphere SDK for Perl script. If you want to stay in the context of a particular server (e.g without having to manually specify --server parameter), you would need to manually change the target. There have been two common set of questions that have re-appeared several times on the VMTN forums:

1) How to automate a particular script and leverage vi-fastpass without having to change the context for each ESX or ESXi host

2) How to automate a particular script and leverage vi-fastpass via cron without having to provide username/password 

Before we dive into the solutions, the key in using vi-fastpass in a non-interactive fashion is how vifptarget is called from a script. When you call the vifptarget, it is actually an alias to source the following shell script located in /opt/vmware/vma/bin/vifptarget. You can see this by using the "which" utility on a particular command.

This alias is defined in /etc/bashrc

You can even redefine the alias name to whatever you like, so long as it's aliasing the actual vifptarget script. Properly sourcing vifptarget script is the key in solving Question 1. If you look at /etc/bashrc, you will also notice that LD_LIBRARY_PATH environmental variable for shared libraries is also defined, this will be the key in solving Question 2 along with properly sourcing vifptarget. You will see these referenced in the example scripts. 

There are two solutions for Question 1:

Solution A:

If you are leveraging vCenter Server and the ESX(i) hosts are being managed by vCenter, then you just need to add vCenter to vi-fastpass and run a simple for loop across the hosts.

Example:
2 x ESXi 4.1 hosts being managed by vCenter

• esx4-4.primp-industries.com
• esx4-5.primp-industries.com

Execute esxcfg-ntp to configure 2 x NTP servers for both ESX(i) host

• ntp1.ucsb.edu
• ntp2.ucsb.edu

Here is a shell script that performs the configuration on the two ESX(i) host being managed by vCenter and reads in a file containing the name of the ESX(i) hosts:

example1.sh

Download: example1.sh

Solution B:

In this solution, vCenter is not available or you each ESX(i) host being managed by vMA's vi-fastpass. The script will be very similar, but instead of setting vCenter as the target, you will be setting an ESX(i) host. What is interesting that may not be well known is if you are managing more than one ESX(i) vi-fastpass target, you only need to set the target once and you can perform the same operation on all other host. The caveat being, if you want to switch out of the default context of your target, you need to specify --server parameter. This script will actually do it for you behind to scenes.

Example:
2 x ESXi 4.1 hosts being managed by vMA

• esx4-4.primp-industries.com
• esx4-5.primp-industries.com

Execute esxcfg-ntp to configure 2 x NTP servers for both ESX(i) host

• ntp1.ucsb.edu
• ntp2.ucsb.edu

Here is a shell script that performs the configuration on the two ESX(i) host being managed by vMA and reads in a file containing the name of the ESX(i) hosts:

 example2.sh

Download: example2.sh

After executing either SolutionA or SolutionB, you now should have the configurations applied to your ESX(i) host as you would if you manually ran it on each host. This is just an example, you can easily substitute multiple esxcfg-* and/or vSphere SDK for Perl scripts. Here is also another example script for a user on the VMTN forums on configuring a vSwitch and adding portgroups.

Solution for Question 2:

Now that we know how to automate a particular operation using vi-fastpass in a script, the next logical question is how do we automate it using a cronjob. I will not go into details of how cron works, take a look at the link for more details. One thing I will mention is that cron does not have all the same PATH environmental variables defined as you normally would expect. You should always use full path to scripts and binaries and any shared library modules that is required to execute a particular utility. What this means for leveraging vi-fastpass via a cronjob is that you need to specify the LD_LIBRARY_PATH. In solving Question 1, all we had to do was properly source the vifptarget and the reason we did not have to specify the shared library path is that it was already defined as a default for vi-admin user. This is not the case for cron and you will need to specify that in either the script you are executing or within the crontab.

Example:

2 x ESXi 4.1

• esx4-4.primp-industries.com
• esx4-5.primp-industries.com

Execute vmware-cmd -l to extract virtual machines on both ESX(i) host and store the output that to a file called /tmp/vms every 5 minutes. Here is what the cron entry should look like for example3.sh

example3.sh

Download: example3.sh

After 5 minutes, you should now see a new file under /tmp/vms that contains all virtual machines registered on your ESX(i) host. Again, you can easily apply this to any other vCLI and/or vSphere SDK for Perl script.

Here another example that gets asked quite often in scheduling the popular VMware vSphere Health Check Script, here is the crontab entry for automatically running the healthcheck script every hour against your vCenter server.

example4.sh

Download: example4.sh

Again, the key is to make sure you specify the full path to not only the script you would like to run but also if there is any type of output, that you specify the full path to the output.

How to increase capacity of /var/log on vMA 4.1

This question pops up from time to time on the VMTN community forums on how to increase the size of /var/log on vMA. By default, the /var/log partition is configured to ~500MB, which is a decent size for system logs, but one of the features of vMA is vilogger which adds a syslog capability for your ESX and ESXi hosts. This allows you to ship all your host logs over to vMA, the problem is the logs are also stored in /var/log along with vMA's system logs. If you are managing several dozen hosts, you can easily fill up this partition using the default configured size.

With vMA 4.0, it was pretty trivial to increase/resize partitions within vMA using LVM and I had written a quick guide on how to do this - How to increase/resize vMA Disks. With the release of vMA 4.1, VMware kind of went backwards and decided not to leverage LVM and lost the benefits with using a volume manager. Resizing the partitions can still be accomplished but it requires a few additional steps and tools. I will show you two methods in which you can increase your /var/log partition, primarily to accommodate larger number of hosts to utilize the vilogger functionality.

Option1: Resize /var/log using gparted 

Here is what a default vMA disk layout looks like and note the size of /var/log

First, you will want to shutdown your vMA host, you may also want to backup any scripts/configs you may have on your vMA host in case you make a mistake and can not recover. Next, you will need to increase the existing virtual disk, by default it is configured for 5GB. In this example, I will increase the disk capacity to 10GB bringing the total to 15GB. You will also need to download a gparted live CD ISO, in this example I am using gparted 0.3.4-11, it is what I have available on my datastore.

Next, you will want to attach the gparted ISO to vMA, you probably will need to add a CD-ROM drive since vMA does not come with one by default. Once you power vMA on, you will need to hit ESC to and select boot from CD-ROM. You should be able to just accept all the defaults and once gparted is loaded, you should see the following screen or something similar depending on the version of gparted you are using.

As you can see, it is the partition layout of your vMA host including the unallocated space, which in this case is 10GB. At a high level, we need to resize the partitions inside the extended partition first before we can resize /var/log partition. First you will select/highlight the "extended (/dev/sd4)" partition which is colored in baby blue color and you will move the right arrow and extend it all the way to the right and then click "Resize/Move" button.

Next, you will select/highlight "/ (dev/sd5)" partition which is colored in blue. Here we want to keep / (root) the same default size of 3.39GB, so we are just going to shift the entire box from the left to far right. You just need to bring the cursor on top of the dark blue section and drag it to the right. Make sure you do not move the left or right arrows, else you will be changing the size of root which is not what we want. Once you have it like the picture below, click on the "Resize/Move" button.

Now you will select/highlight "extended (/dev/sd4)" partition again and you will move the left arrow and drag it towards the right like the picture below and again click on the "Resize/Move" button once you are done.

At this point, your screen should look like the following, if it does not for what ever reason you can "undo" all the changes, since none of these changes take affect until you apply them. 

Now, we will finally increase the size of "/var/log (/dev/sd3)" and consume the full amount of space that we have allocated.

Once you are done, we are now ready to apply these changes, the "apply" button is at the top.

After this, it can take up to several minutes depending on your allocation but if everything went well, you should see this success screen at the end and you just need to reboot the system to see the new changes.

One your vMA host is up, you now should be able to run "df -h" again and you should see that /var/log has now increased with the additional space we assigned it.

Option2: Add a new virtual disk and reconfigure vilogger syslog location

If you did not want to go through the process in option 1 and just want to add another virtual disk and dedicate that for vilogger syslog, you can actually change the path of the logs in /etc/vmware/vMA/vMA.conf. You can control not only the vilogger syslog but also vi-fastpass logs and vilogger daemon logs, but we will only change the "vMALogCollector" which is basically the vilogger syslog component.

I will assume you already have a disk added to vMA and you have created the appropriate mount point (If you are unsure how, do a search online). In this example, I have a 15GB partition in /vmasyslog that I will use as the new home for all my vilogger syslog files.

After you have made the edit to vMA.conf, you will need to restart vilogger daemon and you will need to use sudo and the service command.

Now, you can enable vilogger for your ESX and ESXi host and you should see all logs being redirected to this new partition instead of the default /var/log/vmware/*

Updated vSphere Security Hardening Report Script for vSphere 4.1

VMware released earlier this week the first draft copy of the vSphere 4.1 Security Hardening Guide which provides several changes to the vSphere 4.0 version released last year. Unfortunately there was no change list provided and you have to manually go through both documents to get the differences. Luckily I did the heavy lifting for you and here are the changes in 4.1 version:

Edit: It looks like Charu of VMware has already posted a "diff" of the 4.0 and 4.1 version here.

Added Checks (14):

• VSH07 (Enterprise) - Check for privilege re-assignment after vCenter Server restarts
• VSH10 (Enterprise) - Clean up log files after failed installations of vCenter Server
• VUM06 (Enterprise) - Do not use default self-signed certificates
• VMX23 (Enterprise) - Use secure protocols for virtual serial port access
• VMX24 (DMZ) - Disable certain unexposed features
• VMX56 (Enteprise) - Restrict access to VMsafe network APIs
• HIN02 (Enterprise) - Keep ESX/ESXi system properly patched
• HCM05 (DMZ) - Disable Welcome web page
  
• HMT12 (Enterprise) - Prevent unintended use of VMsafe network APIs
• HMT15 (Enterprise) - Audit for loading of unauthorized kernel modules (ESXi only)
• HMT20 (DMZ) - Ensure that vpxuser auto-password change meets policy
• HMT21 (DMZ) - Ensure that vpxuser password meets length policy
• HCN05 (SSLF) - Disable DCUI to prevent all local administrative control
• HCN06 (Enterprise) - Disable Tech Support Mode unless needed for diagnostics and break-fix
  

Removed Checks (10):

• VMX03 (Enterprise) - Disable copy/paste to remote console
• VMX51 (Enterprise) - Restrict access to VMsafe CPU/memory APIs
• VMX54 (Enterprise) - Restrict access to VMsafe network APIs
• HCM04 (Enterprise) - Ensure that ESX is configured to encrypt all sessions
• HMT10 (Enterprise) - Prevent unintended use of VMsafe CPU/memory APIs
• HMT11 (Enterprise) - Prevent unintended use of VMsafe network APIs
• HCN01 (Enterprise) - Ensure that only authorized users have access to the DCUI
• HCN03 (Enterprise) - Avoid adding the root user to local groups
• HCN04 (SSLF) - Disable tech support mode
• COP06 (DMZ) - Ensure that vpxuser auto-password change in vCenter meets policy

Note: Some of the removed checks may have been replaced with newer and updated information and shows up in the added checks.

To help with your vSphere validation, here is the latest version of the vSphere Security Hardening Report script 1.5 script. There have been a few enhancements to the script which only validates a check based on whether it it is applicable to classic ESX or ESXi, which in the past it would display "N/A". There is also some further validation of the service endpoints for /, /ui, and /mob that may also help reduce manual verification where applicable. You can also join the new vSphere Security Hardening Report VMTN Group for new updates, bug report and discussions.

Here is an updated sample report based on vSphere 4.1:
vmwarevSphereSecurityHardeningReport-SAMPLE.html

One other thing I noticed while going through both the 4.0 and 4.1 security guide is the numbers for the code are all over the place, there are sometimes huge gaps that are unexplained (e.g. VSH6, VSH7 ... VSH10)

Ghetto Groups

Back in December, VMware upgraded their VMTN (VMware Technology Network) forum software Jive and introduced a completely new layout of the forums that would hopefully enhance the user experience. Though it brought many new features, it also brought on several new issues. The one bug that affected me was the incorrect conversion of the ghettoVCB document, because the conversion was unsuccessful it was decided to be left alone until the issue could be resolved. If you visited the document, it would display a "Forbidden" error message. Unfortunately due to the time it took to resolve, even Google cache started to get stale and stopped serving the cached contents.

Luckily, with the help of Alex Maier (VMTN Community Manager) and her team, she was able to get the ball rolling and got the fix tested and rolled out to production. The ghettoVCB document is once again alive and hopefully in no time it will be returned as the first search result on various search engines.

Going through the pain of receiving dozen of emails, private messages, tweets, etc. per week regarding the issue, I came to realize that VMTN document itself was not the right medium to host both content and user discussions. As it stands today, there are over 1,100+ comments which is pretty significant and managing and keeping up with the conversations is a pretty daunting task. I enjoy the feedback that community provides and the collaboration that takes place and I realize that this can be solved by using the new Groups feature.

To be honest, I did not spend much time looking at Groups when the VMTN software was upgraded, but now that the ghettoVCB document has been fixed, I realized this would fit this need perfectly. With the help of categories users can now post their feedback, discussions/issues and feature request and it can be easier consumed by both new users and myself. Starting today, I will have the following groups based on the top 5 most popular and active scripts:

ghettoVCB Group

ghettoVCBg2 Group

vmwarevSphereHealthCheck Group

vmwarevSphereSecurityHardening Group

ghettoUPSHostShutdown Group 

I have also disabled any new comments on these VMTN documents and will ask that all new comments be re-directed to respective VMware Groups. I'm currently working with Alex to see if there is an easy way to convert the existing comments into a document and attached that as a download to help minimize the complexity of the document. In the worse case, the comments will be left alone as read-only as I think the discussion that currently exists are invaluable. All other VMTN documents that I maintain in the vGhetto Repository will continue to use comments and depending on how well the groups go, I may migrate those over as well.

I hope these new groups will be beneficial for everyone and I am looking forward to the collaboration. Thanks for your support and please help spread the word!

How to extract host information from within a VM?

From time to time, I see this question come up asking how one might be able to extract a certain piece of information from either ESX(i) or the management APIs (vSphere API) from within a virtual machine. The simple answer is you can not, by default the guest operating system has no idea of the underlying hypervisor nor does it have the access to the management APIs. This of course, assumes you are following VMware's best practices in isolated and segregating off your management network from your virtual machine network.

Having said that, there are certain bits of information that you can extract about your ESX(i) host from within the guestOS using some of the utilities that is installed with VMware Tools. The first utility is called VMware Toolbox command which can be found on both UNIX/Linux and Windows systems that have tools installed.

UNIX/Linux - /usr/bin/vmware-toolbox-cmd

Windows - C:\Program Files\VMware\VMware Tools\VMwareToolboxCmd.exe

This utility provide some information about ESX(i) and guestOS configuration including basic resource statistics.

One interesting sub-command is the resource statistics about both ESX(i) and guestOS such as speed of the hypervisor's CPUs or the current balloon, swap, memory limit, memory reservations, cpu limit or cpu reservations.

Here is an example of retrieving ESX(i) CPU speed.

As I mentioned before, you do not have access to the management network that your ESX(i) are on and that also means you do not have access to the vSphere APIs. What if you wanted to to associate a specific piece of information from ESX(i) and be able to access that piece of information from the guestOS? You can do so with the vmtoolsd (VMware Tools Daemon) utility.

UNIX/Linux - /usr/bin/vmtoolsd

Windows - C:\Program Files\VMware\VMware Tools\vmtoolsd.exe

This utility has an option called --cmd that allows you to run various commands including one that allows you to extract guest information using the "info-get" parameter. This guestinfo can be set using two methods:

1) Hard-coded within the virtual machine's .vmx configuration file
2) In VMX memory while a virtual machine is running within ESX(i)

Option 1 is pretty straight forward, you can add a custom attribute that has the following format: 

guestinfo.[variable] = [value]

e.g.
          guestinfo.provision.date = "01/11/2011"

You can use the vSphere Client to add this custom attribute while the virtual machine is powered off or you can manually edit the .vmx configuration file. If you use the latter, you will need to reload the VM's configuration, you can use vim-cmd vmsvc/reload.[vmid] to do so. One the VM is powered on, you can extract this piece of information, here is an example:

Option 2 is actually pretty interesting as the configuration of the custom guestinfo variable is not permanently stored. What I mean by this is the configuration is only persisted while the VM is running. This is interesting because if you have runtime information that potentially could change, this allows you to dynamically present this information to the guestOS. One use case could be set for management VMs such vCenter running in a VM and you wanted to know at any given time which ESX(i) host is currently managing it. This can be trivial with vCenter access, but what if the service is offline but the virtual machine is still running? Using the classic ESX Service Console command vmware-cmd you can loop through all powered on VMs and set some custom variables for example the current hostname of the ESX host managing the VM and version it is currently on.

The syntax for the command would be the following:

vmware-cmd [VM_VMX_PATH] setguestinfo [VARIABLE_NAME] [VARIABLE_VALUE]

e.g
          vmware-cmd /vmfs/volumes/4a48004d-f9af7fa0-5bbf-003048d9586b/scofield/scofield.vmx setguestinfo hypervisor.hostname $(hostname)

Note: Notice, you do not need to append the keyword guestinfo, you just need to specify the variable name. Once you are in the guestOS, to access the custom variable, you will need to specify the full name which should be "guestinfo.[VARIABLE]"

As you can see this can be tedious to set for each and every VM, so let's automate this using a script that will go through all powered on VMs and set two custom variables called "hypervisor.hostname" which will set the current hostname of the ESX host and "hypervisor.build" which will retrieve the build of your ESX. To further automate this since you might have multiple ESX hosts running in a DRS cluster, you would store this script in a shared storage (VMFS and/or NFS) and setting up a cronjob that would run at some interval to always provide the latest information to the guestOS.

Here is an example of a shell scrip that does exactly that:

You will want to save this script to your ESX host and make sure the script has executable permissions for it to execute. You will also create a cronjob based on how often you want the script to run and this needs to be configured for every ESX host that you would like to take part in this update.

Here is an example of running the script every hour:
0 * * * * /vmfs/volume/shared-storage/setGuestInfo.sh > /tmp/out

Note: I redirect the output to /tmp/out to ensure that script is in fact working and once you have, you can remove that all together.

Here is an example of extracting these two custom variables on both a Linux and Windows VM:

As I mentioned earlier, option 2 does not persist these custom variables with respect to the VM's configuration file. The variable configuration only takes affect when the VM is powered on and once it is powered off, the information is discarded. To support ESXi, you can use the various vSphere SDKs (vSphere SDK for Perl, PowerCLI, etc) but what you will find is that the behavior of setting this attribute is similar to that of the local vmware-cmd on classic ESX, it does not append the entry into the .vmx configuration file.

Here is a vSphere SDK for Perl script called addVMAdvParamOption.pl that can be used to guestinfo parameter for a given VM. Here is an example of this script:

If you are using vMA, you can also use the vCLI's vmware-cmd utility to set this variable. The format is slightly different than that of the classic ESX's vmware-cmd. Here is an example of this:

Notice, the variable name must include "guestinfo." keyword before specifying the variable name.

If you wanted to persist these configurations, you will need to adjust your provision workflow to append the variables into the VM's .vmx configuration file.