• Skip to primary navigation
  • Skip to main content
  • Skip to primary sidebar

virtuallyGhetto

  • About
  • Privacy
  • VMware Cloud
  • Home Lab
  • Nested Virtualization
  • Automation
    • VMware Kickstart
    • VMware API/SDK/CLI
    • VMware vMA/VIMA
    • VMware OVF / OVFTOOL
  • Apple Mac
  • VCSA
  • VSAN

Updated vSphere Security Hardening Report Script for vSphere 4.1

01/22/2011 by William Lam 5 Comments

VMware released earlier this week the first draft copy of the vSphere 4.1 Security Hardening Guide which provides several changes to the vSphere 4.0 version released last year. Unfortunately there was no change list provided and you have to manually go through both documents to get the differences. Luckily I did the heavy lifting for you and here are the changes in 4.1 version:

Edit: It looks like Charu of VMware has already posted a "diff" of the 4.0 and 4.1 version here.

Added Checks (14):

  • VSH07 (Enterprise) - Check for privilege re-assignment after vCenter Server restarts
  • VSH10 (Enterprise) - Clean up log files after failed installations of vCenter Server
  • VUM06 (Enterprise) - Do not use default self-signed certificates
  • VMX23 (Enterprise) - Use secure protocols for virtual serial port access
  • VMX24 (DMZ) - Disable certain unexposed features
  • VMX56 (Enteprise) - Restrict access to VMsafe network APIs
  • HIN02 (Enterprise) - Keep ESX/ESXi system properly patched
  • HCM05 (DMZ) - Disable Welcome web page
  • HMT12 (Enterprise) - Prevent unintended use of VMsafe network APIs
  • HMT15 (Enterprise) - Audit for loading of unauthorized kernel modules (ESXi only)
  • HMT20 (DMZ) - Ensure that vpxuser auto-password change meets policy
  • HMT21 (DMZ) - Ensure that vpxuser password meets length policy
  • HCN05 (SSLF) - Disable DCUI to prevent all local administrative control
  • HCN06 (Enterprise) - Disable Tech Support Mode unless needed for diagnostics and break-fix

Removed Checks (10):

  • VMX03 (Enterprise) - Disable copy/paste to remote console
  • VMX51 (Enterprise) - Restrict access to VMsafe CPU/memory APIs
  • VMX54 (Enterprise) - Restrict access to VMsafe network APIs
  • HCM04 (Enterprise) - Ensure that ESX is configured to encrypt all sessions
  • HMT10 (Enterprise) - Prevent unintended use of VMsafe CPU/memory APIs
  • HMT11 (Enterprise) - Prevent unintended use of VMsafe network APIs
  • HCN01 (Enterprise) - Ensure that only authorized users have access to the DCUI
  • HCN03 (Enterprise) - Avoid adding the root user to local groups
  • HCN04 (SSLF) - Disable tech support mode
  • COP06 (DMZ) - Ensure that vpxuser auto-password change in vCenter meets policy

Note: Some of the removed checks may have been replaced with newer and updated information and shows up in the added checks.

To help with your vSphere validation, here is the latest version of the vSphere Security Hardening Report script 1.5 script. There have been a few enhancements to the script which only validates a check based on whether it it is applicable to classic ESX or ESXi, which in the past it would display "N/A". There is also some further validation of the service endpoints for /, /ui, and /mob that may also help reduce manual verification where applicable. You can also join the new vSphere Security Hardening Report VMTN Group for new updates, bug report and discussions.

Here is an updated sample report based on vSphere 4.1:
vmwarevSphereSecurityHardeningReport-SAMPLE.html

One other thing I noticed while going through both the 4.0 and 4.1 security guide is the numbers for the code are all over the place, there are sometimes huge gaps that are unexplained (e.g. VSH6, VSH7 ... VSH10)

Filed Under: Uncategorized Tagged With: hardening guide, security, vSphere 4.1

Ghetto Groups

01/20/2011 by William Lam 1 Comment

Back in December, VMware upgraded their VMTN (VMware Technology Network) forum software Jive and introduced a completely new layout of the forums that would hopefully enhance the user experience. Though it brought many new features, it also brought on several new issues. The one bug that affected me was the incorrect conversion of the ghettoVCB document, because the conversion was unsuccessful it was decided to be left alone until the issue could be resolved. If you visited the document, it would display a "Forbidden" error message. Unfortunately due to the time it took to resolve, even Google cache started to get stale and stopped serving the cached contents.

Luckily, with the help of Alex Maier (VMTN Community Manager) and her team, she was able to get the ball rolling and got the fix tested and rolled out to production. The ghettoVCB document is once again alive and hopefully in no time it will be returned as the first search result on various search engines.

Going through the pain of receiving dozen of emails, private messages, tweets, etc. per week regarding the issue, I came to realize that VMTN document itself was not the right medium to host both content and user discussions. As it stands today, there are over 1,100+ comments which is pretty significant and managing and keeping up with the conversations is a pretty daunting task. I enjoy the feedback that community provides and the collaboration that takes place and I realize that this can be solved by using the new Groups feature.

To be honest, I did not spend much time looking at Groups when the VMTN software was upgraded, but now that the ghettoVCB document has been fixed, I realized this would fit this need perfectly. With the help of categories users can now post their feedback, discussions/issues and feature request and it can be easier consumed by both new users and myself. Starting today, I will have the following groups based on the top 5 most popular and active scripts:

ghettoVCB Group
ghettoVCBg2 Group
vmwarevSphereHealthCheck Group
vmwarevSphereSecurityHardening Group
ghettoUPSHostShutdown Group 

I have also disabled any new comments on these VMTN documents and will ask that all new comments be re-directed to respective VMware Groups. I'm currently working with Alex to see if there is an easy way to convert the existing comments into a document and attached that as a download to help minimize the complexity of the document. In the worse case, the comments will be left alone as read-only as I think the discussion that currently exists are invaluable. All other VMTN documents that I maintain in the vGhetto Repository will continue to use comments and depending on how well the groups go, I may migrate those over as well.

I hope these new groups will be beneficial for everyone and I am looking forward to the collaboration. Thanks for your support and please help spread the word!

Filed Under: Uncategorized Tagged With: ghettoVCB, ghettoVCBg2, health check script, security

How to extract host information from within a VM?

01/15/2011 by William Lam 28 Comments

From time to time, I see this question come up asking how one might be able to extract a certain piece of information from either ESX(i) or the management APIs (vSphere API) from within a virtual machine. The simple answer is you can not, by default the guest operating system has no idea of the underlying hypervisor nor does it have the access to the management APIs. This of course, assumes you are following VMware's best practices in isolated and segregating off your management network from your virtual machine network.

Having said that, there are certain bits of information that you can extract about your ESX(i) host from within the guestOS using some of the utilities that is installed with VMware Tools. The first utility is called VMware Toolbox command which can be found on both UNIX/Linux and Windows systems that have tools installed.

UNIX/Linux - /usr/bin/vmware-toolbox-cmd

Windows - C:\Program Files\VMware\VMware Tools\VMwareToolboxCmd.exe

This utility provide some information about ESX(i) and guestOS configuration including basic resource statistics.

One interesting sub-command is the resource statistics about both ESX(i) and guestOS such as speed of the hypervisor's CPUs or the current balloon, swap, memory limit, memory reservations, cpu limit or cpu reservations.

Here is an example of retrieving ESX(i) CPU speed.

As I mentioned before, you do not have access to the management network that your ESX(i) are on and that also means you do not have access to the vSphere APIs. What if you wanted to to associate a specific piece of information from ESX(i) and be able to access that piece of information from the guestOS? You can do so with the vmtoolsd (VMware Tools Daemon) utility.

UNIX/Linux - /usr/bin/vmtoolsd

Windows - C:\Program Files\VMware\VMware Tools\vmtoolsd.exe

This utility has an option called --cmd that allows you to run various commands including one that allows you to extract guest information using the "info-get" parameter. This guestinfo can be set using two methods:

1) Hard-coded within the virtual machine's .vmx configuration file
2) In VMX memory while a virtual machine is running within ESX(i)

Option 1 is pretty straight forward, you can add a custom attribute that has the following format:

guestinfo.[variable] = [value]

e.g.
guestinfo.provision.date = "01/11/2011"

You can use the vSphere Client to add this custom attribute while the virtual machine is powered off or you can manually edit the .vmx configuration file. If you use the latter, you will need to reload the VM's configuration, you can use vim-cmd vmsvc/reload.[vmid] to do so. One the VM is powered on, you can extract this piece of information, here is an example:

Option 2 is actually pretty interesting as the configuration of the custom guestinfo variable is not permanently stored. What I mean by this is the configuration is only persisted while the VM is running. This is interesting because if you have runtime information that potentially could change, this allows you to dynamically present this information to the guestOS. One use case could be set for management VMs such vCenter running in a VM and you wanted to know at any given time which ESX(i) host is currently managing it. This can be trivial with vCenter access, but what if the service is offline but the virtual machine is still running? Using the classic ESX Service Console command vmware-cmd you can loop through all powered on VMs and set some custom variables for example the current hostname of the ESX host managing the VM and version it is currently on.

The syntax for the command would be the following:

vmware-cmd [VM_VMX_PATH] setguestinfo [VARIABLE_NAME] [VARIABLE_VALUE]

e.g
vmware-cmd /vmfs/volumes/4a48004d-f9af7fa0-5bbf-003048d9586b/scofield/scofield.vmx setguestinfo hypervisor.hostname $(hostname)

Note: Notice, you do not need to append the keyword guestinfo, you just need to specify the variable name. Once you are in the guestOS, to access the custom variable, you will need to specify the full name which should be "guestinfo.[VARIABLE]"

As you can see this can be tedious to set for each and every VM, so let's automate this using a script that will go through all powered on VMs and set two custom variables called "hypervisor.hostname" which will set the current hostname of the ESX host and "hypervisor.build" which will retrieve the build of your ESX. To further automate this since you might have multiple ESX hosts running in a DRS cluster, you would store this script in a shared storage (VMFS and/or NFS) and setting up a cronjob that would run at some interval to always provide the latest information to the guestOS.

Here is an example of a shell scrip that does exactly that:

Shell
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
#!/bin/bash
 
IFS=$'\n'
 
for VM in $(vmware-cmd -l);
do
        VM_STATE=$(vmware-cmd "${VM}" getstate | awk -F "= " '{print $2}')
        if [ "${VM_STATE}" == "on" ]; then
                echo "Setting info for ${VM}"
                vmware-cmd "${VM}" setguestinfo hypervisor.hostname "$(hostname)"
                vmware-cmd "${VM}" setguestinfo hypervisor.build "$(vmware -v)"
        fi
done
 
unset IFS

You will want to save this script to your ESX host and make sure the script has executable permissions for it to execute. You will also create a cronjob based on how often you want the script to run and this needs to be configured for every ESX host that you would like to take part in this update.

Here is an example of running the script every hour:

0 * * * * /vmfs/volume/shared-storage/setGuestInfo.sh > /tmp/out

Note: I redirect the output to /tmp/out to ensure that script is in fact working and once you have, you can remove that all together.

Here is an example of extracting these two custom variables on both a Linux and Windows VM:

As I mentioned earlier, option 2 does not persist these custom variables with respect to the VM's configuration file. The variable configuration only takes affect when the VM is powered on and once it is powered off, the information is discarded. To support ESXi, you can use the various vSphere SDKs (vSphere SDK for Perl, PowerCLI, etc) but what you will find is that the behavior of setting this attribute is similar to that of the local vmware-cmd on classic ESX, it does not append the entry into the .vmx configuration file.

Here is a vSphere SDK for Perl script called addVMAdvParamOption.pl that can be used to guestinfo parameter for a given VM. Here is an example of this script:

If you are using vMA, you can also use the vCLI's vmware-cmd utility to set this variable. The format is slightly different than that of the classic ESX's vmware-cmd. Here is an example of this:

Notice, the variable name must include "guestinfo." keyword before specifying the variable name.

If you wanted to persist these configurations, you will need to adjust your provision workflow to append the variables into the VM's .vmx configuration file.

Filed Under: Uncategorized Tagged With: guestinfo, vmtoolsd, vmware tools, vmware-cmd

Ghetto Reflections 2010

12/30/2010 by William Lam 1 Comment

Looking back on 2010, it is hard to believe that virtuallyGhetto was created only 7 months ago. Instead of writing a long post, we thought we would share with you some of the highlights and favorite blog posts/scripts of 2010:

Here were the highlights for virtuallyGhetto in 2010:
May 31st - virtuallyGhetto says hello to the blogosphere
June 25th - virtuallyGhetto is part of the esteemed VMware Planet v12n feed
Sept 27th - virtuallyGhetto made the Top 25 VMware Bloggers List
Nov 19th - Veeam becomes first sponsor for virtuallyGhetto

Here were the top 10 blog posts of 2010 by page views:
Automating ESXi 4.1 Kickstart Tips & Tricks 9,914
ESXi 4.1 - Major Security Issue 4,564
Getting started with vMA 2,976
What is VMware vsish? 2,768
1200+ undocumented .vmx parameters 1,660
Automating vCloud Director and Oracle DB Installation 1,283
Script: Updated ghettoVCB and ghettoVCBg2 to Support vSphere 4.1 1,279
vMA 4.1 - Active Directory IntegrationTip 1,240
How to inject custom drivers into an ESXi 4.1 image using vibddi? 1,239
How to configure and use vMA's vi-fastpass with fpauth and adauth on vSphere 4.1 1,121

 

Here were the top 10 ghetto scripts of 2010 by page views:
ghettoVCB.sh 367,905
ghettoVCBg2.pl 66,683
vmwarevSphereHealthCheck.pl 62,861
ghettoShutdown.pl/upsVIShutdown.pl (DEPRECATED) 48,693
vmwareHealthCheck.pl 36,969
ghettoVCB-restore.sh 30,583
ghetto-esxi-linked-clones.sh 12,227
ghettoUPSHostShutdown.pl 7,820
vmwarevSphereSecurityHardeningReportCheck.pl 5,356
ghettoHostBackupManagement.pl 4,723

*Note: You may have noticed that the ghettoVCB VMTN document is currently inaccessible (displays "Forbidden" error). This is a known issue that was caused by the recent VMTN community upgrade by VMware. We apologize for any inconvenience this may have caused and we are hoping the issue will get resolved when VMware resumes after the holiday period. In the meanwhile, you can access the document via Google cache for the latest version of the script*

We also want to take this moment to thank our readers and the virtualization community for the support that you guys have given us through the comments on the blog, VMTN, linkage, twitter re-tweets, etc. There are two individuals that I would like to personally thank: Duncan Epping who has encouraged me on numerous occasions to start my own blog. In the end, it was the passion and dedication that Duncan put into his own blog to share with the community that really inspired me to start virtuallyGhetto. I would also like to thank Chris Wolf, who has been one of our first avid supporters of ghettoVCB and even today, he is still one of our largest advocate, providing honorable mentions even in his VMworld presentations!

We look forward to 2011 and hope to continue to provide great content and scripts to the VMware and virtualization community. We wish you happy holidays and a great New Year! See you all in 2011!

Filed Under: Uncategorized Tagged With: ghetto

Updated vSphere Health Check & Security Hardening Report Scripts

12/28/2010 by William Lam 1 Comment

Check out the minor update to the popular vSphere Health Check Script, now at v4.1.5, there was a minor typo issue that has now been resolved.

The vSphere 4.0 Security Hardening Guide has gotten a few updates from VMware since it's initial draft released back in January this year. The latest version includes a few revisions and my vSphere Security Hardening Report script v0.2 has not gotten an update since it's first release four days after VMware's draft. I finally found some time over the holiday to update the script and incorporate the feedback I received throughout the year. There are some new features and bug fixes and you can take a look at the change log more details. I encourage you to check out the new 1.0 version here.

Here is an updated sample report.
vmwarevSphereSecurityHardeningReport-SAMPLE.html

If you run into any issues or would like to report any bugs, please leave a message in the appropriate VMTN document.

Filed Under: Uncategorized Tagged With: health check script, security

How to identify the origin of a vSphere login?

12/24/2010 by William Lam 15 Comments

There was a pretty interesting post on the VMTN community forums recently in which a user was trying to identify a rogue vSphere login to their vCenter Server. Unfortunately, the actual user was not able to pin-point which system was logging in with his/her credentials. This potentially could have been some type of automated script that was configured and running and now has been forgotten. The vSphere admin tried to terminate the session which can be done using the vSphere Client or vSphere APIs, but the process would  be re-spawned automatically. 

Although the vSphere Session Manager provides some basic information such as the users logged in, their associated name, login time and their current status, it does not capture the source IP Address of the user. However, with small tweak in vCenter's logging option, you can easily track down a user or rogue client.

Before we get started, you first want to identify the username that you are interested in locating, you can easily do this by logging into the vSphere Client and going to Home->Administration->Sessions:

Let's say we're interested in the user "will.i.am" who is logged into our vSphere infrastructure from a rogue system and we would like to identify his/her source system. Next, you will need to correlate the user with his/her actual session. Anytime you login to vCenter/ESX(i) host, a session will be created and allocated for that particular user. The easiest way to locate the session key is to use the vSphere MOB which requires a web browser and enter the following URL: http://[your_vcenter_server]/mob?moid=SessionManager

Once logged in, you will be brought to the sessionManager page. Here you will see your current session (the MOB also generates a session) which is highlighted in green and all other sessions which may include vSphere Client logins, API/CLI logins/etc. From here, through the process of elimination you will need to go through the sessionList and locate the user and using the loginTime would be the most helpful. Once you have identified the proper user, you will want to record the session key. You will need to do this after you have enabled verbose logging which will be discussed in the next session.

In green you should see the username in which the user is logging in with and in red is the session key.
Now you are ready to locate this rogue user.

By default the vCenter Server logging option is set to "info" which does not contain any information about the client logins. You will need to temporarily change this from info to verbose and this can be done without restarting the vCenter Server. You will now login to the vSphere Client and click on "Administration" at the very top and click vCenter Server Setttings. Next you will click on "Logging Options" on the left pane and change the logging option.

Now we will need to login to the vCenter Server and look at the vpxd-X.log. You will either RDP or if your vCenter Server is running as a VM, you can use the remote console. You will need to go into the following directory: C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter\Logs which is where your vCenter logs will be stored at.

Now, let's say the rogue user is currently logged in and you know after terminating the session, he/she re-spawns the connection. What we will do is terminate the session and allow the rogue client to log back in and what we are after is the initial login details which will help us identify where the user is logging in from. You will need to open the latest vpxd-X.log file and scroll to the very bottom and search for the keyword "[Auth]" which should provide you with a line that includes the rogue username login.

Note: Depending if the rogue user logged in recently or awhile back, you may need to look through more than one of the vpxd-X.log files.

Depending on how verbose your environment is, you may have quite a bit of information in the logs. You use the threadID associated with this particular session to help you trace the lines you are interested in. You can find the threadID on the third column of each line and in this example, it is 02724. You can filter out entries that only contain this threadID to help you identify the rogue client.

As you can see we get quite a bit of detail about the user when we enable verbose logging.

Green - We see the username that is logging in.
Blue - We see the session key, this should match what you initially looked up (in my example I had to terminate the session, so it will not match)
Orange - We see the user agent is coming from browser and it's Chrome
Purple - We see the user was accessing the vSphere MOB
Red - Finally, we see the peer address which is the actual client address.

The above was executed on my desktop and by doing a simple DNS lookup assuming you have DNS resolution, you can track down the rogue user.

What is also interesting is you can tell not only where a user is logging in from, but how they are accessing your vSphere environment by looking at the user agent information. We already know if you are using the vSphere MOB or webAccess, the user agent will display browser information. Here are some others from some simple login tests:

User Agent Client
VMware VI Client/4.x.x vSphere Client
Java/1.6.0_20 VI Java
VI Perl vSphere SDK for Perl
Mozilla/4.0 (compatible; MSIE 6.0; MS Web Services Client Protocol 2.0.50727.4952) PowerCLI

Note: The PowerCLI entry, I am not 100% sure, that is what I received when using Connect-VIServer to our vCenter Server

Another method of tracking down a rogue vSphere login is using simple netstat and identifying any entries that show the Local Address of your vCenter Server IP Address mapping to port 443 which is used for communication. You will then identify the Foreign Address to validate all clients.

As you can see my desktop address of 172.30.0.218 is included in that list along with few others. If you know which systems should have access, then you can easily narrow down the rogue client's address.

Remember once you are done hunting your rogue user to revert the vCenter logging option back to it's original configuration else you may rotate through your vpxd logs pretty quickly.

Filed Under: Uncategorized Tagged With: api, login, mob, session, vSphere

  • « Go to Previous Page
  • Go to page 1
  • Interim pages omitted …
  • Go to page 188
  • Go to page 189
  • Go to page 190
  • Go to page 191
  • Go to page 192
  • Interim pages omitted …
  • Go to page 202
  • Go to Next Page »

Primary Sidebar

Author

William Lam is a Senior Staff Solution Architect working in the VMware Cloud team within the Cloud Services Business Unit (CSBU) at VMware. He focuses on Automation, Integration and Operation for the VMware Cloud Software Defined Datacenters (SDDC)

  • Email
  • GitHub
  • LinkedIn
  • RSS
  • Twitter
  • Vimeo

Sponsors

Privacy & Cookies: This site uses cookies. By continuing to use this website, you agree to their use.
To find out more, including how to control cookies, see here: Cookie Policy