mob

vSphere MoRef (Managed Object Reference) ID Finder Script

11/25/2011 by William Lam 20 Comments

There was an interesting article this morning by my colleague Dave Hill about Looking up Managed Object Reference (MoRef) in vCenter Server, which references a VMware KB article showing you how to use vSphere MOB to find MoRef IDs.

A Managed Object Reference ID also known just as MoRef ID is a unique value that is generated by the vCenter Server and is guaranteed to be unique for a given entity in a single vCenter instance. Steve Jin has a great article going into detail about MoRef's, that you should check out here. MoRef ID's are not only used in vSphere, but they can also be referenced in other VMware products such as vCD (vCloud Director) or SRM to name a few or even by 3rd party/custom tools as a way to uniquely track objects within vCenter.

Using the vSphere MOB is one way of retrieving the MoRef ID, but of course this can be tedious if you are trying to locate MoRef's for multiple entities, in multiple vCenters. I decided to write a quick vSphere SDK for Perl script called moRefFinder.pl that allows a users to quickly query for a variety of Managed Objects (VM,Host,Cluster,Datacenter,Resource Pool,Network, Distributed vSwitch, Folder, vApp and Datastore) in a vCenter Server.

I also noted earlier, MoRef ID is only unique within a vCenter instance, so how do you track these objects across multiple vCenters? Well, VMware introduced a new property called instanceUUID (128 bit UUID) which is unique for a given vCenter Server and you can use this along with the MoRef ID to uniquely track objects across multiple vCenters and the script automatically outputs this value for any query.

Note: There have been some optimization in the latest vSphere SDK for Perl 5.0, it is recommended you use the latest version which is backwards compatible with vSphere 3.x and 4.x

The script requires two parameters:

type - The type of managed object
name - The name of the object as seen in vCenter Server

Here is an example querying for a Virtual Machine:

Here is an example querying for a Host:

Here is an example querying for a Cluster:

Here is an example querying for a Datacenter:

Here is an example querying for a Resource Pool:

Here is an example querying for a Network (Portgroup):

Here is an example querying for a Distributed vSwitch:

Here is an example querying for a Folder:

Here is an example querying for a vApp:

Here is an example querying for a Datastore:

There’s a new mob in town, FDM MOB for ESXi 5

07/15/2011 by William Lam 1 Comment

That's right, vSphere is not the only one with a MOB, the new FDM (Fault Domain Manager) feature also includes a MOB view on an ESXi 5.0 hosts that is part of an FDM/HA enabled cluster. I originally noticed this new URL while parsing through the systems logs an ESXi host to get a better understanding of the startup process and found this little nugget. This page contains private APIs that are currently not exposed for public consumption with respect to FDM service, please use at your own risk.

To access the FDM MOB, you will need to point your browser to the following URL:

https://[esxi5_hostname]/mobfdm

Here is a screenshot of the main summary page:

On the summary page, you have some basic information about the particular host in question, one interesting property is the "clusterState" which will be either a master or slave node, this can be useful in troubleshooting if you do not have access to vCenter

The are two interesting methods that can provide some useful information: RetrieveClusterInfo and RetrieveHostList which should be pretty self explanatory in what they will be doing.

To generate the URL for the RetrieveClusterInfo you will need to point your browser to the following URL:

https://[esxi5_hostname]/mobfdm/?moid=fdmService&method=retrieveClusterInfo

As you can see from the screenshot, it provides a summary for the particular ESXi host within the FDM cluster, including the masterID, this ID will be useful when we call the other method to identify the master node in the FDM cluster.

To generate the URL for the RetrieveHostList you will need to point your browser to the following URL:

https://[esxi5_hostname]/mobfdm/?moid=fdmService&method=retrieveHostList

This method extracts all hosts from the FDM cluster and provides quite a bit of information about each host including the hostname and also the hostID. You can now translate ID found in the last method to identify the master node of the FDM cluster.

When you login to the FDM MOB for an ESXi host that is a master node in the cluster, the page will look slightly different with even more details including all slave nodes and protected VMs within the cluster.

As you can see this can be a useful tool for quickly identifying the master and slave nodes within an FDM cluster without going to your vCenter Server.

You can also get this information within the ESXi Shell, there is a hostlist file in an XML format that you can view the same information found in the RetrieveClusterInfo method located in /etc/opt/vmware/fdm/hostlist

~ # cat /etc/opt/vmware/fdm/hostlist
host-70
FB43716F-84A5-45AD-A5BB-F08BC64148DF-14-5db552f-vcenter50-133host-205esxi50-2.primp-industries.com58:C9:81:F1:3D:A1:47:B8:7A:C0:33:93:71:3A:B9:A1:51:AD:25:51172.30.0.7300:19:bb:26:25:8e00:19:bb:26:25:7e/vmfs/volumes/664220b6-9628e4e3/vmfs/volumes/f0613bc2-56e80c59443host-70esxi50-1.primp-industries.com25:C3:FE:23:B1:DB:5C:F8:94:13:A3:CD:B0:DC:EA:51:72:F1:53:4F172.30.0.7200:1f:29:c9:48:e200:1f:29:c9:48:f8/vmfs/volumes/664220b6-9628e4e3/vmfs/volumes/f0613bc2-56e80c59443

You also get the details of RetrieveHostList and cleaner output of the FDM host using the following script: /opt/vmware/fdm/fdm/prettyPrint.sh. The script can accept three different arguments: hostlist, clusterconfig and compatlist

Here is a screenshot of the hostlist:

Here is a screenshot of the clusterconfig:

Here is screenshot the compatlist:

Automating Active Directory User Management in ESXi Kickstart

02/24/2011 by William Lam 2 Comments

In the previous post we looked at Automating Active Directory Domain Join in ESXi Kickstart. We are now going to look at adding domain users without having to manually go through vSphere Client or external scripts after an ESXi host has been provisioned. We are going to be leveraging vim-cmd to accomplish this during the kickstart installation. Before doing so, you will need to know the the available roles on a default ESXi host and the syntax for a given permission.

To see the available roles, you can run the following command on an already provisioned ESXi host:

vim-cmd vimsvc/auth/roles | less

The default roles on an ESXi host are:

NoAccess
ReadOnly
Admin

To see the existing permissions, you can run the following command on an already provisioned ESXi host:

vim-cmd vimsvc/auth/permissions

These entries will match what you see in the vSphere Client and dictate who has login access to the ESXi host.

To add a new permission, we will be using the "vim-cmd vimsvc/auth/entity_permission_add" and it requires five parameters.

First - Entity (This can be found by looking at the output from permissions)
Second - Username
Third - Boolean on whether this is a group or not (should be false)
Fourth - The role to be applied to the user
Fifth - Boolean on whether to propagate this permission

If you manually add a domain user, you can easily verify the user can login by running the "id" command which will perform a look up on the user. If it is successful, it should return an entry corresponding to something like this:

Note: We need to use the double slash "\" to escape the initial slash when running the query. Also make note of the domain name as it may or may not match your full domain name.

We are now ready to craft a simple script that will add domain users as part of the ESXi kickstart process. The following snippet should be placed in the %firstboot section of your kickstart and after your Active Directory domain join code. Make sure you replace the DOMAIN_NAME variable along with the usernames. In the example I have two separate for loops to handle ReadOnly and Admin users, you do not need both if you are only adding one type of users.

The script basically performs a simple 60sec sleep to ensure the domain join process has completed before continuing. If you do not place a sleep, the subsequent code will fail to execute. The next step is to validate the user by doing a simple lookup using "id" command and upon successful look up of the user, we add the appropriate permissions.

Note: We only have two add these two entities: "vim.Folder:ha-folder-root" and "vim.ComputeResource:ha-compute-res" to properly add a permission.

If everything was successful, after your ESXi installation you now should have your host joined to your Active Directory and a list of domain users who now have permission to login to the ESXi host. You can verify by using the vSphere Client and taking a look at the Permissions tab.

If you would like to create custom roles on your ESXi host, you can use the following command:

vim-cmd vimsvc/auth/role_add

Note: The syntax for the privileges parameter lists only five, but it actually accepts as many as you need with the custom role

Automating Active Directory Domain Join in ESX(i) Kickstart

02/23/2011 by William Lam 10 Comments

I recently received an email asking if it was possible to automate the joining of an ESXi 4.1 host to Active Directory within a kickstart installation. The simple answer is yes; you can easily do so using the same trick found in tip #7 in Automating ESXi 4.1 Kickstart Tips & Tricks post which connects locally to the vSphere MOB using python.

Having said that, there is a small caveat to this solution in which the credentials used to join the ESX(i) host must be exposed in clear-text, this may or may not be acceptable. A potential way of getting this to work is to create a dedicated windows service account with limited privileges to add only ESXi hosts to your domain.

The reason for the use of clear-text password is the vSphere API method that is used to join an ESX(i) host to AD domain, joinDomain(), only supports plain text.

Ideally, the password parameter should accept either an encrypted hash or some type of certificate which can be validated by Active Directory server. The actual solution is pretty straight forward, by crafting the appropriate call to vSphere MOB, we are able to generate a python script during the %firstboot section of ESX(i) kickstart installation which will execute upon the initial boot up of the host.

Update (03/26/11): Thanks to VMTN user klich who has found a solution to embed the password of the Active Directory user in a base64 encoding so that it is not visible in plain text for anyone who has access to the kickstart configuration file. To create the encoded hash, you will need access to either a ESX(i) or UNIX/Linux system which has python interpreter installed.

You will need to run the following command and substitute the password you wish to encode.

python -c "import base64;
print base64.b64encode('MySuperSecurePasswordYo')"

The output will be your encoded hash:

Note: Make sure your ESX(i) hostname is configured with a FQDN (Fully Qualified Domain Name), else you will get an error when trying to join to AD domain.

The following snippet should be added to your %firstboot section of your kickstart. Remember to replace the variables: domainname, ad_username and encodedpassword with your configurations and make sure to leave the password variable blank.

ESX(i) 4.1

cat > /tmp/joinActiveDirectory.py << __JOIN_AD__

import sys,re,os,urllib,urllib2

# MOB url
url = "https://localhost/mob/?moid=ha-ad-auth&method=joinDomain"

# mob login credentials -- use password = "" for build scripting
username = "root"
password = ""

# which domain to join, and associated OU
# e.g.
#       "primp-industries.com"
#       "primp-industries.com/VMware Server OU"
domainname = "primp-industries.com"

# active directory credentials using encoded base64 password
ad_username = "*protected email*"
encodedpassword = "TXlTdXBlclNlY3VyZVBhc3N3b3JkWW8"
ad_password = base64.b64decode(encodedpassword)

#auth
passman = urllib2.HTTPPasswordMgrWithDefaultRealm()
passman.add_password(None,url,username,password)
authhandler = urllib2.HTTPBasicAuthHandler(passman)
opener = urllib2.build_opener(authhandler)
urllib2.install_opener(opener)

#execute method
params = {'domainName':domainname,'userName':ad_username,'password':ad_password}
e_params = urllib.urlencode(params)
req = urllib2.Request(url,e_params)
page = urllib2.urlopen(req).read()
__JOIN_AD__

#execute python script to Join AD
python /tmp/joinActiveDirectory.py

cat > /tmp/joinActiveDirectory.py << __JOIN_AD__

import sys,re,os,urllib,urllib2

# MOB url

url = "https://localhost/mob/?moid=ha-ad-auth&method=joinDomain"

# mob login credentials -- use password = "" for build scripting

username = "root"

password = ""

# which domain to join, and associated OU

# e.g.

# "primp-industries.com"

# "primp-industries.com/VMware Server OU"

domainname = "primp-industries.com"

# active directory credentials using encoded base64 password

ad_username = "*protected email*"

encodedpassword = "TXlTdXBlclNlY3VyZVBhc3N3b3JkWW8"

ad_password = base64.b64decode(encodedpassword)

#auth

passman = urllib2.HTTPPasswordMgrWithDefaultRealm()

passman.add_password(None,url,username,password)

authhandler = urllib2.HTTPBasicAuthHandler(passman)

opener = urllib2.build_opener(authhandler)

urllib2.install_opener(opener)

#execute method

params = {'domainName':domainname,'userName':ad_username,'password':ad_password}

e_params = urllib.urlencode(params)

req = urllib2.Request(url,e_params)

page = urllib2.urlopen(req).read()

__JOIN_AD__

#execute python script to Join AD

python /tmp/joinActiveDirectory.py

If you are using ESX(i) 4.1 Update 1 or if you run into the "urllib2.HTTPError: HTTP Error 403: Forbidden: Possible Cross-Site Request Forgery" you will need to use the following snippet below which includes the session cookie as part of the request to the vSphere MOB due to a recent CSRF patch from VMware identified by VMTN user klitch.

ESX(i) 4.1 Update 1

cat > /tmp/joinActiveDirectory.py << __JOIN_AD__

import sys,re,os,urllib,urllib2,base64

# mob url
url = "https://localhost/mob/?moid=ha-ad-auth&method=joinDomain"

# mob login credentials -- use password = "" for build scripting
username = "root"
password = ""

# which domain to join, and associated OU
# e.g.
#       "primp-industries.com"
#       "primp-industries.com/VMware Server OU"
domainname = "primp-industries.com"

# active directory credentials using encoded base64 password
ad_username = "*protected email*"
encodedpassword = "TXlTdXBlclNlY3VyZVBhc3N3b3JkWW8"
ad_password = base64.b64decode(encodedpassword)

# Create global variables
global passman,authhandler,opener,req,page,page_content,nonce,headers,cookie,params,e_params

# Code to build opener with HTTP Basic Authentication
passman = urllib2.HTTPPasswordMgrWithDefaultRealm()
passman.add_password(None,url,username,password)
authhandler = urllib2.HTTPBasicAuthHandler(passman)
opener = urllib2.build_opener(authhandler)
urllib2.install_opener(opener)

### Code to capture required page data and cookie required for post back to meet CSRF requirements  ###
req = urllib2.Request(url)
page = urllib2.urlopen(req)
page_content= page.read()

# regex to get the vmware-session-nonce value from the hidden form entry
reg = re.compile('name="vmware-session-nonce" type="hidden" value="?([^\s^"]+)"')
nonce = reg.search(page_content).group(1)

# get the page headers to capture the cookie
headers = page.info()
cookie = headers.get("Set-Cookie")

# Code to join the domain
params = {'vmware-session-nonce':nonce,'domainName':domainname,'userName':ad_username,'password':ad_password}
e_params = urllib.urlencode(params)
req = urllib2.Request(url, e_params, headers={"Cookie":cookie})
page = urllib2.urlopen(req).read()

__JOIN_AD__

#execute python script to Join AD
python /tmp/joinActiveDirectory.py

cat > /tmp/joinActiveDirectory.py << __JOIN_AD__

import sys,re,os,urllib,urllib2,base64

# mob url

url = "https://localhost/mob/?moid=ha-ad-auth&method=joinDomain"

# mob login credentials -- use password = "" for build scripting

username = "root"

password = ""

# which domain to join, and associated OU

# e.g.

# "primp-industries.com"

# "primp-industries.com/VMware Server OU"

domainname = "primp-industries.com"

# active directory credentials using encoded base64 password

ad_username = "*protected email*"

encodedpassword = "TXlTdXBlclNlY3VyZVBhc3N3b3JkWW8"

ad_password = base64.b64decode(encodedpassword)

# Create global variables

global passman,authhandler,opener,req,page,page_content,nonce,headers,cookie,params,e_params

# Code to build opener with HTTP Basic Authentication

passman = urllib2.HTTPPasswordMgrWithDefaultRealm()

passman.add_password(None,url,username,password)

authhandler = urllib2.HTTPBasicAuthHandler(passman)

opener = urllib2.build_opener(authhandler)

urllib2.install_opener(opener)

### Code to capture required page data and cookie required for post back to meet CSRF requirements ###

req = urllib2.Request(url)

page = urllib2.urlopen(req)

page_content= page.read()

# regex to get the vmware-session-nonce value from the hidden form entry

reg = re.compile('name="vmware-session-nonce" type="hidden" value="?([^\s^"]+)"')

nonce = reg.search(page_content).group(1)

# get the page headers to capture the cookie

headers = page.info()

cookie = headers.get("Set-Cookie")

# Code to join the domain

params = {'vmware-session-nonce':nonce,'domainName':domainname,'userName':ad_username,'password':ad_password}

e_params = urllib.urlencode(params)

req = urllib2.Request(url, e_params, headers={"Cookie":cookie})

page = urllib2.urlopen(req).read()

__JOIN_AD__

#execute python script to Join AD

python /tmp/joinActiveDirectory.py

After your ESXi host has completed installation, you can login to the ESX(i) host using the vSphere Client and you should see a recent successful task on the domain join process.

You can also verify by clicking on Configurations->Authentication Services and verify that your ESX(i) host is now part of the Active Directory domain.

In the next post, I will go over the details of adding domain users within the ESX(i) kickstart.

As a side note, it is quite unfortunate that ESXi only supports Microsoft Active Directory and does not integrate with other well known directory services such as NIS/NIS+, Kerberos, OpenLDAP, eDirectory, etc; this was actually possible with classic ESX. VMware has continued to focus on Windows-centric solutions and neglect the UNIX/Linux community by not supporting OS agnostic management tools that integrate with their vSphere platform. I hope this will change in the future for true interoperability.

How to identify the origin of a vSphere login?

12/24/2010 by William Lam 15 Comments

There was a pretty interesting post on the VMTN community forums recently in which a user was trying to identify a rogue vSphere login to their vCenter Server. Unfortunately, the actual user was not able to pin-point which system was logging in with his/her credentials. This potentially could have been some type of automated script that was configured and running and now has been forgotten. The vSphere admin tried to terminate the session which can be done using the vSphere Client or vSphere APIs, but the process would be re-spawned automatically.

Although the vSphere Session Manager provides some basic information such as the users logged in, their associated name, login time and their current status, it does not capture the source IP Address of the user. However, with small tweak in vCenter's logging option, you can easily track down a user or rogue client.

Before we get started, you first want to identify the username that you are interested in locating, you can easily do this by logging into the vSphere Client and going to Home->Administration->Sessions:

Let's say we're interested in the user "will.i.am" who is logged into our vSphere infrastructure from a rogue system and we would like to identify his/her source system. Next, you will need to correlate the user with his/her actual session. Anytime you login to vCenter/ESX(i) host, a session will be created and allocated for that particular user. The easiest way to locate the session key is to use the vSphere MOB which requires a web browser and enter the following URL: http://[your_vcenter_server]/mob?moid=SessionManager

Once logged in, you will be brought to the sessionManager page. Here you will see your current session (the MOB also generates a session) which is highlighted in green and all other sessions which may include vSphere Client logins, API/CLI logins/etc. From here, through the process of elimination you will need to go through the sessionList and locate the user and using the loginTime would be the most helpful. Once you have identified the proper user, you will want to record the session key. You will need to do this after you have enabled verbose logging which will be discussed in the next session.

In green you should see the username in which the user is logging in with and in red is the session key.
Now you are ready to locate this rogue user.

By default the vCenter Server logging option is set to "info" which does not contain any information about the client logins. You will need to temporarily change this from info to verbose and this can be done without restarting the vCenter Server. You will now login to the vSphere Client and click on "Administration" at the very top and click vCenter Server Setttings. Next you will click on "Logging Options" on the left pane and change the logging option.

Now we will need to login to the vCenter Server and look at the vpxd-X.log. You will either RDP or if your vCenter Server is running as a VM, you can use the remote console. You will need to go into the following directory: C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter\Logs which is where your vCenter logs will be stored at.

Now, let's say the rogue user is currently logged in and you know after terminating the session, he/she re-spawns the connection. What we will do is terminate the session and allow the rogue client to log back in and what we are after is the initial login details which will help us identify where the user is logging in from. You will need to open the latest vpxd-X.log file and scroll to the very bottom and search for the keyword "[Auth]" which should provide you with a line that includes the rogue username login.

Note: Depending if the rogue user logged in recently or awhile back, you may need to look through more than one of the vpxd-X.log files.

Depending on how verbose your environment is, you may have quite a bit of information in the logs. You use the threadID associated with this particular session to help you trace the lines you are interested in. You can find the threadID on the third column of each line and in this example, it is 02724. You can filter out entries that only contain this threadID to help you identify the rogue client.

As you can see we get quite a bit of detail about the user when we enable verbose logging.

Green - We see the username that is logging in.
Blue - We see the session key, this should match what you initially looked up (in my example I had to terminate the session, so it will not match)
Orange - We see the user agent is coming from browser and it's Chrome
Purple - We see the user was accessing the vSphere MOB
Red - Finally, we see the peer address which is the actual client address.

The above was executed on my desktop and by doing a simple DNS lookup assuming you have DNS resolution, you can track down the rogue user.

What is also interesting is you can tell not only where a user is logging in from, but how they are accessing your vSphere environment by looking at the user agent information. We already know if you are using the vSphere MOB or webAccess, the user agent will display browser information. Here are some others from some simple login tests:

User Agent	Client
VMware VI Client/4.x.x	vSphere Client
Java/1.6.0_20	VI Java
VI Perl	vSphere SDK for Perl
Mozilla/4.0 (compatible; MSIE 6.0; MS Web Services Client Protocol 2.0.50727.4952)	PowerCLI

Note: The PowerCLI entry, I am not 100% sure, that is what I received when using Connect-VIServer to our vCenter Server

Another method of tracking down a rogue vSphere login is using simple netstat and identifying any entries that show the Local Address of your vCenter Server IP Address mapping to port 443 which is used for communication. You will then identify the Foreign Address to validate all clients.

As you can see my desktop address of 172.30.0.218 is included in that list along with few others. If you know which systems should have access, then you can easily narrow down the rogue client's address.

Remember once you are done hunting your rogue user to revert the vCenter logging option back to it's original configuration else you may rotate through your vpxd logs pretty quickly.

What’s new in the vSphere 4.1 MOB

07/13/2010 by William Lam 13 Comments

The vSphere MOB (Managed Object Browser) is not a well known tool and unless you are a developer, you probably do not know of its existence. The simplest way to describe the vSphere MOB is that it is a debugging tool to graphical visualize all entities within the vSphere API through a web browser. It is often used to better understand the vSphere API/inventory and aide in developing your own vSphere application or script. For a detailed deep dive of the vSphere inventory, check out Steve Jin's post here.

The vSphere MOB can be reached by pointing your web browser to either an ESX(i) or vCenter host followed by the path "/mob". You will be prompted for credentials as you normally would prior to gaining access to the MOB.

The vSphere MOB may not always be the easiest to navigate. Understanding the vSphere inventory structure is definitely helpful, otherwise it can be very confusing to users. With the release of vSphere 4.1, there are two major changes in the MOB that can help make the navigation easier for developers.

Previous to vSphere 4.1, the MOB would display managed entities using their Managed Object IDs (MoRef ID) as a way to uniquely identify these objects. This made it very difficult for developers to identify the specific object you are viewing without having to dig further.

In vSphere 4.1, the MOB provides both the MoRef ID and the translated the names of the managed entities.

The vSphere MOB not only displays information about the managed entities but also provides a way to interact with the system. It does this by listing the available API methods for each managed entity at the bottom of the page and can be executed by providing the correct input parameters. The issue arises when trying to execute methods which requires complex input that require more than a simple string or integer value. The format of the input is XML-based and it can be challenging to generate the correct format, especially without any examples.

In the new version of the MOB, you are now presented with an XML template of the required parameters which can then be filled out to execute the method.

The following example will demonstrate the change of an ESXi advanced configuration value for configuring a syslog server:

Once you have filled in the required parameters, you will need to click on the "Invoke Method" to execute the specific method.

Here we verify the advanced configuration key "Syslog.Remote.Hostname" has been changed and is set to our new syslog server.

I think these two changes will make it easier for developers to navigate the vSphere MOB and assist with understanding the vSphere API and it's managed entities.