session

An update on how to retrieve useful information from a vSphere login?

11/07/2016 by William Lam 4 Comments

There was an internal Socialcast question today in which the answer could be found in my how to identify the origin of a vSphere login article. After responding to the question, I had realized that I wrote that article almost 6 years ago and what is even more crazy is that it is still very applicable today. The article explains how you can identify a vSphere login by enabling the "trivial" logging option in vCenter Server (extremely verbose, so please use with caution). Once enabled, you can go through the vpxd.log file and find things about a user login such as the the IP Address of the client as well as the type of vSphere interface they had used to login to whether that is using the vSphere C# Client or PowerCLI for example. Although this extracted information can be very useful, the process to retrieve this is not very ideal, especially having to increase your vCenter Server logging verbosity to the extreme which can force other more critical log events to roll over.

Given that this article written back when vSphere 4.1 was still the current release, I figure I should give the process another look to see if there was a better method in retrieving this information. While quickly browsing around the SessionManager object and specifically the UserSession property, I noticed there have been quite a few enhancements that were introduced in vSphere 5.1. It looks like you can now easily retrieve things like the User Agent, IP Address of the client as well as the number of API invocations for anyone who is currently logged into a given vSphere environment. Perhaps someone internally saw my blog post and thought it would be useful to add these properties directly into the vSphere API rather than poking around in the verbose logs 😀

To exercise these new vSphere APIs, I have create a quick PowerCLI function called Get-vSphereLogins The script will iterate through all currently logged in vSphere sessions and provide the following output: Username, IP Address, API Count & Login Time. It also excludes the current session initiating the query as well as any of the VC Extension logins. Here is a screenshot of my environment using several different vSphere API interfaces to login to my vSphere environment:

With the information above, not only can you tell who is logging in but also where (IP Address) and most importantly how (User Agent) they are logging in. One thing to be aware of is that the User Agent is not always populated and even if it is, it may not provide you with enough information on the specific interface a given user is logging in from. For example, it looks like a script written using the vSphere SDK for Python does not actually set the User Agent, so it is empty.

Here is an updated table using some of the latest vSphere interfaces to log into a vSphere 6.0 Update 2 environment and their respective observed User Agents:

Interface	User Agent
vSphere C# Client	VMware vSphere Client/6.0.0
vSphere Web Client	VMware vim-java 1.0
vSphere MOB	Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML like Gecko) Chrome/54.0.2840.71 Safari/537.36
PowerCLI	PowerCLI/6.5.0
vSphere SDK for Perl	VI Perl
vSphere SDK for Ruby (rbvmomi)	Ruby
vSphere SDK for Python (pyvmomi)	None

Note: In vSphere 6.5, the User Agent that is returned for the vSphere Web Client session looks to be using web-client/6.5.0

Finally, saving the best for last. The VMware Engineer(s) not only added these new properties into the vSphere API, but they have also made them readily available using the vSphere Web Client. To view all the session information, navigate to your vCenter Server instance and under Manage->Sessions you can get the exact same view as using the vSphere API. By default, the IP Address, User Agent & API Invocations are hidden by default. You just need to right click on the table header and add those additional field as shown in the screenshot below.

Longer term, it would be great to see that each of the "official" VMware CLI/SDKs as well as other interfaces can uniquely identify themselves with a well defined string. This not only helps with understanding the types of tools customers are using but also helps with any types of internal audits customers may require. If you think this would be useful to have, please feel free to leave a comment or any other things you feel would be useful to include.

Configuring New vSphere Web Client Session Timeout

09/10/2012 by William Lam 5 Comments

Just like you could in the old vSphere C# Client, users can also configure a session timeout for the new vSphere Web Client in the latest release of vSphere 5.1. This not only ensures that idle sessions automatically disconnect after a certain period of time but also helps reduce the resources consumed on the vCenter Server as each session allocates a certain amount of resources.

To configure the session timeout, you will need to login to the server running the vSphere Web Client service (which is usually your vCenter Server) and find the webclient.properties file and change the default timeout and then restart the vSphere Web Client service. For the VCSA, the default timeout value is 120 minutes and I assume it is the same default for the Windows vCenter Server.

Step 1 - Locate the webclient.properties file:

VCSA 5.x

/var/lib/vmware/vsphere-client/webclient.properties

VCSA 6.x

/etc/vmware/vsphere-client/webclient.properties

Windows vCenter Server 5.x

%ALLUSERPROFILE%\VMware\vCenterServer\cfg\vsphere-client\webclient.properties

Windows vCenter Server 6.x

%ALLUSERPROFILE%\VMware\vSphere Web Client\webclient.properties

Step 2 - Un-comment and change session.timeout value to desired value:

session.timeout = 120

Step 3 - Restart the vSphere Web Client Service:

/etc/init.d/vsphere-client restart

You will need to restart the vSphere Web Client service for the changes to go into effect. For the Windows vCenter Server, just restart the vSphere Web Client service and for the VCSA, run the above command.

In my lab, I configured the time out to be 1 minute, once the session has gone idle for the configured period, you will automatically be logged out and brought back to login page with the following message:

Identifying Idle vCenter Sessions

12/20/2011 by William Lam 3 Comments

I recently received a question on how to identify idle vCenter sessions that may stem from a non-active vSphere Client or vSphere API connection. Properly managing sessions to your vCenter Server is as important as knowing who is logging in. If you exhaust the maximum number of connections, you can easily deny login access to other users and/or services that are needed to connect to vCenter Server which can lead to very bad things. In the worse case, you can even prevent administrators from logging into a new session to help terminate any stale/idle connections.

The vSphere API provides a sessionManagement manager to help you manage all sessions, including checking whether a session is still active and terminating a session. I wrote a quick vSphere SDK for Perl script called sessionManagement.pl which allows you to list all sessions for a given vCenter and terminating a a specific session. Every session has a unique sessionKey that the vCenter Server uses to track and it provides information such username, login time and last active which can be used to create an unattended script to automatically terminate sessions that have gone beyond a certain threshold.

Here is an example script output of performing the "list" operation:

Note: There maybe vCenter Plugins/Extensions that creates sessions such as CapacityIQ, those will be denoted by "vCenter Ext Session" field. These should not be terminated as it can negatively impact 3rd party applications.

Here is an example script output of terminating a specific session using the "disconnect" operation which you will need to specify sessionkey provided in the previous command:

For those of you that are interested in creating your own script to terminate idle sessions, you can use the DateTime Perl module to help with calculating the amount of idle time and comparing that to a pre-defined timeout value.

How to identify the origin of a vSphere login?

12/24/2010 by William Lam 15 Comments

There was a pretty interesting post on the VMTN community forums recently in which a user was trying to identify a rogue vSphere login to their vCenter Server. Unfortunately, the actual user was not able to pin-point which system was logging in with his/her credentials. This potentially could have been some type of automated script that was configured and running and now has been forgotten. The vSphere admin tried to terminate the session which can be done using the vSphere Client or vSphere APIs, but the process would be re-spawned automatically.

Although the vSphere Session Manager provides some basic information such as the users logged in, their associated name, login time and their current status, it does not capture the source IP Address of the user. However, with small tweak in vCenter's logging option, you can easily track down a user or rogue client.

Before we get started, you first want to identify the username that you are interested in locating, you can easily do this by logging into the vSphere Client and going to Home->Administration->Sessions:

Let's say we're interested in the user "will.i.am" who is logged into our vSphere infrastructure from a rogue system and we would like to identify his/her source system. Next, you will need to correlate the user with his/her actual session. Anytime you login to vCenter/ESX(i) host, a session will be created and allocated for that particular user. The easiest way to locate the session key is to use the vSphere MOB which requires a web browser and enter the following URL: http://[your_vcenter_server]/mob?moid=SessionManager

Once logged in, you will be brought to the sessionManager page. Here you will see your current session (the MOB also generates a session) which is highlighted in green and all other sessions which may include vSphere Client logins, API/CLI logins/etc. From here, through the process of elimination you will need to go through the sessionList and locate the user and using the loginTime would be the most helpful. Once you have identified the proper user, you will want to record the session key. You will need to do this after you have enabled verbose logging which will be discussed in the next session.

In green you should see the username in which the user is logging in with and in red is the session key.
Now you are ready to locate this rogue user.

By default the vCenter Server logging option is set to "info" which does not contain any information about the client logins. You will need to temporarily change this from info to verbose and this can be done without restarting the vCenter Server. You will now login to the vSphere Client and click on "Administration" at the very top and click vCenter Server Setttings. Next you will click on "Logging Options" on the left pane and change the logging option.

Now we will need to login to the vCenter Server and look at the vpxd-X.log. You will either RDP or if your vCenter Server is running as a VM, you can use the remote console. You will need to go into the following directory: C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter\Logs which is where your vCenter logs will be stored at.

Now, let's say the rogue user is currently logged in and you know after terminating the session, he/she re-spawns the connection. What we will do is terminate the session and allow the rogue client to log back in and what we are after is the initial login details which will help us identify where the user is logging in from. You will need to open the latest vpxd-X.log file and scroll to the very bottom and search for the keyword "[Auth]" which should provide you with a line that includes the rogue username login.

Note: Depending if the rogue user logged in recently or awhile back, you may need to look through more than one of the vpxd-X.log files.

Depending on how verbose your environment is, you may have quite a bit of information in the logs. You use the threadID associated with this particular session to help you trace the lines you are interested in. You can find the threadID on the third column of each line and in this example, it is 02724. You can filter out entries that only contain this threadID to help you identify the rogue client.

As you can see we get quite a bit of detail about the user when we enable verbose logging.

Green - We see the username that is logging in.
Blue - We see the session key, this should match what you initially looked up (in my example I had to terminate the session, so it will not match)
Orange - We see the user agent is coming from browser and it's Chrome
Purple - We see the user was accessing the vSphere MOB
Red - Finally, we see the peer address which is the actual client address.

The above was executed on my desktop and by doing a simple DNS lookup assuming you have DNS resolution, you can track down the rogue user.

What is also interesting is you can tell not only where a user is logging in from, but how they are accessing your vSphere environment by looking at the user agent information. We already know if you are using the vSphere MOB or webAccess, the user agent will display browser information. Here are some others from some simple login tests:

User Agent	Client
VMware VI Client/4.x.x	vSphere Client
Java/1.6.0_20	VI Java
VI Perl	vSphere SDK for Perl
Mozilla/4.0 (compatible; MSIE 6.0; MS Web Services Client Protocol 2.0.50727.4952)	PowerCLI

Note: The PowerCLI entry, I am not 100% sure, that is what I received when using Connect-VIServer to our vCenter Server

Another method of tracking down a rogue vSphere login is using simple netstat and identifying any entries that show the Local Address of your vCenter Server IP Address mapping to port 443 which is used for communication. You will then identify the Foreign Address to validate all clients.

As you can see my desktop address of 172.30.0.218 is included in that list along with few others. If you know which systems should have access, then you can easily narrow down the rogue client's address.

Remember once you are done hunting your rogue user to revert the vCenter logging option back to it's original configuration else you may rotate through your vpxd logs pretty quickly.