vcenter server appliance

Enabling shell access for Active Directory users via SSH to vCenter Server Appliance (VCSA)

10/09/2017 by William Lam 4 Comments

I had a question the other day on whether it was possible to enable shell access for Active Directory users when logging into the vCenter Server Appliance (VCSA) via SSH? The answer is yes and though this is documented here, it is not very clear whether this is only applicable to SSO-based users only. In any case, the process to enable this is pretty straight forward and simply requires two steps which I have outlined below.

Step 0 - Ensure that your VCSA and/or PSC is joined to Active Directory before proceeding to the next step. If not, take a look at the documentation here for more details.

Step 1 - Login to vSphere Web Client and under Administration->System Configuration->Nodes->Manage->Settings->Access, go ahead and enable boh SSH and bash shell options. The first setting turns on SSH to the VCSA and the second setting allows users (local, SSO and AD) to access the shell on the VCSA.

Step 2 - In the vSphere Web Client and under Administration->Single Sign-On->Users and Groups->Groups, select the SystemConfiguration.BaseShellAdministrators group and add either an AD User and/or Group that you wish to allow to access the shell.

Once you have completed the steps above, you can now SSH to your VCSA/PSC using the AD user (UPN format) that you had authorized earlier. In the example below, I am logging into one of my VCSA using user primp@primp-industries.com and as you can see, I am placed into the appliance shell by default.

At this point I can access all the appliancesh commands just like I normally would if I had logged as a root or administrator@vsphere.local.

If we wish to change to bash shell, we simply just type "shell" which will enable shell access, assuming you had performed Step 2.

One thing that I noticed is that the default home directory for the AD user is /var/lib/nobody and apparently that does not exists by default, so users end up in / directory by default after enabling shell access. I am not sure if this is also related, but the username shows up as nobody as you can see from the prompt. This is something I will share with Engineering to see if we can improve upon as I am sure most of you would rather see the user that is actually logged in.

The good news from an auditing and logging standpoint is that for operations that are logged, it does properly show the username even though the prompt is showing up as nobody.

PowerCLI script to help correlate vCenter, ESXi & vSAN build/versions w/o manual VMware KB lookup

08/02/2017 by William Lam 9 Comments

I can still remember when I was a VI Admin and how annoying it was to try to correlate the build numbers for my ESX(i) hosts and vCenter Servers that I have deployed with the versions listed on VMware's website. This especially gets challenging when there are multiple patch releases (a, b, c or 01, 02, 03) in between major releases (5.5, 6.0, 6.0u1, 6.0u2, 6.5, etc.). Historically, most customers including myself would retrieve the respective build numbers and then manually comparing them to either the release notes and/or download website which was very tedious.

Although VMware has exposed the version number within our vSphere products since day 1 which can also be retrieved programmatically using the vSphere API (here), it unfortunately does not provide more details than simply the major/minor version (e.g. 5,5, 6.0, 6.5, etc) of the software. Recently, VMware had released a series of VMware KBs which provides a mapping between the build numbers for vCenter Server, ESXi and vSAN to their respective versions which can be found in the links below:

These are definitely a great set of resources that I know many customers including myself have been using since its release. Having said that, the process today is still pretty manual since you need to manually retrieve the build numbers for either a VC, ESXi or vSAN Host (can be automated using vSphere APIs) and then comparing that to the KBs to get the correct versions. How cool would it be if you could *easily* just point to YOUR environment and retrieve the version information for either a vCenter Server (Windows or VCSA), ESXi host(s) or vSAN host(s) without needing to manually perform this lookup each time? Well, I have just done that! I have taken all three KBs and converted that information into a simple PowerCLI script called VCESXivSANBuildVersion.ps1 leveraging our vSphere API and it provides three functions:

Get-VCVersion - Retrieves the vCenter Server version given a VC connection
Get-ESXiVersion - Retrieves the ESXi version for all hosts given a vSphere Cluster
Get-VSANVersion - Retrieves the vSAN version for all hosts given a vSAN Cluster

Here is an example output using the first two functions:

For the vCenter Server version output, you will notice that I am also including the OS platform of your vCenter Server, so you can distinguish between a Windows vCenter Server and a vCenter Server Appliance (VCSA) which can be useful to see if you have been #migrate2vcsa ;). For the ESXi version output, you will notice the "OriginalInstallDate" value, this is actually new API property that was introduced in vSphere 6.5 and it provides you with the original installation date of your ESXi host (more details can be found here) which is pretty neat.

Here is an example output using last function:

If you wanted to take this a step further, you could even take this output and dynamically update the vSphere UI using either Custom Attributes or vSphere Tags so you know what version the software is at any given moment. Its easy enough to set this up as a scheduled task that could run periodically so you always have the latest information provided in the vSphere UIs.

Although this is a significant improvement over the existing manual methods, I think most of you will agree that it would be ideal if this information was natively available within the product which means BOTH UI and APIs. I think we all appreciate versioning of software is not always easy and it can change from release to release for a variety of reasons, most of which may not be technical. If the vSphere platform could dynamically pull this information in either real time and/or through an offline mechanism and provide this association by default, it would greatly improve the experience when needing to troubleshoot or perform maintenance of the vSphere platform. If this is something you would like to see, please leave a comment below providing your feedback. I know I have already pinged our PMs about this and I am sure they would love to hear form you as well!

Additional Information:

Note1: Update levels can be found using the vSphere API, take a look at this article here for more details.

Note2: As of ESXi 6.5 Update 1, the Update levels are also included by default in the Embedded Host Client as shown in the screenshot below:

Note3: As of vSAN 6.2, the vSAN Management API already includes vSAN version information that can be queried. Take a look at this script here which exercises this new API. For example above, I decided to not use this new API since customers may be running older releases of vSAN which is not covered by the vSAN Mgmt API.

Note4: VMware has also published simliar build to version mapping for other VMware products which can find the complete list here.

Quick Tip – List all open ports on the VCSA / PSC

07/20/2017 by William Lam Leave a Comment

The list of required ports for both a vCenter Server Appliance (VCSA) and Platform Services Controller (PSC) are pretty well documented here (6.5), here (6.0) and here (5.5) for customers who require this information to setup external connectivity within their networking infrastructure. Having said that, it is may not always be clear on what ports are actually opened as they will usually depend on the type of deployment and the services that are running. Instead, some customers have inquired about getting a list of all open ports directly from the VCSA/PSC to ensure they have the actual configuration which can be used to build firewall rules and/or for auditing purposes.

Today, the only method is to login directly into the VCSA/PSC via SSH (you could also use GuestOps API, so that SSH is NOT required) and fetching this information using iptables. Hopefully, in the future, this can be made available as part of the VAMI API since it already covers some basic inbound firewall rule capabilities. In the mean time, below are examples of how to get all the open ports for both VCSA/PSC

Run the following command to view all open ports on VCSA/PSC:

iptables -L port_filter -n --line-numbers

You will notice in the output above, there is also a chain number on the far left side which is associated with each rule. This chain number can be used to inspect the rule further and some rules include a nice alias to help you identify what the port might be used for.

For example, we can run the following to inspect chain rule #30 and find out this port is being used for syslog. If we want the port number, we simply add the -n option.

iptables -L port_filter 30
iptables -L port_filter 30 -n

Not all of the firewall rules have an alias name and even if they do, it still may not be apparent on what service is opening that particular port. We can actually look at the firewall rule definitions which are located under /etc/vmware/appliance/firewall and you will see a JSON file for each of the VCSA/PSC services that require firewall rules to be opened up. For a given port, you can just grep in this directory to identify the service that is requiring the port.

For example, if we take a look at the vmware-syslog, we see that it requires tcp/udp 514 and tcp 1514 under the "rules" array which defines the list of external ports open. You can ignore the internal ports as those are not exposed to the outside world but used by internal services. In case the services are still not clear, you can always reference the port number back to the documentation which I had linked above to get more details about the particular port.

Native OVF support for Fusion/Workstation 2017 Tech Preview

07/18/2017 by William Lam 1 Comment

The VMware Fusion and Workstation team just released their 2017 Tech Preview releases and there is a ton of new and awesome capabilities which you can read more about here and here. One of the exciting new features, which I was very fortunate to have been involved with is finally here, native OVF property support! Although customers have had the ability to import OVF/OVAs for some time now, if they included OVF properties, they would be ignored and often times this would result in a failed deployment as those properties are required for the initial setup.

A great example of this is trying to run the vCenter Server Appliance (VCSA) on either Fusion or Workstation. Today, the only workaround is to manually edit the VMX file and supplying the correct OVF properties which I have blogged about here. With the latest TP release of Fusion/Workstation, when you import an OVF/OVA that contains OVF properties, the UI will automatically render the required information directly into the UI without needing users to manually touch the VMX files.

Here is a screenshot of deploying the latest VCSA 6.5d OVA (jump to bottom for some additional VCSA tidbits when deploying to Fusion/Workstation):

Visualizing live network traffic on the vCenter Server Appliance using net-glimpse

07/17/2017 by William Lam Leave a Comment

Last week I came across a really interesting OSS project called net-glimpse which allows you to easily visualize your network traffic in real-time and making that available using any standard web browser. I thought it would be neat to see what this might look like running on the vCenter Server Appliance (VCSA). I got it up in running in just a couple of minutes and even shared the results on Twitter as you can see from the tweet below:

Thought it be interesting to see the network traffic visualization on VCSA by running net-glimpse https://t.co/xyznnHnmkx #NotSupported pic.twitter.com/IjeoCV2QTx

— William Lam (@lamw) July 14, 2017

I had couple of folks ask about the setup and I figure I would post a quick write up. While looking at the project, I found that net-glimpse includes quite a bit of customizations in the colors, data collection and how data is displayed. Specifically, rather than relying on the well-known ports that have already been pre-defined, you can also add additional custom ports and specify the label that it should automatically used. This gave me an idea, instead of a generic visualization of the VCSA, we could get specific service information and have those label automatically get displayed.

Auditing/Logging vCenter Server authentication & authorization activities

06/19/2017 by William Lam Leave a Comment

Recently, I have seen an increase in the number of requests from our field and customers inquiring about logging various vCenter Server authentication and authorization activities. The topics vary from identifying which log files contain which activities to to why some of this information is not available in the vCenter Server Events UI or why they are available else where. In most of these cases, customers were also looking for a way to forward these activities to their remote syslog infrastructure for auditing and tracking purposes whether that is using vRealize Log Insight (which all vSphere customers get 25 free OSI licenses!) or some other logging solution.

Having explored this topic lightly in the past and given the amount of interests, I thought I would dive a bit deeper and look at some of the common authentication and authorization workflows and provide examples of what the log entries look like and where you can find them. However, before jumping right in, I think is is worth spending a few minutes looking at the history of authentication (commonly referred to as AuthN) and authorization (commonly referred to as AuthZ) for vCenter Server and where we had started from and where we are at today to give you the full context.

History of vCenter Server AuthN/AuthZ

Prior to vSphere 5.1, vCenter Server handled both Authentication (AuthN) and Authorization (AuthZ). As a Client, you would connect directly to vCenter Server and the AuthN service will verify who you are whether that is a local account on the OS or an Active Directory user which required vCenter Server to be joined to your AD Domain. Once you have been authenticated, the AuthZ service will then take over and verify the privileges you have been assigned to perform specific operations within vCenter Server.

In vSphere 5.1, a new service was introduced called Single Sign-On (SSO) which now takes over for AuthN services from vCenter Server. Once authenticated, it will then allow you to connect to the vCenter Server which then handles AuthZ activities

Although it may not be apparent, one major implication is where are successful and failed authentications being logged? In the past, these would reside within vCenter Server since it handled both AuthN/Authz activities, vCenter Server even included specific authentication Events that can then be seen using the UI and/or API. However, with SSO in the picture, authentication is no longer in vCenter Server but with SSO. This is why when you have a failed login using the vSphere Web Client (Flex/H5) UI it does not show up in vCenter Server and it because the logging is done but within the SSO service (which now resides in the Platform Services Controller for more recent vCenter releases).