A couple of years back I had published a detailed analysis on vCenter Server's Authentication (AuthN) and Authorization (AuthZ) from an auditing and logging standpoint. This has been the go to reference for many of our customers and the posts also includes a number of log samples which I have documented in the following Github repository.
In addition to serving as a reference for our customers, it has also helped our Product and Engineering teams understand where we still had some gaps and how we could improve the overall user experience. As hinted in the recently announced vSphere 6.7 Update 2 release, which will be available soon, there are number of new auditing enhancements that have been made to both vCenter Server and the vCenter Single Sign-On (SSO) service that I think customers will really appreciate.
"Real" client IP address in Events
When you look at a login or logout Event in vCenter Server today, you may have noticed the user's client IP Address is actually of the vCenter Server rather than the actual remote client's address and the reason for this is explained here. In vSphere 6.7 Update 2, the real client IP Address is now captured and is included in all successful login/logout and failed logins. This information can now enable administrators to easily identify unauthorized access and be able to quickly track down the systems initiating the connections.
Below is a screenshot of a login/logout event for vSphere 6.7 which shows the IP Address of the vCenter Server and vSphere 6.7 Update 2 system which now shows the actual client IP Address logging into that vCenter Server. This information is accessible using both the vSphere UI and API. For customers who enable syslog on their vCenter Server Appliance (VCSA), these events will automatically be included and forwarded to their remote syslog target.
vCenter Single Sign-On Events
Historically, vCenter Server Events did not include any vCenter SSO details. If customers wanted some of this information, they would have to take additional steps to forward several SSO log files which still did not include everything. These logs were meant for troubleshooting/debugging and was not designed for auditing purposes. In vSphere 6.7 Update 2, all configuration changes or user activity that involves vCenter SSO will now have a vCenter Server Event that represents the specific operation.
The way this works is there is now a dedicated vCenter SSO Audit log at /var/log/audit/sso-events/audit_events.log for VCSA and C:\ProgramData\VMware\vCenterServer\runtime\VMwareSTSService\logs\audit_events.log for Windows VC which will capture all vCenter SSO related activities. Every 5 minutes, these SSO Events will then be published to vCenter Server which will then make them available in both the vSphere UI and API.
Here is a screenshot of the vSphere UI with various vCenter SSO Events that can now easily be viewed without having SSH'ing to the VCSA and manually looking at a log file.
Similarly, customers can use Automation Tools like PowerCLI or any other vSphere SDK to look up specific vCenter SSO Events such as the example below using PowerCLI.
Get-VIEvent | where {$_.EventTypeId -match "com.vmware.sso.*"} | select EventTypeId,FullFormattedMessage
For customers who enable syslog on their vCenter Server Appliance (VCSA), all vCenter SSO events will automatically be included and forwarded to their remote syslog target, as shown in the screenshot below using vRealize Log Insight. This means customers no longer have to manually configure their VCSA to forward specific SSO logs, this is now automatic for you by simply enabling syslog for your VCSA! Another reason to move to the VCSA if you are still running Windows VC.
For your convenience, below are all the different vCenter SSO Event types that can be generated. You can also get this information by simply viewing the audit_events.log which contains the logged event details and type.
SSO Event | Description |
---|---|
com.vmware.sso.LoginSuccess | SSO Login Success |
com.vmware.sso.LoginFailure | SSO Login Failure |
com.vmware.sso.Logout | SSO Logout Success |
com.vmware.sso.CertificateManager | Add/Remove Certificate |
com.vmware.sso.ConfigurationManagement | Token Policy/Smart Card Configuration |
com.vmware.sso.DomainManagement | Login Message/Banner Configuration |
com.vmware.sso.IdentitySourceManagement | Identity Source Configuration |
com.vmware.sso.LockoutPolicy | Lockout Policy Configuration |
com.vmware.sso.PasswordPolicy | Password Policy Configuration |
com.vmware.sso.PrincipalManagement | SSO User and Group Add/Remove/Update |
com.vmware.sso.RoleManagement | SSO Role Configuration |
com.vmware.sso.SmtpConfiguration | SSO Email Configuration |
com.vmware.sso.SystemManagement | Active Directory Configuration |
com.vmware.sso.Provisioning | Create/Remove IDP |
Global Permission Events
As you may have guessed from the header, we also now have Global Permission Events in vSphere 6.7 Update 2! When a user creates, updates or removes a Global Permission, a new vCenter Event will be published as shown in the screenshot below. For customers who enable syslog on their vCenter Server Appliance (VCSA), all Global Permissions events will automatically be included and forwarded to their remote syslog target, no additional configuration changes required.
Here are the two Global Permission Event types:
Global Permission Event | Description |
---|---|
com.vmware.cis.CreateGlobalPermission | Add/Update Global Permission |
com.vmware.cis.RemoveGlobalPermission | Remove Global Permission |
You can also use PowerCLI or any other vSphere SDK to look up these specific events:
Get-VIEvent | where {$_.EventTypeId -match "com.vmware.cis.*"} | select EventTypeId,FullFormattedMessage
vCenter and Tagging Permission Change Events
Lastly, for any type of permission modification on a vCenter Server Inventory Object (Virtual Machine, ESXi host, Network, etc.) including vSphere Tags and Categories, a vCenter Server Event will now be published. These permission events will be available using both the vSphere UI and API. For customers who enable syslog on their vCenter Server Appliance (VCSA), all permission events will automatically be included and forwarded to their remote syslog target, no additional configuration changes required.
Here are both the new and existing vCenter Permission Event types:
vCenter & Tagging Permission Event | Description |
---|---|
com.vmware.cis.CreatePermission | Add Tag Permission |
PermissionAddedEvent | Add vCenter Permission |
PermissionUpdatedEvent | Update vCenter Permission |
PermissionRemovedEvent | Remove vCenter Permission |
Requirements
In addition to upgrading to vSphere 6.7 Update 2 to get the new auditing enhancements, the new vCenter SSO Auditing Events will only be available for an Embedded vCenter Server (Windows or Appliance). For customers running vCenter with an External Platform Services (PSC), you can converge your deployment to an Embedded vCenter by using the new Convergence CLI Tool included in vSphere 6.7 Update 1 or directly in the vSphere UI with the upcoming vSphere 6.7 Update 2 release.
Really appreciate the info you put out. However, I’m not too comfortable with the command line and afraid to mess something up. I’ve done it before. Is there a easy way to clear the event logs in vCenter 6.7 web UI?
I know this is an old post, i need to setup an alert anytime anyone logs into the vCenter. I don’t see a way to create a vCenter Alarm to send out an email on com.vmware.sso.LoginSuccess. Any idea on what’s the easiest way to do this?
Easiest for this and many other Event-based Automation would be via the VMware Event Broker Appliance (VEBA) Fling. There’s lots of articles on my blog + videos on microsite https://vmweventbroker.io/