virtuallyGhetto

vSphere 6.0

vSphere Content Library versioning 

by 1 Comment

A question came up the other day on how versioning is handled when using the vSphere Content Library feature and specifically when you update an existing VM template that already exists in the Content Library (CL, which I will be using throughout the article). Having used CL since vSphere 6.0 which I have also written about it extensively here, I knew that it had a built-in versioning mechanism which would automatically increment as new updates were applied to the VM Templates stored in CL.

One thing that I do not think many folks are aware of is that you can actually retrieve the CL Item Version within the vSphere Web Client. You can do by just adding the "Content Version" column by right clicking on column headers and select show/hide columns to add or remove specific fields.


I figured, this was pretty straight forward and shared my findings with the individual and thought that was it. While working on my Content Library PowerCLI community module, I came to learn there was more than one version that was being returned by the CL API.

Here is an example screenshot showing the three different versions using the Get-ContentLibraryItems (not to be confused with the OOTB Get-ContentLibraryItem cmdlet):


Taking a look at the API documentation, it was still not 100% clear on what all these versions provided and also when would they increment? I reached out to one of the CL Engineers, Eric who then provided me more information on how each of these versions are being used. Lets go through each one of them and I will include an example which also helped me understand how these different "versions" actually work.

Content Version - This particular version will increment as changes are made to the actual file content itself. This is also the same version that is available in the vSphere Web Client UI as shown earlier.

Lets now go through an example workflow to show when this version will get incremented:

  1. Create a new VM called Foo (ContentVersion=N/A)
  2. Clone VM Foo into CL called Foo VM Template (ContentVersion=1)
  3. Clone completes for Foo VM Template (ContentVersion=2) <--The reason this is 2 and not 1 is that when the initial CL Item is created (think of it as a placeholder), the Content Version is incremented
  4. Deploy Foo VM Template to new VM called Bar (ContentVersion=2, no changes on deploy)
  5. Update VM Bar with some changes and clone Bar back to Foo VM Template (ContentVersion=3) <--Changes were made and we now increment the Content Version

Metadata Version - This particular version will only increment for changes made to either the name or description of the CL template.

Version - This particular version is used for concurrency control and this will always increment along with the Metadata Version for local libraries.

Lets now go through an example workflow for a local library to show how these two versions get incremented:

  1. Create a new VM called Foo (MetadataVersion=N/A)
  2. Clone VM Foo into CL called Foo VM Template (MetadataVersion=1, Version=1)
  3. Update the description of Foo VM Template (MetadataVersion=2, Version=2)
  4. Deploy Foo VM Template to new VM called Bar (MetadataVersion=2, Version=2, no changes on deploy)
  5. Update VM Bar with some changes and clone Bar back to Foo VM Template (MetadataVersion=3, Version=3)

Lets now go through an example workflow for a subscribed library to show how these two versions get incremented:

  • Create a new VM called Foo on a local library named CL1 (MetadataVersion=N/A)
  • Clone VM Foo into CL called Foo VM Template (MetadataVersion=1, Version=1)
  • Update the description of Foo VM Template (MetadataVersion=2, Version=2)
  • Create a new subscribed library to CL1 which will download Foo VM Template (MetadataVersion=2, Version=1) <--MetadataVersion must match the publisher, but version does not have to match

After going through a few of these exercises with a few dummy VMs and using my Get-ContentLibaryItems function, I was able to get a better grasp on CL versioning. Hopefully this helpful for anyone who might want to use the CL APIs to track changes between their VM templates and/or files or just being able to see this within the vSphere Web Client UI.

Quick Tip – List all open ports on the VCSA / PSC

by Leave a Comment

The list of required ports for both a vCenter Server Appliance (VCSA) and Platform Services Controller (PSC) are pretty well documented here (6.5), here (6.0) and here (5.5) for customers who require this information to setup external connectivity within their networking infrastructure. Having said that, it is may not always be clear on what ports are actually opened as they will usually depend on the type of deployment and the services that are running. Instead, some customers have inquired about getting a list of all open ports directly from the VCSA/PSC to ensure they have the actual configuration which can be used to build firewall rules and/or for auditing purposes.

Today, the only method is to login directly into the VCSA/PSC via SSH (you could also use GuestOps API, so that SSH is NOT required) and fetching this information using iptables. Hopefully, in the future, this can be made available as part of the VAMI API since it already covers some basic inbound firewall rule capabilities. In the mean time, below are examples of how to get all the open ports for both VCSA/PSC

Run the following command to view all open ports on VCSA/PSC:

iptables -L port_filter -n --line-numbers


You will notice in the output above, there is also a chain number on the far left side which is associated with each rule. This chain number can be used to inspect the rule further and some rules include a nice alias to help you identify what the port might be used for.

For example, we can run the following to inspect chain rule #30 and find out this port is being used for syslog. If we want the port number, we simply add the -n option.

iptables -L port_filter 30
iptables -L port_filter 30 -n


Not all of the firewall rules have an alias name and even if they do, it still may not be apparent on what service is opening that particular port. We can actually look at the firewall rule definitions which are located under /etc/vmware/appliance/firewall and you will see a JSON file for each of the VCSA/PSC services that require firewall rules to be opened up. For a given port, you can just grep in this directory to identify the service that is requiring the port.

For example, if we take a look at the vmware-syslog, we see that it requires tcp/udp 514 and tcp 1514 under the "rules" array which defines the list of external ports open. You can ignore the internal ports as those are not exposed to the outside world but used by internal services. In case the services are still not clear, you can always reference the port number back to the documentation which I had linked above to get more details about the particular port.

Auditing/Logging vCenter Server authentication & authorization activities

by 1 Comment

Recently, I have seen an increase in the number of requests from our field and customers inquiring about logging various vCenter Server authentication and authorization activities. The topics vary from identifying which log files contain which activities to to why some of this information is not available in the vCenter Server Events UI or why they are available else where. In most of these cases, customers were also looking for a way to forward these activities to their remote syslog infrastructure for auditing and tracking purposes whether that is using vRealize Log Insight (which all vSphere customers get 25 free OSI licenses!) or some other logging solution.

Having explored this topic lightly in the past and given the amount of interests, I thought I would dive a bit deeper and look at some of the common authentication and authorization workflows and provide examples of what the log entries look like and where you can find them. However, before jumping right in, I think is is worth spending a few minutes looking at the history of authentication (commonly referred to as AuthN) and authorization (commonly referred to as AuthZ) for vCenter Server and where we had started from and where we are at today to give you the full context.

UPDATE (04/08/19) - Please take a look at this blog post here for all new auditing enhancements in vSphere 6.7 Update 2 which simplifies the consumption of vCenter and vCenter SSO auditing events.

History of vCenter Server AuthN/AuthZ

Prior to vSphere 5.1, vCenter Server handled both Authentication (AuthN) and Authorization (AuthZ). As a Client, you would connect directly to vCenter Server and the AuthN service will verify who you are whether that is a local account on the OS or an Active Directory user which required vCenter Server to be joined to your AD Domain. Once you have been authenticated, the AuthZ service will then take over and verify the privileges you have been assigned to perform specific operations within vCenter Server.


In vSphere 5.1, a new service was introduced called Single Sign-On (SSO) which now takes over for AuthN services from vCenter Server. Once authenticated, it will then allow you to connect to the vCenter Server which then handles AuthZ activities


Although it may not be apparent, one major implication is where are successful and failed authentications being logged? In the past, these would reside within vCenter Server since it handled both AuthN/Authz activities, vCenter Server even included specific authentication Events that can then be seen using the UI and/or API. However, with SSO in the picture, authentication is no longer in vCenter Server but with SSO. This is why when you have a failed login using the vSphere Web Client (Flex/H5) UI it does not show up in vCenter Server and it because the logging is done but within the SSO service (which now resides in the Platform Services Controller for more recent vCenter releases).

[Read more...] about Auditing/Logging vCenter Server authentication & authorization activities

Updated Nested ESXi 6.0u3 & 6.5d Virtual Appliances

by 26 Comments

I finally found a bit of "extra" spare time to update my Nested ESXi Virtual Appliances to support some of the recent releases of ESXi, 6.0 Update 3 and 6.5d, which enables customers to easily and quickly deploy vSAN 6.6 in their environment for testing, development or learning purposes. If you have not used this appliance before, please have a look at this article which goes into greater detail on how to deploy and use the Nested ESXi VA.

As part of this update, I also spent some time looking at all the feedback that I had received from the community since releasing the VA and I took this opportunity to also add some nice enhancements that folks have been asking about 🙂 Jump towards the bottom to see what's new. To reduce the number of VA's that I need to manage and due to usage, the following VA's have recently been decommissioned. I only plan on supporting the latest versions which you can find in the links below.

Decommissioned VA's:

  • ESXi 5.5 Update 3 (Nested_ESXi5.x_Appliance_Template_v2.ova)
  • ESXi 6.0 Update 2 (Nested_ESXi6.x_Appliance_Template_v5.ova)
  • ESXi 6.5 GA (Nested_ESXi6.5_Appliance_Template_v1.ova)

New VA's:

What's New:

  • Support for DHCP 
    • I know this might sound pretty basic but before you were required to specify a static IP (even if you had DHCP). By default, you no longer need to fill out the networking section as highlighted in yellow below.
  • Support for default root password
    • You no longer need to provide root password, it will default to the famous VMware1! The issue in the past was that I had randomly generated a password which I discarded and when the customization failed, it was very difficult to troubleshoot since I do not actually have the password 😉 Hopefully we do not have any other bugs, but this will make debugging easier and also reduce the amount of input if you want to quickly spin up an ESXi instance.
  • Support for VLAN ID
    • Though not a huge number of requests, there were still of you who asked for 802.1q (trunk) support on Management VMkernel interface. This is an optional field and obviously this is only applicable if you provide a static IP Address.
  • Automatic removal of Customization VIB
    • As some of you may or may not know, the way in which these OVF properties are processed within the Nested ESXi instance is a special firstboot script which reads in these values and then applies the ESXi customization. If everything is successful, there really is no use for this to exists further and although you could set a certain advanced setting to force re-customization, it was quicker to just re-deploy. With that in mind, the customization VIB is now automatically removed once its done its job. I have included a special debug option that would allow it to not be deleted in scenarios where there are issues and we need to take a look at the state of the system. With this change, you really now have a "vanilla" ESXi instance 🙂
  • Fixed dvFilter param for eth1


Hope you enjoy some of these new updates and happy Nesting!

Auditing & Automating Disabled Protocols (TLS/SSLv3) for ESXi 6.0u3 & 6.5 using PowerCLI

by 31 Comments

A couple of weeks back, I had received a question from one of our TAMs in regards to automating the disablement of specific TLS/SSL protocols for their ESXi 6.0 Update 3 hosts. As of vSphere 6.0 Update 3 and vSphere 6.5, customers now have the ability to completely disable TLS 1.0, TLS 1.1 and SSLv3 using the new TLS Reconfiguration Tool. Mike Foley did a nice write up here if you are interested in more details.

The TLS Reconfiguration Tool works well if you have the same version of vSphere for both your vCenter Server and ESXi host, but has challenges when you are in a mixed environment like this particular customer. In their environment, they are running vCenter Server 6.5 and ESXi 6.5 Update 3 which prevented them from using the TLS Reconfiguration Tool as this is a limitation with the tool today.

UPDATE (05/11/17) - Added support for ESXi 6.5 hosts as well

Given the TLS Reconfiguration Tool was written in Python, I was able to take a closer look at its implementation and I found that the settings that controlled the disabled protocols were just merely a few ESXi Advanced Settings which meant that this could be automated using standard vSphere Automation Tools that our customers were already familiar with. As part of this exercise, I also discovered the tool currently does NOT support disabling TLS/SSLv3 protocols for the Small Footprint CIM Broker (SFCB) service which is also required if you want to be in full compliance for a particular TLS protocol. Although there is not a direct SFCB API that allows you to manage the sfcb.cfg configuration file, there is still a way we can automate this without requiring SSH to the ESXi host which would technically be the alternative. Lastly, I was a bit surprised to see the TLS Reconfiguration Tool did not have a "query" option for listing the current disabled protocols for all ESXi hosts, but they do have it for vCenter Server itself.

To help this particular customer and others who may have specific TLS compliance requirements, I have created the following PowerCLI script called ESXiDisableProtocolConfiguration.ps1 which includes the following two functions:

  • Get-ESXiDPC - Retrieve the current disabled protocols for all ESXi hosts within a vSphere Cluster
  • Set-ESXiDPC - Configure the specific disabled protocols for all ESXi hosts within a vSphere Cluster

[Read more...] about Auditing & Automating Disabled Protocols (TLS/SSLv3) for ESXi 6.0u3 & 6.5 using PowerCLI

Maximum number of vCenter Servers per Single Sign-On (SSO) Domain

by 9 Comments

This particular question and its variations have been raised quite a bit lately by our field and customers. For me, this was an opportunity to see if we can provide some additional clarification and help explain some of the nuances that may have been causing some of the confusion around the supported maximums for both vCenter Server and the Platform Services Controller (PSC).

In the vSphere 6.5 Configuration Maximum, there are three specific maximums that helps us answer our question on the maximum number of vCenter Servers per vCenter Single Sign-On (SSO) Domain. I will go through each of the maximums and provide some additional context that will help us derive the answer to our question.

The first is the "Linked vCenter Servers" which defines the maximum number of vCenter Servers that can be supported in an Enhanced Linked Mode (ELM) configuration. What is interesting about this particular maximum is that it actually answers the majority of our question. By definition, an ELM consists of a single SSO Domain. This then means that you can only have a maximum of 10 vCenter Servers per SSO Domain.

vCenter Server Maximum

Configuration Maximum
Linked vCenter Servers (w/External PSC) 10
Linked vCenter Servers (w/Embedded PSC) 15

Note: As of vSphere 6.7, you can have up to 15 Embedded VCSA's within an ELM.

The second is the "Maximum PSCs per vSphere Domain" which defines the maximum number of PSC's that can be part of a single SSO Domain, pretty straight forward. The third is the "Maximum PSCs per site behind a load balancer" which just adds an additional constraint when using a load balancer with your PSCs.

Platform Services Controller Maximum

Configuration Maximum
Maximum PSCs per vSphere Domain 10
Maximum PSCs per site behind a load balancer 4

[Read more...] about Maximum number of vCenter Servers per Single Sign-On (SSO) Domain