A preview of native syslog support in VCSA 6.0

Proper logging of VMware hosts, services and application logs are becoming more and more critical these days and their usage goes beyond just troubleshooting. In many of our customer environments, extended log retention is often mandatory to satisfy auditing and compliance requirements. Support for remote syslog has been around in ESXi for quite some time and has included several enhancements over the years, however logging for vCenter Server itself has not changed much over the years. Historically, vCenter Server started out as a Windows application and outside of standard filesystem logging there is also Microsoft Event Logs which was not really all that useful. With the release of the vCenter Server Appliance (VCSA), syslog support became more attainable, at least without additional 3rd party tools.

I can even remember when I was an administrator, I had to get creative on how to forward vCenter Server logs to a remote syslog server which I had blogged about back in 2012. Though the solution works, it was not ideal especially when you are running several dozen to several hundred vCenter Server instances like many of our customers do today. When I had discovered that there was a Common Logging initiative within VMware for vSphere 6.0, I was pretty excited and I can only guess that this also put a big smile on many of our GSS folks faces ;)

As you can imagine this was no small undertaking, especially with the organic growth of services and applications within vCenter Server. The goal was not only to support native remote syslog but to also standardize on the location, rotation, retention of all the logs and most importantly providing a consistent time stamp of events so that an administrator or 3rd party tool can easily correlate operations across multiple VMware log files. Though complete native syslog support in vCenter Server is not 100% ready just yet, much of the plumbing and foundation has already been finished and in fact you can see some of this in the latest release VCSA 6.0.

With VCSA 6.0, there is partial support for native remote syslog which is configurable through the VMware Syslog Service under the new vCenter Server System Configuration found within the vSphere Web Client.

vcenter_server_6_syslog_1
There are four settings that you will need to configure:

  • Common Log Level - * (everything), info, notice, warn, error, crit, alert & emerg
  • Host - Hostname/IP Address of a *single* remote syslog server
  • Port - Port of the remote syslog server (514 for UDP & 1514 for TCP is already opened on the VCSA firewall)
  • Protocol - Supports tcp, udp & tls

A restart is not required when configuring the syslog service and logs will automatically be forwarded to the remote syslog server which is quite nice. You can also view the health status of the syslog service and its connectivity to the remote syslog server by clicking onto the "Summary" view as seen in the screenshot below. For more information about the new syslog service, check out the official documentation here.

vcenter_server_6_syslog_2
So what exactly does partial syslog support really mean? What logs are being forwarded to a syslog server when the syslog service is enabled?

There are currently two major sets of logs that are forwarded to a remote syslog server when the new syslog service is configured:

  1. All logs from ESXi hosts that are connected to the vCenter Server will be forwarded
  2. A partial set of vCenter Server services (details in table below) will be forwarded
Service Name Service Description Service Log Location
applmgmt-audit Appliance Management /var/log/vmware/applmgmt/applmgmt-audit/applmgmt-audit-syslog.log
audispd Audit Event Dispatcher /var/log/audit/audispd/audispd-syslog.log
auditd Audit System /var/log/audit/auditd/auditd-syslog.log
rbd Auto Deploy /var/log/vmware/rbd/rbd-syslog.log
vmafdd VMware Authentication Framework /var/log/vmware/vmafdd/vmafdd-syslog.log
vmcad VMware Certificate Service /var/log/vmware/vmcad/vmcad-syslog.log
vmdird VMware Directory Service /var/log/vmware/vmdird/vmdird-syslog.log
watchdog-rhttpproxy Watchdog for Reverse HTTP Proxy service /var/log/vmware/rhttpproxy/watchdog-rhttpproxy/watchdog-rhttpproxy-syslog.log
watchdog-syslog Watchdog for Syslog service /var/log/vmware/syslog/watchdog-syslog/watchdog-syslog-syslog.log
watchdog-vmware-vpostgres Watchdog for vPostgres DB service /var/log/vmware/vpostgres/watchdog-vmware-vpostgres/watchdog-vmware-vpostgres-syslog.log
watchdog-vpxd Watchdog for vCenter Server service /var/log/vmware/vpxd/watchdog-vpxd/watchdog-vpxd-syslog.log
watchdog-vws Watchdog for vCenter Web Services service /var/log/vmware/vws/watchdog-vws/watchdog-vws-syslog.log

Note: The information above was extracted from /etc/vmware-syslog/custom-file-location.conf

Here is a screenshot of my vRealize Log Insight instance ingesting the logs that have been forwarded over from my VCSA 6.0:

vcenter_server_6_syslog_7
Although not all the vCenter Server services have been integrated into this new native syslog mechanism, you can see where things headed and hopefully in the not too distant future we will have full native syslog support for all application and system logs found withint vCenter Server. One thing that I really do like is that I can go to one single location to configure my remote syslog server and automatically receive all logs from the ESXi hosts being managed by that vCenter Server and forwarded to the configured syslog server. This definitely makes it operationally friendly so that you have one less thing to configure when provisioning new ESXi hosts.

One limitation that I found when configuring your remove syslog server is that there is no way to reset the values to NULL and the UI also limits the number of remote syslog server to just one, even though you can specify multiple targets. One way to get around this UI limitation is by editing the underlying configuration file which is located in /etc/vmware-syslog/syslog.conf

Here is an example of what the syslog.conf looks like for the above configuration:

*.info @log.primp-industries.com:514;RSYSLOG_SyslogProtocol23Format

If you wish to add a second or even third syslog server, you simply just need to duplicate the existing line and update the hostname or IP Address of your syslog server.

*.info @log.primp-industries.com:514;RSYSLOG_SyslogProtocol23Format
*.info @log2.primp-industries.com:514;RSYSLOG_SyslogProtocol23Format

If you are manually editing the syslog.conf, you will need to restart the syslog service by running the following command for the changes to take effect:

/etc/init.d/vmware-syslog restart

Some of you might say this is great and all, but one of the most important log files which is the vCenter Server log (vpxd.log) is not being being forwarded. How useful is this really to me? I know I definitely asked that question :) Though not ideal, there is a small configuration change you can apply to easily get vpxd.log to also forward to a remote syslog server using the new syslog service.

You will need to change the vCenter Server advanced setting "config.log.outputToSyslog" property (can also be done using vSphere API) from false to true as seen in the screenshot below.

vcenter_server_6_syslog_3
The above assumes you have already configured the syslog service and for this change to go into effect, you will need to restart the vCenter Server service. This can be done using the System Configuration and under the vCenter Server Service, by just right clicking and selecting "Restart".

vcenter_server_6_syslog_4
If we now look at our vRealize Log Insight instance or whatever syslog server you are using, you should now see entries from the vpx.log being forwarded:

vcenter_server_6_syslog_6
You can also perform this change from the command-line by editing the vCenter Server configuration file at /etc/vmware-vpx/vpx.cfg and modifying <outputToSyslog>true</outputToSyslog>

vcenter_server_6_syslog_5
Once you have saved the changes, you will need to restart the vCenter Server by running the following command:

/etc/init.d/vmware-vpxd restart

For those of you who are considering vSphere 6.0 and using the VCSA, this is something I definitely recommend checking out to help simplify the management of both your logs for vCenter Server and your ESXi hosts. I know the VMware Engineering team is working hard on making native syslog support even easier in the future and I look forward to the complete solution hopefully in the near future.

Ultimate automation guide to deploying VCSA 6.0 Part 4: vCenter Server Management Node

In this last and final article, I will share alternative methods of deploying vCenter Server management node using the VCSA 6.0 appliance. Take a look at the various deployment methods below and their respective instructions for more details. If you are deploying using one of the scripts below, you will need to extract the contents of the VCSA ISO. If you are deploying to Workstation/Fusion, you will need to extract the VCSA ISO and add the .ova extension to the following file VMware-VCSA-all-6.0.0-2562643->vcsa->vmware-vcsa before deploying.
vcsa-mgm-node
Disclaimer: Though these alternative deployment options work, they are however not officially supported by VMware. Please use at your own risk.

Deploying to an existing vCenter Server using ovftool (shell script)

I have created a shell script called deploy_vcsa6_mgmt_to_vc.sh which requires using ovftool 4.1 (included in the VCSA ISO) to specify the appropriate OVF "guestinfo" properties for a vCenter Server Management Node deployment. You will need to edit the script and modify several variables based on your environment.

Here is an example of executing the script:

vcsa-6.0-vcenter-server-management-deployment

Deploying to an ESXi host using ovftool (shell script)

I have created a shell script called deploy_vcsa6_mgmt_to_esxi.sh which requires using ovftool 4.0 or greater to specify the appropriate OVF "guestinfo" properties for a vCenter Server Management Node deployment. You will need to edit the script and modify several variables based on your environment. The behavior of this script is similar to the one above, except you are deploying directly to an ESXi host.

Deploying to an existing vCenter Server using ovftool (PowerCLI)

I have created a PowerCLI script called Deployment-VCSA-Mgmt.ps1 which uses ovftool and specifies the appropriate OVF "guestinfo" properties for a vCenter Server Management Node deployment. You will need to edit the script and modify several variables based on your environment.

Deploying to VMware Fusion & Workstation

To properly deploy the new VCSA 6.0, the proper OVF properties MUST be set prior to the booting of the VM. Since VMware Fusion and Workstation do not support OVF properties, you will need to manually deploy the VCSA, but not power it on. Once the deployment has finished, you will need to add the following entries to the VCSA's VMX file and replace it with your environment settings. Once you have saved your changes, you can then power on the VM and the configurations will then be read into the VM for initial setup.

guestinfo.cis.deployment.node.type = "management"
guestinfo.cis.system.vm0.hostname = "192.168.1.50"
guestinfo.cis.vmdir.domain-name = "vghetto.local"
guestinfo.cis.vmdir.site-name = "vghetto"
guestinfo.cis.vmdir.password = "VMware1!"
guestinfo.cis.appliance.net.addr.family = "ipv4"
guestinfo.cis.appliance.net.addr = "192.168.1.64"
guestinfo.cis.appliance.net.pnid = "192.168.1.64"
guestinfo.cis.appliance.net.prefix = "24"
guestinfo.cis.appliance.net.mode = "static"
guestinfo.cis.appliance.net.dns.servers = "192.168.1.1"
guestinfo.cis.appliance.net.gateway = "192.168.1.1"
guestinfo.cis.appliance.root.passwd = "VMware1!"
guestinfo.cis.appliance.ssh.enabled = "true"
guestinfo.cis.appliance.ntp.servers = "0.pool.ntp.org"

For more information, you can take a look at this article here.

Deploying using new supported scripted install (bonus)

As mentioned earlier, there is also a new scripted installer included inside of the VMware-VCSA ISO under /vcsa-cli-installer which supports Windows, Mac OS X and Linux, but must be connected directly to an ESXi host. There are several templates that are also included within the /vcsa-cli-installer/templates. I thought as a bonus I would also share the template I have been using to deploy replicated PSC instances using a static IP Address which some of you may find useful.

The use the scripted installer, you just need to change into the appropriate OS platform directory (win32,mac or lin64) and there should be a binary called vcsa-deploy. To use this template, you just need to save the JSON to a file and then specify that as the first argument to vcsa-deploy utility.

Here is an example of deploying a PSC using the vcsa-deploy scripted installer.

vcsa-6.0-vcenter-server-management-node-scripted-install

How to customize the login UI for vRealize {Operations Manager, Log Insight, Automation}?

With so much excitement and positive feedback (internal/external) regarding my article on customizing the login UI for the new vSphere 6.0 Web Client, I knew it was only a matter of time before folks started asking about customizing other VMware login UIs. As I have mentioned already, going beyond just the aesthetics such as adding an organizations logo or colors, it is often a mandatory requirement for many organizations to display a security or warning banner to users prior to logging in. I was recently added into an internal Socialcast thread asking whether it would be possible to do the same for vRealize Operations Manager (vROps).

I figure I take a quick look to see if this was possible and what it might take. I wanted to also take this opportunity and share a few other solutions that other VMware folks have found in terms of customizing the login UIs for both vRealize Log Insight (thanks GSS Engineer Alan Castonguay for sharing the details) and vRealize Automation (thanks to Justin Jones for his awesome tool). You can find all the details below as well as some additional tidbits through my exploration.

Something that can be helpful in the future as more products integrate with vCenter's SSO (PSC in vSphere 6.0) is that you only need to customize the login page once and it will be available to all other solutions.

Disclaimer: This is not officially supported by VMware. Please make sure to perform a backup of all original files prior to editing in case you need to restore the system defaults.

vROps (vRealize Operations Manager)

Here are the two locations if you wish to customize the login UI for vROps 6.0. The first is the login.jsp file that controls the login UI. If you wish to simply replace the entire image, it will require some tweaking as the login UI is actually composed of several graphical elements making this task a bit more difficult. The second is the images directory which you will want to upload any content you wish to use for the login UI.

Note: Please make sure to perform a backup of all original files prior to editing in case you need to restore the system defaults.

  • /usr/lib/vmware-vcops/tomcat-web-app/webapps/vcops-web-ent/pages/login.jsp
  • /usr/lib/vmware-vcops/tomcat-web-app/webapps/vcops-web-ent/images

Due to the various tweaks, I have created a sample login.jsp which you can download and reference here. This will allow you to replace the entire background for the vROps login UI as well as adding in some text that you wish to display. I know how big of a fan Rawlinson Rivera is of Justin Bieber, so I thought I use his favorite background for creating what an a custom vROps login UI can potentially look like.

customize-vrealize-operations-manager-login-ui

vRLI (vRealize Log Insight)

Here are the two locations if you wish to customize the login UI for vRLI 2.5. The first is the main login background image which is a 600x410 image if you wish to stick with the default layout. The second is a 300x78 transparent image for the vRLI logo, you can either keep this or replace it with your own.

Note: Please make sure to perform a backup of all original files prior to editing in case you need to restore the system defaults.

  • /usr/lib/loginsight/application/3rd_party/apache-tomcat-6.0.36/webapps/ROOT/images/misc/login-bg.png
  • /usr/lib/loginsight/application/3rd_party/apache-tomcat-6.0.36/webapps/ROOT/images/logo/vmware-logo-big-white-v2.png

If you wish to add additional text to the login page, you can edit the following file which controls the login UI.

  • /usr/lib/loginsight/application/3rd_party/apache-tomcat-6.0.36/webapps/ROOT/loginsight/login/login.css

Here is a quick example by inserting the following above Line 20:

Here is an example of what custom login UI for vRLI could potentially look like:

customize-vrealize-log-insight-login-ui

vRA (vRealize Automation)

As a bonus, if you are interested in customizing the Login UI for vRA, be sure to check out fellow Automation colleague Justin Jones who has built this really cool utility called vRA Brand Customizer to help with customizing vRA login UI for the various tenants in your environment. I would recommend keeping an eye on this tool for some really cool stuff coming in the future ;)

Multiple VMDKs in VCSA 6.0?

One thing you might notice after deploying the new VCSA 6.0 is that it now includes 11 VMDKs. If you are like me, you are probably asking why are there so many? If you look at past releases of the VCSA, it only contained two VMDKS. The first disk was used for both the OS and the various VMware applications like vCenter Server, vSphere Web Client, etc. and the second disk was where all the application data was stored such as the VCDB, SSODB, Logs, etc.

There were several challenges with this design, one issue was that you could not easily increase the disk capacity for a particular application component. If you needed more storage for the VCDB but not for your logs or other applications, you had no choice but to increase the entire volume. In fact, this was actually a pretty painful process because a logical volume manager (LVM) was also not used. This meant that you needed to stop the vCenter Server service, add a new disk, format it and then copy all the data from the old volume to the new. Another problem with the old design is that you can not apply Storage QoS on important data such as the VCDB which you may want on a faster tier of storage or putting your Log data on slower and cheaper tier of storage by leveraging something like VM Storage Policies which works on a per VMDK basis.

For these reasons, VCSA 6.0 is now comprised of 11 individual VMDKs as seen in the screenshot below.

11-vmdks-vcsa-6.0-0
Here is a useful table that I have created which provides the mappings of each of the VDMKs to their respective functions.

Disk Size Purpose Mount Point
VMDK1 12GB / and Boot / and /boot
VMDK2 1.2GB Temp Mount /tmp/mount
VMDK3 25GB Swap SWAP
VMDK4 25GB Core /storage/core
VMDK5 10GB Log /storage/log
VMDK6 10GB DB /storage/db
VMDK7 5GB DBLog /storage/dblog
VMDK8 10GB SEAT (specifically events and tasks) /storage/seat
VMDK9 1GB NetDumper /storage/netdump
VMDK10 10GB AutoDeploy /storage/autodeploy
VMDK11 5GB Inventory Service /storage/invsvc

In addition, increasing disk capacity for a particular VMDK has been greatly simplified as the VCSA 6.0 now uses LVM to manage each of the partitions. You can now, on the fly increase disk space for a particular volume while the vCenter Server is still running and the changes will go live immediately. You can refer to this article here for the process as it is a simple two step process.

Here are some useful commands to get more details of the filesystem structure in the new VCSA.

lsblk

11-vmdks-vcsa-6.0-2

isscsi

11-vmdks-vcsa-6.0-3

How to configure SMP-FT using Nested ESXi in vSphere 6?

Symmetric Multi-Processing Fault Tolerance (SMP-FT) has been a long-awaited feature by many VMware customers. With the release of vSphere 6.0, the SMP-FT capability is now finally available and if you want to try out this new feature and see how it works from a "functional" perspective, you can easily do so by running it in a Nested ESXi environment. SMP-FT no longer uses the "record/replay" capability like its younger brother Uniprocessing Fault Tolerance (UP-FT). Instead, SMP-FT now uses a new Fast Checkpointing technique which not only improves the overall performance of its predecessor but also greatly simplifies and reduces additional configurations when running in a Nested ESXi environment.

Disclaimer: Running SMP-FT in a Nested ESXi environment does not replace or substitute actual testing of physical hardware. For any type of performance testing, please test SMP-FT using real hardware.

Requirements:

  • pESXi host running either ESXi 5.5 or 6.0
  • vCenter Server 6.0
  • 2 x Nested ESXi VMs running ESXi 6.0 (vHW9+)
  • Shared storage for the Nested ESXi VMs

Instructions:

Step 1 - Created a Nested ESXi VM using guestOS type "ESXi 5.5/6.0 or later". You will need at least 2 vCPU or greater, 4GB of memory or greater for the installation of ESXi and most importantly, a VMXNET3 network adapter. The reason a VMXNET3 adapter is required is that SMP-FT has a requirement for 10Gbit network connection and the VMXNET3 driver can simulate a 10Gbit connection for a Nested ESXi VM. For further instructions on creating a Nested ESXi VM, please take a look at this article. If you are unable to add VMXNET3 adapter, you may need to first change the guestOS type to "Other 64-bit", add the adapter and then change the guestOS type back.

smp-ft-nested-esxi-0
Step 2 - Install ESXi 6.0 on the Nested ESXi VM and ensure you also have a vCenter Server 6.0 deployed if you have not done so already and add your Nested ESXi instances to a new vSphere Cluster which has vSphere HA enabled.

Step 3 - You will need to enable both vMotion and Fault Tolerance traffic type for the VMkernel interface that you wish to run FT traffic across.

smp-ft-nested-esxi-1
Step 4 - At this point, you can create a real or dummy VM and power it on. Once you have the powered on VM, you can now enable either UP-FT or SMP-FT by right clicking and selecting "Enable Fault Tolerance".

smp-ft-nested-esxi-2
As you can see from the screenshot above, I have successfully enabled FT on a VM with 4vCPU running inside of a Nested ESXi VM, how cool is that!? Hopefully this will help you get more familiar with the new SMP-FT feature when you are ready to give it a real spin on real hardware :)

Note: Intel Sandy Bridge is recommended when using SMP-FT (real physical hardware) but if you have older CPUs, you enable "Legacy FT" mode by adding the following VM Advanced Setting "vm.uselegacyft" to the VM you are enabling FT on.