• Skip to primary navigation
  • Skip to main content
  • Skip to primary sidebar

virtuallyGhetto

  • About
  • Privacy
  • VMware Cloud
  • Home Lab
  • Nested Virtualization
  • Automation
    • VMware Kickstart
    • VMware API/SDK/CLI
    • VMware vMA/VIMA
    • VMware OVF / OVFTOOL
  • Apple Mac
  • VCSA
  • VSAN

esxi 6.0

Log filtering capability in ESXi 6.0

05/20/2015 by William Lam 4 Comments

When it comes to troubleshooting, something that you can never have too much of are logs! However, you can have excessive logs in the form of repeated log entries for a particular event which not only adds to the amount of logs you must sift through but it also adds unnecessary load and processing to the network. This is especially problematic if you these repeated entries also being forwarded from multiple sources to a centralized syslog server.

With earlier release of ESX (yes, classic ESX), it was possible to filter out specific log entries from the host side and prevent them from showing up in the local logs stored on the filesystem after N-occurrences and would also prevent them from being forwarded to a syslog server. However, when ESXi was first introduced, this particular capability wast not ported over which I can only assume was based on usage from our customer base. In speaking with GSS, this is usually something they require for troubleshooting purposes, although I have also seen a few customers ask about this capability on several occasions in the past.

Having said that, I was pleasantly surprised to learn from Alan Castonguay (former GSS Engineer, now working over in our MBU) that ESXi 6.0 actually now includes a new log filtering capability or rather I should say, it has re-introduced the log filtering capability 🙂

To enable log filtering, you will need to add the following parameter to /etc/vmsyslog.conf

enable_logfilters = true

To add a specific log filter, you will need to add an entry to /etc/vmware/logfilters using the following format: numLogs | ident | logRegexp

  • numLogs - Number of times the log entry can appear in the log before it is then filtered and ignored
  • ident - The ident source of the log, you can find all ident by looking in /etc/vmsyslog.conf.d/*.conf
  • logRegexp - regular expression confirming to the Python regular expression syntax

Below is an example where I only want the log entry "SOCKET connect failed, error 2: No such file or directory.*" to show up only two times in the logs before it is filter and ignored and the source of this log entry is from the hostd logs.

2| Hostd | SOCKET connect failed, error 2: No such file or directory.*

UPDATE (03/18/16): I just got a heads up from GSS that the "hostd" logs is actually case sensitive when creating the filter. You will need to use a capital "H" in case you are not seeing results from your regexp.

Once you have created all your log filters, you will need to reload the syslog service by running the following ESXCLI command:

esxcli system syslog reload

One thing to be aware of is that once a log entry has been filtered out and the local logs have been rotated out, that particular entry will no longer show up in future logs. It is definitely recommended that you use the log filtering feature sparingly and ensure that you are also forwarding your logs to a centralized syslog server like Log Insight for example so that you have all log entries at your disposal for both troubleshooting and auditing purposes. There is already a request to add this to our official VMware documentation so that it customers can easily find this in the syslog configuration section of the ESXi documentation, but for now I have documented it here so others can benefit from this capability if needed.

Share this...
  • Twitter
  • Facebook
  • Linkedin
  • Reddit
  • Pinterest

Filed Under: ESXi, vSphere 6.0 Tagged With: esxi 6.0, logfilter, syslog, vSphere 6.0

Easily automate ESXi 6.0 Active Directory join using domainjoin-cli

04/06/2015 by William Lam 8 Comments

A nice little enhancement that I recently came across in ESXi 6.0 is the inclusion of the Likewise utility called domainjoin-cli which allows you to join a system to an Active Directory Domain. Previously, if you wanted to automate the process of joining an ESXi host to an Active Directory Domain, you had to either manually configure it using the vSphere Web/Client, using Host Profiles or creating an external script using the vSphere APIs.

All of these options were mostly executed during the post-provisioning process and if you wanted to include Active Directory configuration as part of the provisioning process, you may have had to resort to something like calling into the vSphere MOB within a Kickstart script as I had shown back in 2011 in this article here. The solution I came up with was not ideal but it worked for those that did not want to have additional steps after initial provisioning.

With the domainjoin-cli utility now included in the ESXi Shell of ESXi 6.0, you easily automate the joining an Active Directory Domain with just a couple of lines added to your Kickstart or provisioning scripts. Before you can use the command-line utility, you will need to ensure the Likewise Service Manager Daemon is running by running the following two commands which will start the service and also ensure the service automatically starts up:

/etc/init.d/lwsmd start
chkconfig lwsmd on

esxi6_active_domain_join_1
Next, to join to your Active Directory Domain, you will need to specify the following 3 parameters:

  1. join - Specifying the operation is a join versus a leave
  2. AD Domain Name - Active Directory Domain to join
  3. AD Username - Active Directory username to join to the domain
  4. AD Password - Active Directory password to join to the domain (optional as you will be prompted if it is not specified)

Here is an example of what the command looks like joining my Active Directory Domain in my lab:

/usr/lib/vmware/likewise/bin/domainjoin-cli join primp-industries.com administrator [PASSWORD]

esxi6_active_domain_join_2
You should see a success message if the ESXi host was successfully joined to the Active Directory Domain and you will want to reboot your ESXi host for the changes to take full effect. This is definitely a simpler method to include into an ESXi Kickstart script to automate the joining of an Active Directory Domain and hopefully you will find this handy when using ESXi 6.0.

Share this...
  • Twitter
  • Facebook
  • Linkedin
  • Reddit
  • Pinterest

Filed Under: Automation, ESXi, vSphere 6.0 Tagged With: active directory, domainjoin-cli, esxi 6.0, kickstart, lwsmd, vSphere 6.0

Quick Tip – Upgrading VMware Tools for Nested ESXi 6.0

04/03/2015 by William Lam 2 Comments

I have received several questions about this in the last couple of weeks regarding the process of upgrading VMware Tools for running Nested ESXi 5.x and 6.0 when the physical ESXi host has been upgraded to ESXi 6.0. Instead of individual replies, I thought I would share this quick tip. First off, VMware Tools for Nested ESXi provides a very specific set of capabilities for Nested ESXi guests as shown below:

  • Provides guest OS information of the nested ESXi Hypervisor (eg. IP address, configured hostname, etc.).
  • Allows the nested ESXi VM to be cleanly shut down or restarted when performing power operations with the vSphere Web/C# Client or vSphere APIs.
  • Executes scripts that help automate ESXi guest OS operations when the guest’s power state changes.
  • Supports the Guest Operations API (formally known as the VIX API).

Unlike traditional VMware Tools which may provide updated capabilities with each new release, VMware Tools for Nested ESXi exposes only a subset of those capabilities which has not changed between ESXi 5.x and 6.0. This is an important fact to be aware of because you may see "Unsupported older version" for the VMware Tools status in the vSphere Web/C# Client and this is perfectly fine and expected.

Here is a screenshot of a Nested ESXi 5.5 VM with VMware Tools installed running on top of an upgraded physical ESXi 6.0 host:

upgrading_nested_esxi_vmware_tools_vsphere_6_1
In this scenario, the VMware Tools status will be reported as "Unsupported older version" because the version of VMware Tools does not match the latest version of VMware Tools included with ESXi 6.0. However, you should not be alarm as the expected functionality listed above will continue to work without any problems and you can just ignore the UI warning. The only way to get rid of this warning is to upgrade the Nested ESXi VM to ESXi 6.0 which I go over in more details below. I know upgrading may not be an option if you still wish to run ESXi 5.x, but as far as I know, there will not be an update to VMware Tools VIB for ESXi 6.0 as it is now pre-installed with ESXi 6.0.

Here is a screenshot of the same VM which has now been upgraded to ESXi 6.0 running on top of an upgraded physical ESXi 6.0 host:

upgrading_nested_esxi_vmware_tools_vsphere_6_2
In this case, the VMware Tools status will be reported as "Unsupported older version" because the version of VMware Tools does not match the latest version of VMware Tools included with ESXi 6.0. However, because VMware Tools now comes pre-installed with ESXi 6.0. We can easily remedy this by removing the VMware Tools VIB we installed for ESXi 5.x by running the following ESXCLI command and then rebooting:

esxcli software vib remove -n esx-tools-for-esxi

Once the ESXi host has rebooted, the VMware Tools that is pre-installed with ESXi 6.0 will automatically start up if it detects it is running as a VM. If you now look at your vSphere Web/C# Client, you will see that VMware Tools status shows current and is also the default behavior if you are running Nested ESXi 6.0 VM on top of physical ESXi 6.0 host.

upgrading_nested_esxi_vmware_tools_vsphere_6_3
With VMware Tools being pre-installed with ESXi 6.0 and only loaded when it detects it is being run as a VM, you no longer need to worry about manually installing additional VIBs get the benefits of having VMware Tools installed for your Nested ESXi VMs.

Share this...
  • Twitter
  • Facebook
  • Linkedin
  • Reddit
  • Pinterest

Filed Under: ESXi, Nested Virtualization, vSphere 6.0 Tagged With: esxi 6.0, nested, nested virtualization, vmtoolsd, vmware tools, vSphere 6.0

A preview of native syslog support in VCSA 6.0

03/30/2015 by William Lam 29 Comments

Proper logging of VMware hosts, services and application logs are becoming more and more critical these days and their usage goes beyond just troubleshooting. In many of our customer environments, extended log retention is often mandatory to satisfy auditing and compliance requirements. Support for remote syslog has been around in ESXi for quite some time and has included several enhancements over the years, however logging for vCenter Server itself has not changed much over the years. Historically, vCenter Server started out as a Windows application and outside of standard filesystem logging there is also Microsoft Event Logs which was not really all that useful. With the release of the vCenter Server Appliance (VCSA), syslog support became more attainable, at least without additional 3rd party tools.

I can even remember when I was an administrator, I had to get creative on how to forward vCenter Server logs to a remote syslog server which I had blogged about back in 2012. Though the solution works, it was not ideal especially when you are running several dozen to several hundred vCenter Server instances like many of our customers do today. When I had discovered that there was a Common Logging initiative within VMware for vSphere 6.0, I was pretty excited and I can only guess that this also put a big smile on many of our GSS folks faces 😉

As you can imagine this was no small undertaking, especially with the organic growth of services and applications within vCenter Server. The goal was not only to support native remote syslog but to also standardize on the location, rotation, retention of all the logs and most importantly providing a consistent time stamp of events so that an administrator or 3rd party tool can easily correlate operations across multiple VMware log files. Though complete native syslog support in vCenter Server is not 100% ready just yet, much of the plumbing and foundation has already been finished and in fact you can see some of this in the latest release VCSA 6.0.

With VCSA 6.0, there is partial support for native remote syslog which is configurable through the VMware Syslog Service under the new vCenter Server System Configuration found within the vSphere Web Client.

vcenter_server_6_syslog_1
There are four settings that you will need to configure:

  • Common Log Level - * (everything), info, notice, warn, error, crit, alert & emerg
  • Host - Hostname/IP Address of a *single* remote syslog server
  • Port - Port of the remote syslog server (514 for UDP & 1514 for TCP is already opened on the VCSA firewall)
  • Protocol - Supports tcp, udp & tls

A restart is not required when configuring the syslog service and logs will automatically be forwarded to the remote syslog server which is quite nice. You can also view the health status of the syslog service and its connectivity to the remote syslog server by clicking onto the "Summary" view as seen in the screenshot below. For more information about the new syslog service, check out the official documentation here.

vcenter_server_6_syslog_2
So what exactly does partial syslog support really mean? What logs are being forwarded to a syslog server when the syslog service is enabled?

There are currently two major sets of logs that are forwarded to a remote syslog server when the new syslog service is configured:

  1. All logs from ESXi hosts that are connected to the vCenter Server will be forwarded
  2. A partial set of vCenter Server services (details in table below) will be forwarded
Service Name Service Description Service Log Location
applmgmt-audit Appliance Management /var/log/vmware/applmgmt/applmgmt-audit/applmgmt-audit-syslog.log
audispd Audit Event Dispatcher /var/log/audit/audispd/audispd-syslog.log
auditd Audit System /var/log/audit/auditd/auditd-syslog.log
rbd Auto Deploy /var/log/vmware/rbd/rbd-syslog.log
vmafdd VMware Authentication Framework /var/log/vmware/vmafdd/vmafdd-syslog.log
vmcad VMware Certificate Service /var/log/vmware/vmcad/vmcad-syslog.log
vmdird VMware Directory Service /var/log/vmware/vmdird/vmdird-syslog.log
watchdog-rhttpproxy Watchdog for Reverse HTTP Proxy service /var/log/vmware/rhttpproxy/watchdog-rhttpproxy/watchdog-rhttpproxy-syslog.log
watchdog-syslog Watchdog for Syslog service /var/log/vmware/syslog/watchdog-syslog/watchdog-syslog-syslog.log
watchdog-vmware-vpostgres Watchdog for vPostgres DB service /var/log/vmware/vpostgres/watchdog-vmware-vpostgres/watchdog-vmware-vpostgres-syslog.log
watchdog-vpxd Watchdog for vCenter Server service /var/log/vmware/vpxd/watchdog-vpxd/watchdog-vpxd-syslog.log
watchdog-vws Watchdog for vCenter Web Services service /var/log/vmware/vws/watchdog-vws/watchdog-vws-syslog.log

Note: The information above was extracted from /etc/vmware-syslog/custom-file-location.conf

Here is a screenshot of my vRealize Log Insight instance ingesting the logs that have been forwarded over from my VCSA 6.0:

vcenter_server_6_syslog_7
Although not all the vCenter Server services have been integrated into this new native syslog mechanism, you can see where things headed and hopefully in the not too distant future we will have full native syslog support for all application and system logs found withint vCenter Server. One thing that I really do like is that I can go to one single location to configure my remote syslog server and automatically receive all logs from the ESXi hosts being managed by that vCenter Server and forwarded to the configured syslog server. This definitely makes it operationally friendly so that you have one less thing to configure when provisioning new ESXi hosts.

One limitation that I found when configuring your remove syslog server is that there is no way to reset the values to NULL and the UI also limits the number of remote syslog server to just one, even though you can specify multiple targets. One way to get around this UI limitation is by editing the underlying configuration file which is located in /etc/vmware-syslog/syslog.conf

Here is an example of what the syslog.conf looks like for the above configuration:

*.info @log.primp-industries.com:514;RSYSLOG_SyslogProtocol23Format

If you wish to add a second or even third syslog server, you simply just need to duplicate the existing line and update the hostname or IP Address of your syslog server.

*.info @log.primp-industries.com:514;RSYSLOG_SyslogProtocol23Format
*.info @log2.primp-industries.com:514;RSYSLOG_SyslogProtocol23Format

If you are manually editing the syslog.conf, you will need to restart the syslog service by running the following command for the changes to take effect:

/etc/init.d/vmware-syslog restart

Some of you might say this is great and all, but one of the most important log files which is the vCenter Server log (vpxd.log) is not being being forwarded. How useful is this really to me? I know I definitely asked that question 🙂 Though not ideal, there is a small configuration change you can apply to easily get vpxd.log to also forward to a remote syslog server using the new syslog service.

You will need to change the vCenter Server advanced setting "config.log.outputToSyslog" property (can also be done using vSphere API) from false to true as seen in the screenshot below.

vcenter_server_6_syslog_3
The above assumes you have already configured the syslog service and for this change to go into effect, you will need to restart the vCenter Server service. This can be done using the System Configuration and under the vCenter Server Service, by just right clicking and selecting "Restart".

vcenter_server_6_syslog_4
If we now look at our vRealize Log Insight instance or whatever syslog server you are using, you should now see entries from the vpx.log being forwarded:

vcenter_server_6_syslog_6
You can also perform this change from the command-line by editing the vCenter Server configuration file at /etc/vmware-vpx/vpxd.cfg and modifying <outputToSyslog>true</outputToSyslog>

vcenter_server_6_syslog_5
Once you have saved the changes, you will need to restart the vCenter Server by running the following command:

/etc/init.d/vmware-vpxd restart

For those of you who are considering vSphere 6.0 and using the VCSA, this is something I definitely recommend checking out to help simplify the management of both your logs for vCenter Server and your ESXi hosts. I know the VMware Engineering team is working hard on making native syslog support even easier in the future and I look forward to the complete solution hopefully in the near future.

Share this...
  • Twitter
  • Facebook
  • Linkedin
  • Reddit
  • Pinterest

Filed Under: ESXi, vSphere 6.0 Tagged With: esxi 6.0, syslog, vCenter Log Insight, vCenter Server, vcenter server appliance, vcsa, vcva, vmsyslog, vpx.cfg, vpxd.log, vSphere 6.0

  • « Go to Previous Page
  • Go to page 1
  • Go to page 2
  • Go to page 3
  • Go to page 4

Primary Sidebar

Author

William Lam is a Senior Staff Solution Architect working in the VMware Cloud team within the Cloud Services Business Unit (CSBU) at VMware. He focuses on Automation, Integration and Operation for the VMware Cloud Software Defined Datacenters (SDDC)

  • Email
  • GitHub
  • LinkedIn
  • RSS
  • Twitter
  • Vimeo

Sponsors

Privacy & Cookies: This site uses cookies. By continuing to use this website, you agree to their use.
To find out more, including how to control cookies, see here: Cookie Policy