virtuallyGhetto

Search Results for: Log Insight

Configure log forwarding from Tanzu Kubernetes Grid (TKG) to vRealize Log Insight Cloud

by 1 Comment

As much as I enjoy kubectl’ing logs in real time for troubleshooting and debugging purposes, this usually does not scale beyond a couple of Kubernetes (K8s) Clusters if you are lucky. Even then, you will not retain any of the historical logs which may be required for deeper analysis or for auditing purposes. This is usually solved by having a centralized log management platform and while working with Tanzu Kubernetes Grid (TKG) running on VMware Cloud on AWS, a solution like vRealize Log Insight Cloud (vRLIC) makes a lot of sense.

While browsing through the vRLIC console, I noticed that it supports a number of log sources including K8s which was exactly what I was looking for. However, after going through the instructions in configuring fluentd on my TKG Cluster, I found that that nothing was being sent. After a bit of debugging, I realized a few steps were actually missing that was required to setup this up on TKG Cluster.

I eventually figured it out and will be sharing this feedback with the vRLIC folks but in the meantime, you can follow the instructions below on how to forward both system and application logs from your TKG Cluster or any K8s deployment for that matter which has outbound connectivity to connect to vRLIC.


[Read more…] about Configure log forwarding from Tanzu Kubernetes Grid (TKG) to vRealize Log Insight Cloud

How to purge all logs in vRealize Log Insight?

by 10 Comments

I had just deployed a new vRealize Log Insight (vRLI) 4.0 instance in my home lab environment to investigate a behavior that I was seeing with another product, non-vRLI related. Due to the nature of the work, I needed to have a pristine vRLI environment each time to study the results. I had already forwarded some logs into vRLI and rather than deploying another instance or re-deploy the current instance, what I really wanted to be able to do is to just wipe all the logs in vRLI but did not see an option within the UI. I also could have used VM snapshots, but was hoping there was a cleaner solution that vRLI provided out of the box.

The next place I looked immediately after was Mr. Log Insight’s site aka Steve Flanders blog but there was nothing there about this other than archiving. After a few Google searches, I came across this exact same question on the vRLI Ideas site but sadly there was no solution and it was dated back in 2014. Though Steve makes a good point about just letting the logs rotate out automatically, in my case, this was not an option and I needed a pristine environment.

Being the curious one, I figured there has to be a way, even if it is not officially recommended nor supported. As you probably have guessed, I did find a way but I would caution that you read the disclaimer below before proceeding further. This was something I needed to do in my lab to test a few scenarios that was non-vRLI related, but I needed syslog target, so this is why I am using vRLI 🙂

Disclaimer: This is probably not officially supported nor recommended by VMware. Please use at your own risk. YOU WILL LOSE ALL YOUR LOGS

[Read more…] about How to purge all logs in vRealize Log Insight?

How to customize the login UI for vRealize {Operations Manager, Log Insight, Automation}?

by 9 Comments

With so much excitement and positive feedback (internal/external) regarding my article on customizing the login UI for the new vSphere 6.0 Web Client, I knew it was only a matter of time before folks started asking about customizing other VMware login UIs. As I have mentioned already, going beyond just the aesthetics such as adding an organizations logo or colors, it is often a mandatory requirement for many organizations to display a security or warning banner to users prior to logging in. I was recently added into an internal Socialcast thread asking whether it would be possible to do the same for vRealize Operations Manager (vROps).

I figure I take a quick look to see if this was possible and what it might take. I wanted to also take this opportunity and share a few other solutions that other VMware folks have found in terms of customizing the login UIs for both vRealize Log Insight (thanks GSS Engineer Alan Castonguay for sharing the details) and vRealize Automation (thanks to Justin Jones for his awesome tool). You can find all the details below as well as some additional tidbits through my exploration.

Something that can be helpful in the future as more products integrate with vCenter’s SSO (PSC in vSphere 6.0) is that you only need to customize the login page once and it will be available to all other solutions.

Disclaimer: This is not officially supported by VMware. Please make sure to perform a backup of all original files prior to editing in case you need to restore the system defaults.

vROps (vRealize Operations Manager)

Here are the two locations if you wish to customize the login UI for vROps 6.0. The first is the login.jsp file that controls the login UI. If you wish to simply replace the entire image, it will require some tweaking as the login UI is actually composed of several graphical elements making this task a bit more difficult. The second is the images directory which you will want to upload any content you wish to use for the login UI.

Note: Please make sure to perform a backup of all original files prior to editing in case you need to restore the system defaults.

  • /usr/lib/vmware-vcops/tomcat-web-app/webapps/vcops-web-ent/pages/login.jsp
  • /usr/lib/vmware-vcops/tomcat-web-app/webapps/vcops-web-ent/images

Due to the various tweaks, I have created a sample login.jsp which you can download and reference here. This will allow you to replace the entire background for the vROps login UI as well as adding in some text that you wish to display. I know how big of a fan Rawlinson Rivera is of Justin Bieber, so I thought I use his favorite background for creating what an a custom vROps login UI can potentially look like.

customize-vrealize-operations-manager-login-ui

vRLI (vRealize Log Insight)

Here are the two locations if you wish to customize the login UI for vRLI 2.5. The first is the main login background image which is a 600×410 image if you wish to stick with the default layout. The second is a 300×78 transparent image for the vRLI logo, you can either keep this or replace it with your own.

Note: Please make sure to perform a backup of all original files prior to editing in case you need to restore the system defaults.

  • /usr/lib/loginsight/application/3rd_party/apache-tomcat-6.0.36/webapps/ROOT/images/misc/login-bg.png
  • /usr/lib/loginsight/application/3rd_party/apache-tomcat-6.0.36/webapps/ROOT/images/logo/vmware-logo-big-white-v2.png

If you wish to add additional text to the login page, you can edit the following file which controls the login UI.

  • /usr/lib/loginsight/application/3rd_party/apache-tomcat-6.0.36/webapps/ROOT/loginsight/login/login.css

Here is a quick example by inserting the following above Line 20:

Here is an example of what custom login UI for vRLI could potentially look like:

customize-vrealize-log-insight-login-ui

vRA (vRealize Automation)

As a bonus, if you are interested in customizing the Login UI for vRA, be sure to check out fellow Automation colleague Justin Jones who has built this really cool utility called vRA Brand Customizer to help with customizing vRA login UI for the various tenants in your environment. I would recommend keeping an eye on this tool for some really cool stuff coming in the future 😉

Automating Log Insight 2.0 configurations

by Leave a Comment

Last week I had a chance to deploy the latest release of vCenter Log Insight 2.0 (currently in public beta) in my lab to give it a spin. I must say, I am very impressed with the slick new UI and some of the new capabilities like the scale-out and high availability feature.

automate-log-insight-2-0-configuration-0
The actual deployment of the Virtual Appliance is pretty straight forward and the only thing I would mention when selecting the OVF Deployment Size is that the default “Small” option is not the smallest configuration possible. There is actually an “Extra Small” option that can be selected in the drop-down menu which is targeted for POCs and lab evaluations. This will help with minimizing the resource constraints for lab environments.

Something that I am always interested in when evaluating a new solution is to see how easy an automated and unattended configuration is. With the help of some of the Log Insight folks, I was able to create the following shell script which will perform a basic configuration of Log Insight which includes the backend database, admin password and NTP servers:

You will need to edit the following variables within the script:

  • LOG_INSIGHT_ADMIN_PASSWORD
  • LOG_INSIGHT_DB_PASSWORD
  • NTP_SERVERS

Here is an example of running the script against a newly deployed Log Insight system:

automate-log-insight-2-0-configuration-1
The above is just an example of what could be automated for Log Insight. If you take a look at the Configuration section of Log Insight, there are many more options.

automate-log-insight-2-0-configuration-2
If you decide you want to automate additional configurations. The way you would accomplish this is to first configure everything from the Log Insight configuration UI. Once you are happy with the configuration, SSH into your Log Insight system. In /storage/core/loginsight/config you will find a couple of configuration files loginsight-config.xml#X with a numeric number at the end. If you take a look at the file with the highest number, it will contain the latest changes to Log Insight and the configurations you made using the UI. You can then take that file and update the script to automate the other configuration options.

Forwarding Logs From The vCloud Suite To vCenter Log Insight

by 18 Comments

An exciting new product was just announced last week by VMware called vCenter Log Insight, which will be part of the vCenter Operations Management Suite when released. The announcement also includes a public beta for customers to try out the new log analytics product that allows administrators to easily get an understanding of both their physical and virtual infrastructure through the collection of log data. You can get more details on how vCenter Log Insight works by checking out this article by the Jon Herlocker, who is in the Office of CTO and focusing on vCenter Log Insight.

I had known about vCenter Log Insight for quite sometime now and like others within VMware, I had the opportunity to test drive the product early on and provide feedback to the engineering team. One of neatest thing about vCenter Log Insight, in my opinion, is the simplistic setup and the tight integration between vCenter Server and vCenter Operations Manager. During the setup of vCenter Log Insight, I was reminded about an article that I had written about forwarding vCenter Server logs to a syslog server. I thought, would it not be cool if we could forward logs from other products within the vCloud Suite to vCenter Log Insight using the same syslog-ng trick? I decided to compile a list of logs from each of the products within the vCloud Suite shared that internally and thanks to my colleague Michael White who also help vet the list by circulating it within engineering.

I then decided to create a very simple script called configurevCloudSuiteSyslog.sh that would allow users to easily configure each of the vCloud Suite products to forward their appropriate logs to vCenter Log Insight. The script is very simple to use, you just need to scp the script to one of the supported appliances within the vCloud Suite and specify the VMware solution name and the IP Address of your vCenter Log Insight Server.

Here is an example of running the script on the VCSA (vCenter Server Appliance):

Based on the VMware solution selected, the appropriate logs will be appended to /etc/syslog-ng/syslog-ng.conf to be forwarded off to your vCenter Log Insight Server. The syslog-ng client will automatically be restarted for the changes to go into effect as part of the script. In my environment, I have deployed the majority of products within the vCloud Suite installed and have configured each of them to forward their logs to vCenter Log Insight. This can be very useful from a troubleshooting perspective and being able to view and filter through all the relevant logs from a single location.

It was really interesting to see what the next “chattiest” VMware solution was from a log perspective in my environment, which turned out to be VIN after vCenter Server and ESXi host. I hope to see deeper integration between vCenter Log Insight and the rest of the vCloud Suite in future releases, but for now, if you have not tried out vCenter Log Insight, I would highly recommend you give it a try and provide any feedback you may have in the dedicated VMTN community forum.

If you are interested in the specifics logs that are being collected for each of VMware products, you can find the complete list below. Not all products from the vCloud Suite are listed here and some such as vCloud Director and vCloud Networking & Security provide native syslog configuration from the application standpoint which can be configured using either their UIs or APIs.

vCenter Operations Manager Analytics (VCOPS):

/var/log/vmware/diskadd.log
/var/log/vmware/vcops-admin.log
/var/log/vmware/vcops-firstboot.log
/var/log/vmware/vcops-watch.log 

vCenter Operations Manager UI (VCOPS):

/var/log/vmware/admin.log
/var/log/vmware/ciq-firstboot.log
/var/log/vmware/ciq.log
/var/log/vmware/diskadd.log
/var/log/vmware/lastupdate.log
/var/log/vmware/mod_jk.log
/var/log/vmware/vcops-admin.cmd.log
/var/log/vmware/vcops-admin.log
/var/log/vmware/vcops-firstboot.log
/var/log/vmware/vcops-watch.log
/var/log/vmware/diskadd.log
/var/log/vmware/vcops-admin.log
/var/log/vmware/vcops-firstboot.log
/var/log/vmware/vcops-watch.log 

vCenter Orchestrator (VCO):

/opt/vmo/app-server/server/vmo/log/boot.log
/opt/vmo/app-server/server/vmo/log/console.log
/opt/vmo/app-server/server/vmo/log/server.log
/opt/vmo/app-server/server/vmo/log/script-logs.log
/opt/vmo/configuration/jetty/logs/jetty.log 

vCenter Server Appliance (VCSA):

/var/log/vmware/vpx/vpxd.log
/var/log/vmware/vpx/vpxd-alert.log
/var/log/vmware/vpx/vws.log
/var/log/vmware/vpx/vmware-vpxd.log
/var/log/vmware/vpx/inventoryservice/ds.log 

vCloud Connector Node (VCC):

/opt/vmware/hcagent/logs/hca.log 

vCloud Connector Server (VCC):

/opt/vmware/hcserver/logs/hcs.log 

vSphere Data Protection (VDP):

/space/avamar/var/log/av_boot.rb.log
/space/avamar/var/log/dpnctl.log
/space/avamar/var/log/dpnnetutil-av_boot.log
/usr/local/avamar/var/log/dpnctl.log
/usr/local/avamar/var/log/av_boot.rb.log
/usr/local/avamar/var/log/av_boot.rb.err.log
/usr/local/avamar/var/log/dpnnetutil-av_boot.log
/usr/local/avamar/var/avi/server_log/flush.log
/usr/local/avamar/var/avi/server_log/avinstaller.log.0
/usr/local/avamar/var/vdr/server_logs/vdr-server.log
/usr/local/avamar/var/vdr/server_logs/vdr-configure.log
/usr/local/avamar/var/flr/server_logs/flr-server.log
/data01/cur/err.log
/usr/local/avamarclient/bin/logs/VmMgr.log
/usr/local/avamarclient/bin/logs/MountMgr.log
/usr/local/avamarclient/bin/logs/VmwareFlrWs.log
/usr/local/avamarclient/bin/logs/VmwareFlr.log 

vCloud Director (VCD):

/opt/vmware/vcloud-director/logs/vcloud-container-debug.log
/opt/vmware/vcloud-director/logs/vcloud-container-info.log
/opt/vmware/vcloud-director/logs/jmx.log 

vSphere Infrastructure Navigator (VIN):

/var/log/vadm/system.log
/var/log/vadm/engine.log
/var/log/vadm/activecollector.log
/var/log/vadm/dbconfig.log
/var/log/vadm/db/postgresql.log 

vSphere Management Assistance (VMA):

/var/log/vmware/vma/vifpd.log 

vSphere Replication (VR):

/var/log/vmware/hbrsrv.log 

Enhanced vCenter Server Audit Event & Logging in vSphere 6.7 Update 2

by 7 Comments

A couple of years back I had published a detailed analysis on vCenter Server’s Authentication (AuthN) and Authorization (AuthZ) from an auditing and logging standpoint. This has been the go to reference for many of our customers and the posts also includes a number of log samples which I have documented in the following Github repository.

In addition to serving as a reference for our customers, it has also helped our Product and Engineering teams understand where we still had some gaps and how we could improve the overall user experience. As hinted in the recently announced vSphere 6.7 Update 2 release, which will be available soon, there are number of new auditing enhancements that have been made to both vCenter Server and the vCenter Single Sign-On (SSO) service that I think customers will really appreciate.

“Real” client IP address in Events

When you look at a login or logout Event in vCenter Server today, you may have noticed the user’s client IP Address is actually of the vCenter Server rather than the actual remote client’s address and the reason for this is explained here. In vSphere 6.7 Update 2, the real client IP Address is now captured and is included in all successful login/logout and failed logins. This information can now enable administrators to easily identify unauthorized access and be able to quickly track down the systems initiating the connections.

[Read more…] about Enhanced vCenter Server Audit Event & Logging in vSphere 6.7 Update 2