PowerEdge: How to Make VMware vCenter Server Appliance Log Rotation More Effective

Facts:
VMware vCenter Server Appliance (vCSA) 7.x

vCSA 7.x log rotation is less effective than previous versions. By default, rotations are slower, more events are kept, and the /storage/log/rsyslogd directory grows rapidly. This leads to excessive size of log bundles and the /storage/log location filling up.

Procedure:
Set up a more efficient log rotation using the steps below:

Always take an offline snapshot of your vCenter Server making any changes.

1. Reduce the log rotation of the EAM service by setting a more appropriate limit of days to keep in rotation, for example 30 days. See VMware article 85249.

On the vCenter Server shell, run the following commands:

1. Back up the file:

   /usr/lib/vmware-eam/web/conf/server.xml       
   # cp /usr/lib/vmware-eam/web/conf/server.xml /usr/lib/vmware-eam/web/conf/server.xml.bak

2. Using the text editor vi, modify the server.xml file that was backed up in the first step. Locate the line:

   <Valve className="org.apache.catalina.valves.AccessLogValve"

3. Add the following parameter:

4. maxDays="<numberOfDays>"

   Save the changes, and restart the ESXi Agent Manager service.

   See the example screenshot below:
    

   Figure 1: Max Days Setting

5. Restart the EAM:

   # service-control --restart vmware-eam

6. Correct issue with imfile overshoot. See VMware article 81327.
7. Take a backup of the original postgres-archiver syslog configuration:

   # cp /etc/vmware-syslog/vmware-services-vmware-postgres-archiver.conf /etc/vmware-syslog/vmware-services-vmware-postgres-archiver.conf.orig

8. Using text editor vi, edit:

   /etc/vmware-syslog/vmware-services-vmware-postgres-archiver.conf 

9. Comment out the existing content, using #, and add the content below:

   # vmware-postgres-archiver logs
   input(type="imfile"
   File="/var/log/vmware/vpostgres/pg_archiver.log.stdout"
   Tag="postgres-archiver"
   Severity="info"
   Facility="local0")
   input(type="imfile"
   File="/var/log/vmware/vpostgres/pg_archiver.log.stderr"
   Tag="postgres-archiver"
   Severity="info"
   Facility="local0")
      d. Restart rsyslog:
          # systemctl restart rsyslog

10. Set log rotation configurations for SSO, EAM, and lookups
11. Copy the files that are attached to VMware article 85475 "configs.zip" to the respective file listed below in vCenter Server Appliance:

    configs.zip                                     >  Your vCenter Server
    -----------                                         -------------------
    SSO_server.xml                            >  /usr/lib/vmware-sso/vmware-sts/conf/server.xml
    SSO_logging.properties                >  /usr/lib/vmware-sso/vmware-sts/conf/logging.properties
    LOOKSVC_server.xml                   >  /usr/lib/vmware-lookupsvc/conf/server.xml
    LOOKSVC_logging.properties       >  /usr/lib/vmware-lookupsvc/conf/logging.properties

12. Restart the vmware-stsd and lookups services:

    # service-control --stop vmware-stsd && service-control --start vmware-stsd 
    
    # service-control --stop lookupsvc && service-control --start lookupsvc

13. Copy "tomcat_compress.cron" file "tomcat_compress.zip" to /etc/cron.d on the vCenter Server appliance.
14. If required, reduce the logrotate from four weeks to two weeks using vi to edit the file:

    /etc/logrotate.conf

15. If required, reduce the procstate log rotation from 10 to 5 using vi to edit the file:

    /etc/logrotate.d/procstate.lr

16. Stop the rsyslog service. This command does not fully stop the service. A reboot is necessary during a change window.

    systemctl stop rsyslog 

17.  Clean out the /storage/log/vmware/rsyslogd directory.
18. The /storage/log location and the /var/log locations are the same.