Article Number: 000208326
Avamar 19.7+ has implemented a new feature for checking proxy status. This feature connects to the CIM service on every proxy (with a default polling cycle of 30 minutes) to check the status of services like Jetty, CIM, avagent, FLR, and vmtools.
In some environments, this default proxy status polling cycle can lead to exhaustion of Tomcat memory, resulting in service disruptions such as Snapshot Manager, Deploy Manager (PDM), Client Manager (AAM), and Avamar DTLT homepage. These disruptions will persist until the Tomcat service is restarted.
Snapshot Manager Impact:
When attempting to use the Snapshot Manager, the Snapshot Manager reports a timeout error (curl 28) after 60 minutes."
08:49:00 avvcbimage Info <42650>: Creating snapshot 'Avamar-1668329340b6216cd5819bac4f61bece141dc1f5035c2c07c6', quiesceFS=1 08:49:00 avvcbimage Info <41136>: SnapshotManager: createSnapshot for vmMoref = vm-161785 08:49:00 avvcbimage Info <0000>: SnapshotManager: POST to /snapshotmanager/vm/vm-161785/action/createSnapshot 09:49:00 avvcbimage Info <44024>: POST: curl_easy_perform failed. Path:/snapshotmanager/vm/vm-161785/action/createSnapshot Curl Error: 28 09:49:00 avvcbimage Info <41137>: SnapshotManager: failed to create snapshot 09:49:00 avvcbimage FATAL <0000>: [IMG0009] failed to create snapshot, possible timeoutThen the mark vm annotation will take takes another 60 mins.
09:49:00 avvcbimage Info <43003>: backup Failed, only set annotation to LastBackupStatus 09:49:00 avvcbimage Info <43109>: AvSnapMgr::addSetAnnotation, vm-161785 09:49:00 avvcbimage Info <0000>: SnapshotManager: POST to /snapshotmanager/vm/vm-161785/action/markAnnotations 09:49:00 avvcbimage Info <42146>: Login by loginWS test is OK. 09:49:00 avvcbimage Info <0000>: Entering the 'final' phase of termination, timer trigger 10:49:00 avvcbimage Info <44024>: POST: curl_easy_perform failed. Path:/snapshotmanager/vm/vm-161785/action/markAnnotations Curl Error: 28
root@ave198:/usr/local/avamar/var/log/vcs/#: grep 'java.lang.OutOfMemoryError' /usr/local/avamar-tomcat/logs/* /usr/local/avamar-tomcat/logs/catalina.2022-12-31.log: java.lang.OutOfMemoryError: Java heap space /usr/local/avamar-tomcat/logs/catalina.2022-12-31.log: java.lang.OutOfMemoryError: Java heap space /usr/local/avamar-tomcat/logs/catalina.2022-12-31.log: java.lang.OutOfMemoryError: Java heap spaceProxy Deployment Manager:
Avamar Primary Bug 337410 that in some environments, the default proxy status polling cycle of 30 minutes can lead to exhaustion of Tomcat memory.
# Is enable status check schedule, default value is true enable_status_check_schedule=trueAfter
# Is enable status check schedule, default value is true enable_status_check_schedule=false
Short-term workaround for backup failures:
To resolve backup failures without restarting Tomcat or making changes to the dm.properties server file, you can disable the Snapshot Manager on the Avamar proxy.
For further guidance, please refer to the following Knowledge Base article: KB 000181611 'How to Disable Snapshot Manager on the Avamar Proxy.'"
26 Jun 2023
7
Solution