Avamar: "Could not save console server data to server" due to port scanners causing high Java CPU Utilization
Summary: Port Scanners cause high Java CPU Utilization resulting in a Management Console Server (MCS) flush error: "Could not save console server data to server."
Symptoms
This event can be logged as a Dial Home Event, an MC UI event or seen on a Putty session on the Avamar Utility Node:
Symptom: 22402, Desc: Could not save console server data to server.
The java process on the Avamar Utility Node is at 100% CPU or higher:
top
top - 10:57:10 up 140 days, 19:57, 2 users, load average: 2.55, 2.52, 2.48
Tasks: 187 total, 1 running, 186 sleeping, 0 stopped, 0 zombie
Cpu(s): 74.5%us, 0.2%sy, 0.0%ni, 25.4%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 12028M total, 10789M used, 1238M free, 187M buffers
Swap: 4086M total, 68M used, 4018M free, 7507M cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
12108 admin 20 0 1720m 1.3g 11m S 299 10.9 112556:00 java
13476 admin 20 0 1125m 541m 11m S 0 4.5 157:08.53 java
14055 root 20 0 1485m 158m 10m S 0 1.3 184:58.50 java
One or more of the following errors are detected in the /usr/local/avamar/var/mc/server_log/axis2.log:
2013-01-24 23:01:42,517 [1031746922@qtp-1453087591-170] ERROR org.apache.axis2.engine.AxisEngine - The service cannot be found for the endpoint reference (EPR) /php.cgi?/etc/passwd
org.apache.axis2.AxisFault: The service cannot be found for the endpoint reference (EPR) /php.cgi?/etc/passwd
at org.apache.axis2.engine.DispatchPhase.checkPostConditions(DispatchPhase.java:65)
at org.apache.axis2.engine.Phase.invoke(Phase.java:334)
2014-04-09 12:44:56,908 [main] INFO org.apache.axis2.deployment.ServiceDeployer - Deploying Web service: mcService-server-7.0.0-SNAPSHOT.jar - file:/data01/avamar/var/mc/server_tmp/Jetty_0_0_0_0_9443_mcsdk.axis2.war____.xz4n3v/webapp/WEB-INF/services/mcService-server-7.0.0-SNAPSHOT.jar
2014-04-09 12:44:59,015 [main] INFO org.apache.axis2.deployment.ServiceDeployer - Deploying Web service: mcsdk10-server-7.0.0-SNAPSHOT.jar - file:/data01/avamar/var/mc/server_tmp/Jetty_0_0_0_0_9443_mcsdk.axis2.war____.xz4n3v/webapp/WEB-INF/services/mcsdk10-server-7.0.0-SNAPSHOT.jarCause
High Java utilization on the Avamar Utility Node node can cause the Management Console Server (MCS) flushes (backups) to fail.
An increase in constant CPU utilization after port scans send garbage query strings to the MC web service over port 9443.
As a result, the MCS is not available for the backup process and can cause the flush to fail.
Resolution
1. Confirm if a port scanner (such as Nessus) is scanning any of the Avamar nodes. If possible, disable port scanning to the Avamar Utility Node.
2. Log in to the Avamar Utility Node as admin and load the keys Avamar: How to Log in to an Avamar Server and Load Various Keys.
3. Use the top command to check if java has a high CPU usage (over 100% is high):
top
4. Check if any port 9443 messages appear in the axis log:
grep 9443 /usr/local/avamar/var/mc/server_log/axis2.log
2014-04-09 12:44:56,908 [main] INFO org.apache.axis2.deployment.ServiceDeployer - Deploying Web service: mcService-server-7.0.0-SNAPSHOT.jar - file:/data01/avamar/var/mc/server_tmp/Jetty_0_0_0_0_9443_mcsdk.axis2.war____.xz4n3v/webapp/WEB-INF/services/mcService-server-7.0.0-SNAPSHOT.jar
2014-04-09 12:44:59,015 [main] INFO org.apache.axis2.deployment.ServiceDeployer - Deploying Web service: mcsdk10-server-7.0.0-SNAPSHOT.jar - file:/data01/avamar/var/mc/server_tmp/Jetty_0_0_0_0_9443_mcsdk.axis2.war____.xz4n3v/webapp/WEB-INF/services/mcsdk10-server-7.0.0-SNAPSHOT.jar
5. Verify when the java process was started:
ps -ef | grep java
-
- If the date of the java process matches the data from Step 4 (in this example 2014-04-09), then the process may be in an unresponsive state
- If the date of the java process is new or does not match, stop using this solution
6. Switch to root privilege:
su -
7. Kill the offending java process:
a. Locate the java Process ID (PID):
ps -ef | grep java
b. Kill the process.
kill -9 <PID>
8. Exit from root privilege:
exit
9. Stop and start MCS using Avamar: How to restart Management Control Server (MCS)
10. Verify that the MCS flush is now working and no "22402" errors occur.
mcserver.sh --flush
11. If the error messages persist, review Avamar - Symptom Code 22402 - Could not save console server data to server (Resolution Path) for additional troubleshooting.
If further assistance is required, create a Service Request.