Avamar: "Could not save console server data to server" due to port scanners causing high Java CPU Utilization

This event can be logged as a Dial Home Event, an MC UI event or seen on a Putty session on the Avamar Utility Node:

Symptom: 22402, Desc: Could not save console server data to server.

The java process on the Avamar Utility Node is at 100% CPU or higher:

top

top - 10:57:10 up 140 days, 19:57,  2 users,  load average: 2.55, 2.52, 2.48
Tasks: 187 total,   1 running, 186 sleeping,   0 stopped,   0 zombie
Cpu(s): 74.5%us,  0.2%sy,  0.0%ni, 25.4%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:     12028M total,    10789M used,     1238M free,      187M buffers
Swap:     4086M total,       68M used,     4018M free,     7507M cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
12108 admin     20   0 1720m 1.3g  11m S  299 10.9 112556:00 java
13476 admin     20   0 1125m 541m  11m S    0  4.5 157:08.53 java
14055 root      20   0 1485m 158m  10m S    0  1.3 184:58.50 java

One or more of the following errors are detected in the /usr/local/avamar/var/mc/server_log/axis2.log:

2013-01-24 23:01:42,517 [1031746922@qtp-1453087591-170] ERROR org.apache.axis2.engine.AxisEngine  - The service cannot be found for the endpoint reference (EPR) /php.cgi?/etc/passwd
org.apache.axis2.AxisFault: The service cannot be found for the endpoint reference (EPR) /php.cgi?/etc/passwd
        at org.apache.axis2.engine.DispatchPhase.checkPostConditions(DispatchPhase.java:65)
        at org.apache.axis2.engine.Phase.invoke(Phase.java:334)

2014-04-09 12:44:56,908 [main] INFO  org.apache.axis2.deployment.ServiceDeployer  - Deploying Web service: mcService-server-7.0.0-SNAPSHOT.jar - file:/data01/avamar/var/mc/server_tmp/Jetty_0_0_0_0_9443_mcsdk.axis2.war____.xz4n3v/webapp/WEB-INF/services/mcService-server-7.0.0-SNAPSHOT.jar
2014-04-09 12:44:59,015 [main] INFO  org.apache.axis2.deployment.ServiceDeployer  - Deploying Web service: mcsdk10-server-7.0.0-SNAPSHOT.jar - file:/data01/avamar/var/mc/server_tmp/Jetty_0_0_0_0_9443_mcsdk.axis2.war____.xz4n3v/webapp/WEB-INF/services/mcsdk10-server-7.0.0-SNAPSHOT.jar

High Java utilization on the Avamar Utility Node node can cause the Management Console Server (MCS) flushes (backups) to fail.

An increase in constant CPU utilization after port scans send garbage query strings to the MC web service over port 9443.

As a result, the MCS is not available for the backup process and can cause the flush to fail.

1. Confirm if a port scanner (such as Nessus) is scanning any of the Avamar nodes. If possible, disable port scanning to the Avamar Utility Node.

2. Log in to the Avamar Utility Node as admin and load the keys Avamar: How to Log in to an Avamar Server and Load Various Keys.

3. Use the top command to check if java has a high CPU usage (over 100% is high):

top

4. Check if any port 9443 messages appear in the axis log:

grep 9443 /usr/local/avamar/var/mc/server_log/axis2.log

2014-04-09 12:44:56,908 [main] INFO  org.apache.axis2.deployment.ServiceDeployer  - Deploying Web service: mcService-server-7.0.0-SNAPSHOT.jar - file:/data01/avamar/var/mc/server_tmp/Jetty_0_0_0_0_9443_mcsdk.axis2.war____.xz4n3v/webapp/WEB-INF/services/mcService-server-7.0.0-SNAPSHOT.jar
2014-04-09 12:44:59,015 [main] INFO  org.apache.axis2.deployment.ServiceDeployer  - Deploying Web service: mcsdk10-server-7.0.0-SNAPSHOT.jar - file:/data01/avamar/var/mc/server_tmp/Jetty_0_0_0_0_9443_mcsdk.axis2.war____.xz4n3v/webapp/WEB-INF/services/mcsdk10-server-7.0.0-SNAPSHOT.jar

5. Verify when the java process was started:

ps -ef | grep java

• • If the date of the java process matches the data from Step 4 (in this example 2014-04-09), then the process may be in an unresponsive state
  • If the date of the java process is new or does not match, stop using this solution

6. Switch to root privilege:

su - 

7. Kill the offending java process:

a. Locate the java Process ID (PID):

ps -ef | grep java

b. Kill the process.

kill -9 <PID>

8. Exit from root privilege:

exit

9. Stop and start MCS using Avamar: How to restart Management Control Server (MCS)

10. Verify that the MCS flush is now working and no "22402" errors occur.

mcserver.sh --flush

11. If the error messages persist, review Avamar - Symptom Code 22402 - Could not save console server data to server (Resolution Path) for additional troubleshooting.

If further assistance is required, create a Service Request.