Avamar: MCS stops responding or is unresponsive after upgrade

• Avamar was recently upgraded to 19.12.
• Thread dump in mcserver.out may have the following deadlock:

Found one Java-level deadlock:
=============================
"RMI TCP Connection(58477)-10.3.4.58":
  waiting to lock monitor 0x00007f9e0402a1e8 (object 0x0000000704525f40, a com.rsa.sslj.x.aP),
  which is held by "RMI TCP Connection(58299)-10.3.50.30"
"RMI TCP Connection(58299)-10.3.50.30":
  waiting to lock monitor 0x00007f9e10007fe8 (object 0x00000007358f6998, a com.rsa.sslj.x.aR),
  which is held by "RMI TCP Connection(58298)-10.3.50.30"
"RMI TCP Connection(58298)-10.3.50.30":
  waiting to lock monitor 0x00007f9e0402a1e8 (object 0x0000000704525f40, a com.rsa.sslj.x.aP),
  which is held by "RMI TCP Connection(58299)-10.3.50.30"

• No java hprof file is created.

• In mcserver.log this may show:

12/22-08:01:34.00504 [main#1] jdk.internal.event.EventHelper.logX509CertificateEvent
FINE: X509Certificate: Alg:{0}, Serial:{1}, Subject:{2}, Issuer:{3}, Key type:{4}, Length:{5}, Cert Id:{6}, Valid from:{7}, Valid until:{8}
12/22-08:01:34.00542 [main#1] com.avamar.asn.NetworkProxy.printRegistry
WARNING: exception occurs while enumberating RMI registry java.rmi.ConnectException: 
Connection refused to host: avamarhostname.emc.com; nested exception is:

        java.net.ConnectException: Connection refused (Connection refused)

Engineering believes that the following environmental influences can cause issues:

1. Network Issues:
   • Ensure that the server and client can communicate over the network. Check for any network issues or firewall settings that might be blocking the connection.
2. Configuration Problems:
   • Verify that the RMI server is correctly configured to accept connections. Ensure that the java.rmi.server.hostname property is set correctly on the server side1.
   • Using SSL, ensure that the SSL certificates are correctly configured and trusted by both the client and server1.
3. Port Availability:
   • Make sure that the port used for the RMI connection is open and not being used by another application. You can specify a different port if needed.
4. Firewall Settings:
   • Check the firewall settings on both the client and server machines to ensure that the required ports for RMI communication are open.
5. Version Compatibility:
   • Ensure that the Java versions on both the client and server are compatible. Sometimes, differences in Java versions can cause connection issues.

The sslj6.5 version (introduced in Avamar 19.12) has the following changes and differences from the previous version:

Permanent fix in 19.12 SP1 - ETA to be determined.

There are two workarounds that can be done to alleviate this in the meantime.

Workaround option 1: Disable rmi_over_ssl.

1.  Make a backup copy of /usr/local/avamar/var/mc/server_data/prefs/mcserver.xml:

cp -p mcserver.xml orig_mcserver.xml-$(date -I)

mcserver.sh --flush

<entry key="rmi_over_ssl" value="false" />

mcserver.sh --stop --force
mcserver.sh --start

Workaround option 2: Replace sslj.jar file (Open Service Request with Avamar Server Replication Capacity team)

Support Team - review the "Internal Notes" section for steps in replacing the sslj.jar file.