Avamar: MCS does not start with an error: Server already started on port: 7778

Attempting to restart the Management Console Server (MCS) service on Avamar fails with the following error:

Closing all connections to database. Server already started on port: 7778

The log (/usr/local/avamar/var/mc/server_log/mcserver.out) shows the following:

=== BEGIN === check.mcs (prestart)
check.mcs                        passed
=== PASS === check.mcs PASSED OVERALL (prestart)
Starting Administrator Server at Fri Jan 10 11:06:39 ICT 2025 ...
Upgrade MCS Preference: processing time = 728ms
Upgrade MCDB: processing time = 5165ms
Check node list: processing time = 2294ms
Prepare MC SSL Properties: processing time = 1ms
Administrator Server startup failed at Fri Jan 10 11:06:53 ICT 2025: Closing all connections to database. Server already started on port: 7778

An old stalled MCS or Java Process (PID) is still running, despite the MCS service being down.
 

Confirm the issue by doing the following root (elevate if required):

1. Run the following command:

netstat -tulpn |egrep "7778|Active|Proto"

Example output:

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 :::7778                 :::*                    LISTEN      109714/java 

2. Then run the following command (substituting the PID number, shown in green):

ps -eaf |grep 109714 |grep -v "grep" |cut -c1-130

Example output:

UID         PID   PPID  C STIME TTY          TIME CMD
admin     109714      1  3 Jul05 ?        17:54:23 /usr/java/latest/bin/java -Xmx1G -XX:+HeapDumpOnOutOfMemoryError -server -ea -cp

Once it has been confirmed that this is the issue, do the following:

1.Manually kill the old MCS or Java process:

a. Verify the PID number (based on the netstat -tulpn output above where the PID is 109714):

ps -eaf |grep 109714 |grep -v "grep" |cut -c1-130

Example output:

UID         PID   PPID  C STIME TTY          TIME CMD
admin     109714      1  3 Jul05 ?        17:54:23 /usr/java/latest/bin/java -Xmx1G -XX:+HeapDumpOnOutOfMemoryError -server -ea -cp

b. Kill the process:

kill 109714

c. Verify that the PID is no longer running:

ps -eaf |grep 109714 |grep -v "grep" |cut -c1-130

Example output:

UID         PID   PPID  C STIME TTY          TIME CMD

2. Switch to admin:

su - admin

3. Restart MCS per Avamar: How to Restart Management Console Server

If there are still issues, create a swarm with the SCR team.

Sample restart output:

...
dpnctl: INFO: Starting MCS...
dpnctl: INFO: To monitor progress, run in another window: tail -f /tmp/dpnctl-mcs-start-output-52848
        ([ -r /etc/profile ] && . /etc/profile ; /usr/local/avamar/bin/mcserver.sh --start ; echo $? >/tmp/dpnctl-mcs-start-status-52848) >/tmp/dpnctl-mcs-start-output-52848 2>&1
        /bin/cat /tmp/dpnctl-mcs-start-output-52848
[ "/bin/cat /tmp/dpnctl-mcs-start-output-52848" exit status = 0 ]
        /bin/cat /tmp/dpnctl-mcs-start-status-52848 2>&1
[ "/bin/cat /tmp/dpnctl-mcs-start-status-52848 2>&1" exit status = 0 ]
output of "[ -r /etc/profile ] && . /etc/profile ; /usr/local/avamar/bin/mcserver.sh --start":
- - - - - - - - - - - - - - - BEGIN
Database server is running...
INFO: Starting messaging service.
INFO: Started messaging service.
=== BEGIN === check.mcs (prestart)
check.mcs                        passed
=== PASS === check.mcs PASSED OVERALL (prestart)
Starting Administrator Server at: Wed Jul 26 11:10:24 BRT 2024
Starting Administrator Server...
Warning: org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser: Property 'http://www.oracle.com/xml/jaxp/properties/entityExpansionLimit' is not recognized.
Compiler warnings:
  WARNING:  'org.apache.xerces.jaxp.SAXParserImpl: Property 'http://javax.xml.XMLConstants/property/accessExternalDTD' is not recognized.'
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Administrator Server started.
INFO: Starting Data Domain SNMP Manager....
INFO: Connecting to MCS Server: avamar.company.com at port: 7778...
INFO: Successfully connected to MCS Server: avamar.company.com at port: 7778.
INFO: No trap listeners were started, Data Domain SNMP Manager didn't start.
- - - - - - - - - - - - - - - END
...