MCS donnot shut down and btw donnot restart after dpnctl cli command

Ideally please do not kill all java on an Avamar grid. There are different services that will terminate if you run killall java so if you need to stop MCS first try:

mcserver.sh --stop

-> this will stop MCS and the MCDB (MCS database).

If the above doesn't work then try:

ps -ef --forest | grep mcserver | awk '{print $2}' | xargs -t -I {} kill {}

-> The above uses a ps -ef --forest to look for MCS.  You can search for mcserver which is what would need to stop.  In some very rare cases multiple users or bad attempts at running dpnctl start/stop mcs may have duplicate processes causing this.

Example:

root@AVAMAR_TEST_GRID:/var/log/#: ps -ef --forest | grep mcserver
admin 148508 1 1 Jun05 ? 04:03:22 /usr/java/latest//bin/java -Xmx1G -XX:+HeapDumpOnOutOfMemoryError -server -ea -cp /usr/local/avamar/lib/avamar-mcsecurity.jar:/usr/local/avamar/lib/commons-cli-1.2.jar:/usr/local/avamar/lib/commons-lang-2.6.jar:/usr/local/avamar/lib/commons-logging-adapters-1.1.jar:/usr/local/avamar/lib/rsa-jsafe-cryptojFIPS-5.0.jar:/usr/local/avamar/lib/rsa-jsafe-util-5.0.jar:/usr/local/avamar/lib/spring-jdbc-3.1.1.RELEASE.jar:/usr/local/avamar/lib/spring-tx-3.1.1.RELEASE.jar:/usr/local/avamar/lib/mcserver.jar:/usr/local/avamar/var/mc/server_data/prefs:lib:/usr/local/avamar/lib/asn_server.jar:/usr/local/avamar/lib/mail.jar:/usr/local/avamar/lib/activation.jar:/usr/local/avamar/lib/xercesImpl.jar:/usr/local/avamar/lib/xmlsec.jar:/usr/local/avamar/lib/xmldsig.jar:lib/postgresql-8.4-702.jdbc4.jar:/usr/local/avamar/lib/jaxrpc.jar:/usr/local/avamar/lib/commons-logging-1.0.4.jar:/usr/local/avamar/lib/commons-validator.jar:/usr/local/avamar/lib/commons-discovery-0.2.jar:/usr/local/avamar/lib/commons-collections-3.2.1.jar:/usr/local/avamar/lib/saaj.jar:/usr/local/avamar/lib/wbem.jar:/usr/local/avamar/lib/wsdl4j-1.5.1.jar:/usr/local/avamar/lib/log4j-1.2.14.jar:/usr/local/avamar/lib/axis.jar:/usr/local/avamar/lib/vim25.jar:/usr/local/avamar/lib/commons-codec-1.3.jar:/usr/local/avamar/lib/jema_api.jar:/usr/local/avamar/lib/jetty-6.1.23.jar:/usr/local/avamar/lib/jetty-util-6.1.23.jar:/usr/local/avamar/lib/servlet-api-3.0.jar:/usr/local/avamar/lib/jsp-api-2.1.jar:/usr/local/avamar/lib/jsp-2.1.jar:/usr/local/avamar/lib/core-3.1.1.jar:/usr/local/avamar/lib/ant-1.6.5.jar:/usr/local/avamar/lib/apputils25.jar:/usr/local/avamar/lib/j2ssh-core-0.2.9.jar:/usr/local/avamar/lib/snmp4j-1.11.jar:/usr/local/avamar/lib/mccommons.jar:/usr/local/avamar/lib/user-authentication.jar:/usr/local/avamar/lib/nis-1.2.1.jar:/usr/local/avamar/lib/providerutil-1.2.1.jar:/usr/local/avamar/lib/commons-lang-2.6.jar:/usr/local/avamar/lib/commons-io-1.4.jar:/usr/local/avamar/lib/jutf7-1.0.0.jar:/usr/local/avamar/lib/i18n:/usr/local/avamar/lib/certj.jar:/usr/local/avamar/lib/rsa-jsafe-cryptojce-6.2.jar:/usr/local/avamar/lib/rsa-jsafe-cryptojcommon-6.2.jar:/usr/local/avamar/lib/rsa-jsafe-jcmFIPS-6.2.jar:/usr/local/avamar/lib/sslj.jar:/usr/local/avamar/lib/amqp-client-3.1.3.jar:/usr/local/avamar/lib/events-dispatcher-api-1.0.0.jar:/usr/local/avamar/lib/events-dispatcher-amqp-1.0.0.jar:/usr/local/avamar/lib/jackson-core-asl-1.9.13.jar:/usr/local/avamar/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/avamar/lib/mcebms-client.jar:/usr/local/avamar/lib/httpclient-4.3.2.jar:/usr/local/avamar/lib/httpcore-4.3.1.jar:/usr/local/avamar/lib/javax.ws.rs-api-2.0-m10.jar:/usr/local/avamar/lib/rabbit-config.jar:/usr/local/avamar/lib/jsch-0.1.53.jar:/usr/local/avamar/lib/aopalliance-1.0.jar:/usr/local/avamar/lib/aspectjweaver-1.6.12.jar:/usr/local/avamar/lib/spring-aop-3.1.1.RELEASE.jar:/usr/local/avamar/lib/spring-asm-3.1.1.RELEASE.jar:/usr/local/avamar/lib/spring-beans-3.1.1.RELEASE.jar:/usr/local/avamar/lib/spring-context-3.1.1.RELEASE.jar:/usr/local/avamar/lib/spring-core-3.1.1.RELEASE.jar:/usr/local/avamar/lib/spring-expression-3.1.1.RELEASE.jar:/usr/local/avamar/lib/spring-amqp-1.2.0.RELEASE.jar:/usr/local/avamar/lib/spring-rabbit-1.2.0.RELEASE.jar:/usr/local/avamar/lib/spring-web-3.1.1.RELEASE.jar:/usr/local/avamar/lib/spring-jdbc-3.1.1.RELEASE.jar:/usr/local/avamar/lib/spring-tx-3.1.1.RELEASE.jar::/usr/local/avamar/lib/spring-security-config-3.1.0.RELEASE.jar:/usr/local/avamar/lib/spring-security-core-3.1.0.RELEASE.jar:/usr/local/avamar/lib/spring-security-crypto-3.1.0.RELEASE.jar:/usr/local/avamar/lib/spring-security-web-3.1.0.RELEASE.jar -Djava.rmi.server.hostname=hop-4s-1x3-util.svt.lab.emc.com -Djava.util.logging.config.file=/usr/local/avamar/lib/mcserver_logging.properties -Djava.security.egd=file:/dev/./urandom -Djava.io.tmpdir=/usr/local/avamar/var/mc/server_tmp -Djava.util.prefs.PreferencesFactory=com.avamar.mc.util.MCServerPreferencesFactory -Djavax.xml.parsers.DocumentBuilderFactory=org.apache.xerces.jaxp.DocumentBuilderFactoryImpl -Djavax.net.ssl.keyStore=lib/rmi_ssl_keystore -Djava
root 55163 52821 0 17:11 pts/1 00:00:00 \_ grep mcserver

Once you kill the above identified Avamar Java processess only.  Then run:

mcserver.sh --stop

-> to make sure the MCS database is stopped.

You should then be able to run the below to start MCS and the schedule service:

mcserver.sh --start --verbose

-> above is a verbose CLI method to watch MCS come up.  Please note: if you are having networking issues this may be delayed if you have multiple vCenters.  For example: if you have 5 vCenters and 3 of them are down due to networking issues (unrelated to MCS) this may have a longer startup time due to MCS attempting to sync to all vCenters upon startup.  If you do have vCenters, wait even if it shows an error upon starting and then run mcserver.sh --status to verify the status about 5-10 minutes later to confirm if it starts or is up normally.

Once up; you can then run: dpnctl start sched to bring back up the schedule service which stops when MCS stops.

Final note:

It's best to contact Avamar Support for these issues; however above is provided so at the very least you can avoid running the killall java or attempt to kill all active java processes in system memory.

If you do run killall java this may stop the ldls (Avamar downloader service), Tape Library servcies, DTLT site, AVI upgrade site, or ASCD service.    If you made a mistake and did run killall java please run: ascd start as the admin user and confirm with Avamar support for this.

Alternatively you could also run the Avamar Proactive Check health check script to confirm the services look okay and verify with support below