Astreinte
1 Copper

MCS donnot shut down and btw donnot restart after dpnctl cli command

Jump to solution

Hello,

As we experienced some troubles to operate administration and maintenance, we planned to restart MCS like we always did.

dpnctl stop mcs && dpnctl start mcs,sched

 

But we are in the same state since 5 hours....nothing more tha, "Shutting down MCS..."

 

admin@AVAMAR:~/>: dpnctl stop mcs && dpnctl start mcs,sched
Identity added: /home/admin/.ssh/admin_key (/home/admin/.ssh/admin_key)

dpnctl: INFO: Shutting down MCS...

 

In another CLI session, maintenance operation are still running, we passed from crunchwait to balance recently :

 

All reported states=(ONLINE), runlevels=(fullaccess), modes=(mhpu+0hpu+0hpu)
System-Status: ok
Access-Status: full

Last checkpoint: cp.20190520114138 finished Mon May 20 13:44:34 2019 after 02m 53s (OK)
Last GC: finished Mon May 20 11:31:43 2019 after 31m 28s >> recovered 164.41 GB (OK)
Last hfscheck: finished Mon May 20 13:40:35 2019 after 57m 23s >> checked 66642 of 66642 stripes (OK)

Maintenance windows scheduler capacity profile is active.
The maintenance window is currently running.
Currently running task(s): balance
Next backup window start time: Mon May 20 21:00:00 2019 CEST
Next maintenance window start time: Tue May 21 11:00:00 2019 CEST

 

If we try to run a dpnctl stop from another CLI, we get an error about a lock and we donnot want to override the warning fearing to lost consitency and have to force rollback checkpoint 🙂

 

Any idea what is going on ? Should we wait more or force restart ?

Reply
1 Solution

Accepted Solutions
Highlighted
Astreinte
1 Copper

Re: MCS donnot shut down and btw donnot restart after dpnctl cli command

Jump to solution

Problem solved :

We just had to kill Java processes !

 

Thank you for your attention, you can get back to a normal life 🙂

View solution in original post

Reply
2 Replies
Highlighted
Astreinte
1 Copper

Re: MCS donnot shut down and btw donnot restart after dpnctl cli command

Jump to solution

Problem solved :

We just had to kill Java processes !

 

Thank you for your attention, you can get back to a normal life 🙂

View solution in original post

Reply
Highlighted
MikeFromAvamar
1 Copper

Re: MCS donnot shut down and btw donnot restart after dpnctl cli command

Jump to solution

Ideally please do not kill all java on an Avamar grid. There are different services that will terminate if you run killall java so if you need to stop MCS first try:

mcserver.sh --stop

-> this will stop MCS and the MCDB (MCS database).

If the above doesn't work then try:

ps -ef --forest | grep mcserver | awk '{print $2}' | xargs -t -I {} kill {}

-> The above uses a ps -ef --forest to look for MCS.  You can search for mcserver which is what would need to stop.  In some very rare cases multiple users or bad attempts at running dpnctl start/stop mcs may have duplicate processes causing this.

Example:

root@AVAMAR_TEST_GRID:/var/log/#: ps -ef --forest | grep mcserver
admin 148508 1 1 Jun05 ? 04:03:22 /usr/java/latest//bin/java -Xmx1G -XX:+HeapDumpOnOutOfMemoryError -server -ea -cp /usr/local/avamar/lib/avamar-mcsecurity.jar:/usr/local/avamar/lib/commons-cli-1.2.jar:/usr/local/avamar/lib/commons-lang-2.6.jar:/usr/local/avamar/lib/commons-logging-adapters-1.1.jar:/usr/local/avamar/lib/rsa-jsafe-cryptojFIPS-5.0.jar:/usr/local/avamar/lib/rsa-jsafe-util-5.0.jar:/usr/local/avamar/lib/spring-jdbc-3.1.1.RELEASE.jar:/usr/local/avamar/lib/spring-tx-3.1.1.RELEASE.jar:/usr/local/avamar/lib/mcserver.jar:/usr/local/avamar/var/mc/server_data/prefs:lib:/usr/local/avamar/lib/asn_server.jar:/usr/local/avamar/lib/mail.jar:/usr/local/avamar/lib/activation.jar:/usr/local/avamar/lib/xercesImpl.jar:/usr/local/avamar/lib/xmlsec.jar:/usr/local/avamar/lib/xmldsig.jar:lib/postgresql-8.4-702.jdbc4.jar:/usr/local/avamar/lib/jaxrpc.jar:/usr/local/avamar/lib/commons-logging-1.0.4.jar:/usr/local/avamar/lib/commons-validator.jar:/usr/local/avamar/lib/commons-discovery-0.2.jar:/usr/local/avamar/lib/commons-collections-3.2.1.jar:/usr/local/avamar/lib/saaj.jar:/usr/local/avamar/lib/wbem.jar:/usr/local/avamar/lib/wsdl4j-1.5.1.jar:/usr/local/avamar/lib/log4j-1.2.14.jar:/usr/local/avamar/lib/axis.jar:/usr/local/avamar/lib/vim25.jar:/usr/local/avamar/lib/commons-codec-1.3.jar:/usr/local/avamar/lib/jema_api.jar:/usr/local/avamar/lib/jetty-6.1.23.jar:/usr/local/avamar/lib/jetty-util-6.1.23.jar:/usr/local/avamar/lib/servlet-api-3.0.jar:/usr/local/avamar/lib/jsp-api-2.1.jar:/usr/local/avamar/lib/jsp-2.1.jar:/usr/local/avamar/lib/core-3.1.1.jar:/usr/local/avamar/lib/ant-1.6.5.jar:/usr/local/avamar/lib/apputils25.jar:/usr/local/avamar/lib/j2ssh-core-0.2.9.jar:/usr/local/avamar/lib/snmp4j-1.11.jar:/usr/local/avamar/lib/mccommons.jar:/usr/local/avamar/lib/user-authentication.jar:/usr/local/avamar/lib/nis-1.2.1.jar:/usr/local/avamar/lib/providerutil-1.2.1.jar:/usr/local/avamar/lib/commons-lang-2.6.jar:/usr/local/avamar/lib/commons-io-1.4.jar:/usr/local/avamar/lib/jutf7-1.0.0.jar:/usr/local/avamar/lib/i18n:/usr/local/avamar/lib/certj.jar:/usr/local/avamar/lib/rsa-jsafe-cryptojce-6.2.jar:/usr/local/avamar/lib/rsa-jsafe-cryptojcommon-6.2.jar:/usr/local/avamar/lib/rsa-jsafe-jcmFIPS-6.2.jar:/usr/local/avamar/lib/sslj.jar:/usr/local/avamar/lib/amqp-client-3.1.3.jar:/usr/local/avamar/lib/events-dispatcher-api-1.0.0.jar:/usr/local/avamar/lib/events-dispatcher-amqp-1.0.0.jar:/usr/local/avamar/lib/jackson-core-asl-1.9.13.jar:/usr/local/avamar/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/avamar/lib/mcebms-client.jar:/usr/local/avamar/lib/httpclient-4.3.2.jar:/usr/local/avamar/lib/httpcore-4.3.1.jar:/usr/local/avamar/lib/javax.ws.rs-api-2.0-m10.jar:/usr/local/avamar/lib/rabbit-config.jar:/usr/local/avamar/lib/jsch-0.1.53.jar:/usr/local/avamar/lib/aopalliance-1.0.jar:/usr/local/avamar/lib/aspectjweaver-1.6.12.jar:/usr/local/avamar/lib/spring-aop-3.1.1.RELEASE.jar:/usr/local/avamar/lib/spring-asm-3.1.1.RELEASE.jar:/usr/local/avamar/lib/spring-beans-3.1.1.RELEASE.jar:/usr/local/avamar/lib/spring-context-3.1.1.RELEASE.jar:/usr/local/avamar/lib/spring-core-3.1.1.RELEASE.jar:/usr/local/avamar/lib/spring-expression-3.1.1.RELEASE.jar:/usr/local/avamar/lib/spring-amqp-1.2.0.RELEASE.jar:/usr/local/avamar/lib/spring-rabbit-1.2.0.RELEASE.jar:/usr/local/avamar/lib/spring-web-3.1.1.RELEASE.jar:/usr/local/avamar/lib/spring-jdbc-3.1.1.RELEASE.jar:/usr/local/avamar/lib/spring-tx-3.1.1.RELEASE.jar::/usr/local/avamar/lib/spring-security-config-3.1.0.RELEASE.jar:/usr/local/avamar/lib/spring-security-core-3.1.0.RELEASE.jar:/usr/local/avamar/lib/spring-security-crypto-3.1.0.RELEASE.jar:/usr/local/avamar/lib/spring-security-web-3.1.0.RELEASE.jar -Djava.rmi.server.hostname=hop-4s-1x3-util.svt.lab.emc.com -Djava.util.logging.config.file=/usr/local/avamar/lib/mcserver_logging.properties -Djava.security.egd=file:/dev/./urandom -Djava.io.tmpdir=/usr/local/avamar/var/mc/server_tmp -Djava.util.prefs.PreferencesFactory=com.avamar.mc.util.MCServerPreferencesFactory -Djavax.xml.parsers.DocumentBuilderFactory=org.apache.xerces.jaxp.DocumentBuilderFactoryImpl -Djavax.net.ssl.keyStore=lib/rmi_ssl_keystore -Djava
root 55163 52821 0 17:11 pts/1 00:00:00 \_ grep mcserver

Once you kill the above identified Avamar Java processess only.  Then run:

mcserver.sh --stop

-> to make sure the MCS database is stopped.

You should then be able to run the below to start MCS and the schedule service:

mcserver.sh --start --verbose

-> above is a verbose CLI method to watch MCS come up.  Please note: if you are having networking issues this may be delayed if you have multiple vCenters.  For example: if you have 5 vCenters and 3 of them are down due to networking issues (unrelated to MCS) this may have a longer startup time due to MCS attempting to sync to all vCenters upon startup.  If you do have vCenters, wait even if it shows an error upon starting and then run mcserver.sh --status to verify the status about 5-10 minutes later to confirm if it starts or is up normally.

Once up; you can then run: dpnctl start sched to bring back up the schedule service which stops when MCS stops.

 

Final note:

It's best to contact Avamar Support for these issues; however above is provided so at the very least you can avoid running the killall java or attempt to kill all active java processes in system memory.

 

If you do run killall java this may stop the ldls (Avamar downloader service), Tape Library servcies, DTLT site, AVI upgrade site, or ASCD service.    If you made a mistake and did run killall java please run: ascd start as the admin user and confirm with Avamar support for this.

 

Alternatively you could also run the Avamar Proactive Check health check script to confirm the services look okay and verify with support below

Reply