Highlighted
air_funny
1 Copper

Failed to connect to Administrator Server.Connection refused

Jump to solution

Hello fellows,

     After one day's power cut down,I cannot login EMC Avamar administrator.Like below:

1.jpg

Then,I visit romote server to search for a reason,when I restart the service following EMC document,I find it seems fail to restart Amavar service.Like below:

2.jpg

so,anyone can serve a solution for me?THX!

1 Solution

Accepted Solutions
clevya
3 Argentium

Re: Failed to connect to Administrator Server.Connection refused

Jump to solution

Hello,

As noted in the message from dpnctl the GSAN was not shutdown cleanly (probably because of the power outage). The GSAN is very much like a high transaction rate database with a lot of in memory activity. If the power is pulled from the system there may be a large number of transactions that were completed in memory that will never be committed to disk, this will leave the GSAN in an inconsistent (or unclean) state which means access to the data may be compromised.


Because of the above, whenever the GSAN is not shutdown cleanly you MUST rollback to the latest checkpoint, as indicated in your screenshot using option 4 (restart but do not rollback) is NOT RECOMMENDED and will leave the data in an inconsistent state or the system may not even start up at all.

Whenever possible you should follow Instructions To Safely Shut Down and Restart an Avamar Grid During Maintenance to shutdown a server before a power outage, the same KB can be used to restart the services once the outage is over. Following the above will avoid a rollback and any potential dataloss.

Note that the KB also indicates "If the system reports that gsan did not come down cleanly, select the option to rollback to the last checkpoint."

In your case you should have chosen option 1, when given the option to rollback. Option 4 should never be selected and I have asked for it to be removed in future versions of Avamar.

As a side note, please also remember that all "dpnctl" commands should be run as admin with the SSH keys loaded.

To resolve your problem I would run the following commands:


1. Ctrl+c on the currently running dpnctl command.

2. Load the ssh keys:

su - admin

ssh-agent bash

ssh-add /home/admin/.ssh/dpnid

3. Kill all gsan process (run this twice to make sure all processes are dead):

mapall --noerror 'killall gsan'

4. Stop backup sched (this may give an error that you can ignore):

dpnctl stop sched

5. Shutdown other services:

dpnctl stop

6. Restart and rollback:

dpnctl start --force_rollback --force_mcs_restore

###In the above make sure to chose option 1 when asked if you want to rollback or not

If there are further issues I would recommend you open a service request to get further assistance.

Regards,

Adam

4 Replies
clevya
3 Argentium

Re: Failed to connect to Administrator Server.Connection refused

Jump to solution

Hello,

As noted in the message from dpnctl the GSAN was not shutdown cleanly (probably because of the power outage). The GSAN is very much like a high transaction rate database with a lot of in memory activity. If the power is pulled from the system there may be a large number of transactions that were completed in memory that will never be committed to disk, this will leave the GSAN in an inconsistent (or unclean) state which means access to the data may be compromised.


Because of the above, whenever the GSAN is not shutdown cleanly you MUST rollback to the latest checkpoint, as indicated in your screenshot using option 4 (restart but do not rollback) is NOT RECOMMENDED and will leave the data in an inconsistent state or the system may not even start up at all.

Whenever possible you should follow Instructions To Safely Shut Down and Restart an Avamar Grid During Maintenance to shutdown a server before a power outage, the same KB can be used to restart the services once the outage is over. Following the above will avoid a rollback and any potential dataloss.

Note that the KB also indicates "If the system reports that gsan did not come down cleanly, select the option to rollback to the last checkpoint."

In your case you should have chosen option 1, when given the option to rollback. Option 4 should never be selected and I have asked for it to be removed in future versions of Avamar.

As a side note, please also remember that all "dpnctl" commands should be run as admin with the SSH keys loaded.

To resolve your problem I would run the following commands:


1. Ctrl+c on the currently running dpnctl command.

2. Load the ssh keys:

su - admin

ssh-agent bash

ssh-add /home/admin/.ssh/dpnid

3. Kill all gsan process (run this twice to make sure all processes are dead):

mapall --noerror 'killall gsan'

4. Stop backup sched (this may give an error that you can ignore):

dpnctl stop sched

5. Shutdown other services:

dpnctl stop

6. Restart and rollback:

dpnctl start --force_rollback --force_mcs_restore

###In the above make sure to chose option 1 when asked if you want to rollback or not

If there are further issues I would recommend you open a service request to get further assistance.

Regards,

Adam

air_funny
1 Copper

Re: Failed to connect to Administrator Server.Connection refused

Jump to solution

I really appeciate for your reply,I'll have a try following your method,thank you!

0 Kudos
RRR
5 Osmium

Re: Failed to connect to Administrator Server.Connection refused

Jump to solution

please mark Adam's answer as correct, so other can easily spot it.

0 Kudos
Nayaks1
2 Iron

Re: Failed to connect to Administrator Server.Connection refused

Jump to solution

As always an amazing and detailed explanation

0 Kudos