STARTUP and SHUTDOWN - normal and what to do if it fails - all models

The goal of this discussion is to provide a place for everything related to STARTUP and SHUTDOWN for all models in one place.

There are a few useful discussions already which address parts of this, and I will include those links here as well.

This will be a work in progress and we welcome questions and contributions from all.

Lets get started...

SHUTDOWN:

BEFORE you shutdown:

*Make sure you either have physical access to the system or have a good iDRAC connection- so able to power-on again after shutdown. 

To check ahead of time for issues before initiating the shutdown:
1. Open a putty session to the Avamar node as 'admin' user:
2. Run these separately on AV as root and check output:
avosshutdown precheck
dpnctl shutdown --precheck

NORMAL SHUTDOWN:

The basic answer, whether a DP4400 or other IDPA model is this (order important):

1. IMPORTANT- YOU MUST DO THIS FIRST:
   1. logon to the acm dashboard as root with the IDPA  common root pw (put name or IP or acm server in browser and add :8543 (i.e. https://IDPA_name_or_IP:8543 ) 
   2. Click the button in the upper right corner to select Shutdown Appliance (This will do a proper shutdown of each of the IDPA components, such as Avamar, Data Domain, etc so no data is lost or corrupted.)
2. You should then use iDRAC for each esxi host to do a Graceful Shutdown, which will shutdown the esxi host(s) properly. (DP4400 has 1 esxi host, other models have 3 hosts).

Here are more detailed steps:

You MUST shut down the IDPA from the ACM console.
Prerequisites
● Ensure that there are no backup jobs running on the Avamar Backup Server.

If there are backup jobs running on Avamar Backup Server when the IDPA appliance shutdown operation is in progress, the shutdown operation waits for the Avamar jobs to complete with the status Waiting for shutdown of Backup Server. It is recommended that you wait for the backup jobs to complete. However, if you must shut down the appliance immediately, and then you must log in to the Avamar UI and cancel the backup jobs that are in progress.

● When shutting down IDPA, physical intervention or the use of iDRAC will be needed to restart the system. If you are remotely shutting down the IDPA appliances, ensure that either you have physical access to the system or have configured iDRAC on the system.

To shut down IDPA, perform the following actions:
Steps
1. On the ACM dashboard Home tab, click the Shutdown Appliance icon.
2. Enter the ACM root password, and click Yes.
The appliance shut down progress is displayed.
The IDPA appliance shuts down the components in the following order:
● Backup Server (Avamar)
● Search (Data Protection Search, DPS)
● Reporting and Analytics (Data Protection Advisor, DPA)
● IDPA System Manager (Data Protection Centeral, DPC)
● Cloud Disaster Recovery Agent
● Protection Storage (Data Domain)
● Appliance Configuration Manager (ACM)
● vCenter Server (VCSA)
● Compute Node (ESXI)

If the ACM-initiated shutdown fails to shut down the Backup Server and the Protection Storage or both, then the ACM displays a message listing the component(s) that failed to shut down. The ACM then continues to shut down the other components.

For DP4400, after all the point products (VMs) shut down, the ESXi needs 15 minutes to go into maintenance mode and shutdown.
If the ESXi does not shut down, you need to manually shut down the ESXi. DP4400 takes more than 30 minutes to shut down. For more information about shutting down ESXi manually , see Shut down ESX manually in the Product Guide.
If any components fail to shut down, you must manually shut down the components. For more information about manual shut down of IDPA components, see Troubleshooting shut down in the Product Guide and/or see below.

IF NORMAL SHUTDOWN FAILS:

Troubleshoot shutdown

During the shut down process, if the appliance or any of the components fail to shut down automatically, you can manually shut down the IDPA appliance and its individual components.
If the Avamar or Data Domain systems fail to shut down, aside from performing a manual shut down of these components, you must also shut down the infrastructure components, which include vCenter and ESX servers.

If the shutdown is already hung, you can putty to the acm server to check the logs:

/usr/local/dataprotection/var/configmgr/server_data/logs/server.log
You could grep for "Avamar shutdown precheck failed"

Avamar shutdown validation errors and suggested actions:

Shut down Avamar manually

This section provides you information about how to shut down the Avamar component manually in the DP5300, DP5800, and DP4400 appliances.

To manually shut down the physical Avamar (DP8xxxx only) or Avamar Virtual Edition (AVE) (DP4400/DP5xxxx) component, perform the following actions.

Login to the Avamar or  AVE server with SSH by using the Avamar/AVE IP address.
a. Create a checkpoint by running the following command.
mccli checkpoint create --override_maintenance_scheduler
b. Backup the MCS data by running the following command:
mcserver.sh --flush
c. Stop all Avamar services by running the following command.
dpnctl stop all
d. Power off the physical Avamar or if the AVE then power off from vCenter.
NOTE: If you have the DP4400 or DP5xxxx appliance, then you must shutdown ESX(s) if full HW power down is desired. For more information about manual shut down of ESX, see Shut down ESX manually in the Product Guide and/or see below.

Shut down Data Domain manually

This section provides you information about how to shut down the Data Domain component manually.

To manually shut down the Data Domain component or the DDVE, perform the following actions.
NOTE: The manual shut down for Data Domain system is applicable for the DP5xxxx and DPxxxx appliances.
NOTE: The manual shut down for DDVE is applicable for the DP4400 appliance.
Steps
1. Open a new SSH session to login to the Data Domain system or DDVE.
2. Shut down the Data Domain or DDVE system by running the following command.
system poweroff
NOTE: If you have the DP4400 appliance, then you must shutdown ESX if full power-down is desired. For more information about manual shut down of ESX, see Shut down ESX manually in the Product Guide, or see below.

Shut down VMs in vCenter manually:

This procedure is applicable for all models of IDPA. Perform the following actions:

1. Open a browser and enter the IP address to access vCenter.
The login page is displayed.
2. Enter your username and password on the vCenter login page.
3. Shut down the Data Protection virtual application (vApp).
NOTE: All virtual machines and virtual applications under Data Protection virtual application are automatically shut down in the proper order. (does NOT include vcsa and acm)
4. If needed after above then the VCSA can be shutdown by connecting to the esxi host client for the esxi hosting the VCSA VM, and Shutdown the VM.
5. If needed after above then the ACM can be shutdown by connecting to the esxi host client for the esxi hosting the ACM VM, and Shutdown the VM.

Shut down ESXI manually

This procedure to shut down ESX manually is applicable for all models of IDPA. To manually shut down the ESX, perform the following actions.

1. Login to each ESXi host.
2. Set all ESXi hosts into maintenance mode by running the following command on each host at the host console:
esxcli system maintenanceMode set -e true -m noAction
3. Shut down all ESXi host using the vSphere client or the ESXi host client.
4. If issues with above you can connect to iDRAC for each ESXI host and select Graceful Shutdown (NOT Power Cycle or Reset etc) on the dashboard.
(Only do this after proper shutdown from ACM dashboard and other steps above have been tried first.)

In the next post we will look at STARTUP......

We welcome questions and contributions, and I will post more....

If you found this useful please give KUDOS to help others find the information also.

thanks