Avamar: How to run the proactive_check.pl health check script on an Avamar Server
Summary: This article aims to elucidate the functionality of the proactive health check (PHC) script and provide a comprehensive guide on its execution.
Instructions
proactive_check.pl script must be run on the Avamar grid (Single-Node or Utility Node) in the /home/admin/proactive_check directory.
1. Log in to the Avamar Utility Node as admin.
2. Check if the proactive_check directory exists:
ls -ld proactive_check
Output when the directory exists:
admin@avamar-server:~/>: ls -ld proactive_check
drwx------ 2 admin admin 282 May 27 13:18 proactive_check
Output when the directory does not exist:
admin@avamar-server:~/>: ls -ld proactive_check
ls: cannot access 'proactive_check': No such file or directory
3. If the proactive_check directory exists, go to step 4, otherwise create the directory. Ensure that the path is in /home/admin:
cd /home/admin
mkdir proactive_check
ls -ld proactive_check
Example:
admin@avamar-server:~/>: cd /home/admin
admin@avamar-server:~/>:
admin@avamar-server:~/>: mkdir proactive_check
admin@avamar-server:~/>:
admin@avamar-server:~/>: ls -ld proactive_check
drwx------ 2 admin admin 282 May 27 13:38 proactive_check
4. Go to the proactive_check directory and check if the proactive_check.pl or proactive_check.uu files exist:
cd /home/admin/proactive_check
ls -lh proactive_check*
Output when one of the files or both exist:
admin@avamar-server:~/>: cd /home/admin/proactive_check
admin@avamar-server:~/proactive_check/>: ls -lh proactive_check*
-rwxr-xr-x 1 admin admin 399K May 27 05:04 proactive_check.pl
-rwx--x--x 1 admin admin 123K Jan 19 15:56 proactive_check.uu
Output when the files do not exist:
admin@avamar-server:~/>: cd /home/admin/proactive_check
admin@avamar-server:~/proactive_check/>: ls -lh proactive_check*
ls: cannot access 'proactive_check': No such file or directory
5. If the files do not exist, go to step 6, otherwise rename the existing files:
mv proactive_check.pl x-proactive_check.pl.`date -I`
mv proactive_check.uu x-proactive_check.uu.`date -I`
Example:
admin@avamar-server:~/proactive_check/>: mv proactive_check.pl x-proactive_check.pl.`date -I`
admin@avamar-server:~/proactive_check/>: mv proactive_check.uu x-proactive_check.uu.`date -I`
6. Download the proactive_check.pl or proactive_check.uu from the Central Dell site.
proactive_check.uu" file, smaller than the full "proactive_check.pl," by pasting the content into a file on the Avamar grid, given familiarity with vi editor.
7. Still logged as admin, go to the proactive_check directory and confirm that the file was transferred correctly:
cd /home/admin/proactive_check
ls -lh proactive_check*
Output:
admin@avamar-server:~/>: cd /home/admin/proactive_check admin@avamar-server:~/proactive_check/>: ls -lh proactive_check* -rwxr-xr-x 1 admin admin 399K May 27 05:04 proactive_check.pl Or admin@avamar-server:~/>: cd /home/admin/proactive_check admin@avamar-server:~/proactive_check/>: ls -lh proactive_check* -rwx--x--x 1 admin admin 123K Jan 19 15:56 proactive_check.uu
8. If proactive_check.uu is transferred, convert the file:
perl proactive_check.uu
Example:
admin@avamar-server:~/proactive_check/>: perl proactive_check.uu
The above command creates the proactive_check.pl, which can be verified by running the following:
admin@avamar-server:~/proactive_check/>: ls -lh proactive_check.*
-rw------- 1 admin admin 123K May 27 16:13 proactive_check.uu
-rwx--x--x 1 admin admin 399K May 27 16:14 proactive_check.pl
9. If proactive_check.pl is transferred, make it executable:
chmod 755 proactive_check.pl
Before:
admin@avamar-server:~/proactive_check/>: ls -lh proactive_check.pl
-rw------- 1 admin admin 399K May 27 16:17 proactive_check.pl
After:
admin@avamar-server:~/proactive_check/>: chmod 755 proactive_check.pl
admin@avamar-server:~/proactive_check/>: ls -lh proactive_check.pl
-rwxr-xr-x 1 admin admin 399K May 27 16:17 proactive_check.pl
10. Confirm the Avamar version:
gsan --version
Example Output:
admin@avamar-server:~/>: gsan --version
version: 19.10.0-135
build date: Dec 8 2023 07:01:51
msg format: 13-10
SSL: TLSv1 OpenSSL 1.0.2p-fips 14 Aug 2018 (OwB:1.5.11.0, BSAFE Micro Edition Suite:5.0.2.1)
Zlib: 1.2.11
LZO: 1.08 Jul 12 2002
platform: Linux
OS version: SLES-64
Processor: x86_64
-
If running the script on Avamar versions 19.1 and lower, the script must be run as admin
-
If running the script on Avamar versions 19.2 or higher, the script must be run as root:
su -
cd /home/admin/proactive_check
Script Usage Times:
- Run the script as a "Proactive health check" tool with the
--logoffflag (approximately 3 to 5 minutes) - Run the script as a "Closing SR Health Check" tool (approximately 1 to 3 minutes)
Using the script as a health check tool:
Run the script without any parameters to check the overall health of Avamar:
./proactive_check.pl
Example running the script in Avamar 19.10 version. (as root):
root@avamar-server:/home/admin/proactive_check/#: ./proactive_check.pl
proactive_check.pl 8.13
Latest script version DISABLED
===== START HEALTHCHECK
Please Wait... Gathering System Info - This may take a minute or two
Avamar Hostname avamar-server.lab.com
Avamar Server Version 19.10.0-135
GSAN Version 19.10.0-135
MCS Version 19.10.0-135
avtar Version: 19.10.100-135
Hardware Manufacturer vmware
System ID 1234567890@00:AA:11:BB:22:CC
Node Type Single Node AVE .5TB
Operating System SLES12SP5-2024-R1
Avamar Hardening RPM 7.0.2-2.x86_64
Registered Media Access Nodes NONE
SWID ELMAVM123456YZ
License PASSED
AVI Installer PASSED 19.10.0.135
Datadomain dd-server Vers:7.13.0.20-1082704 S/N:AUXXXXXXXXXX7W
Datadomain dd_only_mode WARNING
ddrmaint Patches PASSED
Replication Partner Source to avamar-target.lab.com
Hardware EOSL PASSED
Remote Management PASSED
Version Supported PASSED
MC flush in past 24 hours PASSED
HFSCheck in past 36 hours PASSED
Checkpoint Status PASSED
Status.dpn PASSED
Mandatory Client Upgrades PASSED
Data Domain Version PASSED
/etc/profile PASSED
mcserver.xml PASSED
preferences.dtd permissions PASSED
MCS database size PASSED
Java symlink PASSED
Certificate Expiration PASSED
GSAN Patches PASSED
File Permissions PASSED
FIPS Status PASSED Disabled
Bonding Configuration PASSED
ascd status PASSED
Cron Running PASSED
Checkpoint Retention PASSED
Config Settings PASSED
HFSCheck overtime allowed PASSED
Swap Space PASSED
HFSCheck run time PASSED
MCS Patches INFO
Open File Settings PASSED
GSAN status PASSED
MCS status PASSED
ddrmaint service PASSED
ESE status WARNING
Backup Scheduler running PASSED
Maintenance scheduler running PASSED
Cron jobs enabled PASSED
Unattended startup PASSED
See detailed ERROR information in hc_results.txt
FINISHED
It is then possible to check for more information by reviewing the hc_results.txt file:
cat hc_results.txt
Example Output with more information:
root@avamar-server:/home/admin/proactive_check/#: cat hc_results.txt
========================================================================
Run Date: Mon May 27 14:51:11 2024 Version 8.13
========================================================================
command line:
DISCLAIMER: The results from this script are intended for the exclusive use of EMC Support & Development Engineers to diagnose potential problems so that they can use their trained skills to see exactly how the issues might or might not affect an individual server's performance
Please review KB534454 Avamar troubleshooting hierarchy if there are any issues found
# --> Latest script version DISABLED
# --> Avamar Hostname avamar-server.lab.com
# --> Avamar Server Version 19.10.0-135
# --> GSAN Version 19.10.0-135
# --> MCS Version 19.10.0-135
# --> Java processes (10)
# --> avtar Version: 19.10.100-135
# --> Hardware Manufacturer vmware
# --> System ID 1234567890@00:AA:11:BB:22:CC
# --> Node Type Single Node AVE .5TB
# --> Operating System SLES12SP5-2024-R1
# --> Avamar Hardening RPM 7.0.2-2.x86_64
# --> Registered Media Access Nodes NONE
WARNING: License has an expiration date.
# --> SWID ELMAVM123456YZ
# --> License PASSED
# --> AVI Installer PASSED 19.10.0.135
# --> Datadomain hdd-server Vers:7.13.0.20-1082704 S/N:AUXXXXXXXXXX7W
WARNING: Backups to the Data Domain are not locked. If this Avamar system is intended to be used as a pure metadata node (no backups going to Avamar), consider locking down the backups to the Data Domain to prevent accidental backups going to the Avamar system
RESOLUTION: See KB531973 and the latest admin guide for Avamar AVE Installs and the dd_only_mode flag. Please also see KB 000047627 and set maxcompdatastripe=20971520 and checkdiratomicrefs=true
# --> Datadomain dd_only_mode WARNING
# --> ddrmaint Patches PASSED
# --> Replication Partner Source to avamar-target.lab.com
# --> Hardware EOSL PASSED
# --> Remote Management PASSED
# --> Version Supported PASSED
# --> MC flush in past 24 hours PASSED
# --> HFSCheck in past 36 hours PASSED
# --> Checkpoint Status PASSED
# --> Status.dpn PASSED
# --> Mandatory Client Upgrades PASSED
# --> Data Domain Version PASSED
# --> /etc/profile PASSED
# --> mcserver.xml PASSED
# --> preferences.dtd permissions PASSED
The MCS database size is 358 MB
# --> MCS database size PASSED
# --> Java symlink PASSED
# --> Certificate Expiration PASSED
# --> GSAN Patches PASSED
# --> File Permissions PASSED
# --> FIPS Status PASSED Disabled
# --> Bonding Configuration PASSED
# --> ascd status PASSED
# --> Cron Running PASSED
# --> Checkpoint Retention PASSED
# --> Config Settings PASSED
# --> HFSCheck overtime allowed PASSED
# --> Swap Space PASSED
# --> HFSCheck run time PASSED
INFO: System is not on TLA recommended MCS hotfix 338823
RESOLUTION: See hot fix 338823 for more information
# --> MCS Patches INFO
# --> Open File Settings PASSED
# --> GSAN status PASSED
# --> MCS status PASSED
# --> ddrmaint service PASSED
WARNING: ESE status is down
RESOLUTION: Start ESE service if required
# --> ESE status WARNING
# --> Backup Scheduler running PASSED
# --> Maintenance scheduler running PASSED
# --> Cron jobs enabled PASSED
# --> Unattended startup PASSED
The script can be run multiple times. The results are appended to the hc_history.log file.
Hints and Notes:
At least once during the life cycle of a Service Request (SR), the script MUST be run with the "--logoff" flag to document the issue being investigated and tie the SR number to the server.
Using the logoff flag ensures that Technical Support Engineers (TSE) who work on the grid in the future can review the "medical history" of the grid.
- Avamar: How to use sched.sh to check historical backup, replication and maintenance activities
- Avamar: How to manage capacity with the capacity.sh script
-
replrpt(print replication info to screen likereplrpt.sh)
When the flags --sched, --capacity or --replrpt are used, the output is sent to the screen and not to a file for that individual flag. If the --hc flag is used, all listed flags are applied and the output for each flag is sent to its respective hc_*.txt file.
Run the following command with the --hc flag to create the entire set of individual hc_*.txt files:
./proactive_check.pl --hcAdditional Information
Occasionally, the script may report that the Avamar server has multiple issues.
The following article explains how various issues interact and which ones should be handled first: Avamar: How to apply the "Avamar troubleshooting hierarchy" approach correctly.
For a complete flag listing for the proactive check script, run:
./proactive_check.pl --help