Avamar: An outdated dpnavsys on Gen4T nodes causes warnings or errors when running the health check script
Summary: Some Gen4T nodes (predominantly in a mixed hardware environment) have an outdated dpnavsys utility, causing warnings or errors when running the health check script.
Symptoms
The health check script (proactive_check.pl) reports the following when run on an Avamar grid with Gen4T Nodes:
...
WARNING: Node (0.n) SSD Drive expected but not identified
RESOLUTION: Check the Media Type output from 'avsysreport pdisk'. Expected to see 'SSD' or 'Solid State'. Check hardware health if not seen
NOTE: This may cause the Micron SSD results to be wrong
# --> SSD Found FAILED
...
(Where 0.n is the node reporting the issue)
The SSD drive on the problematic node reports no issues:
1. Log in to the Avamar Utility Node as admin.
2. Elevate to root.
3. Load the ssh keys per Avamar: How to Log in to an Avamar Server and Load Various Keys.
4. Connect to the problematic node using the ssn command:
ssn --user=root 0.6
(In the above example, the connection is to node 0.6)
5. Once connected to the node, run the following command to verify that the SSD is not reporting any issues:
arcconf getconfig 1 pd
The output should be similar to the following:
Device #12
Device is a Hard drive
State : Raw (Pass Through)
Block Size : 512 Bytes
Supported : Yes
Programmed Max Speed : SAS 12.0 Gb/s
Transfer Speed : SAS 12.0 Gb/s
Reported Channel,Device(T:L) : 0,20(20:0)
Reported Location : Enclosure 0, Slot 12(Connector 0, Connector 1)
Reported ESD(T:L) : 2,0(0:0)
Vendor : HITACHI
Model : HUSMM112 CLAR200
Firmware : C29C
Serial number : 0PY4R85A
World-wide name : 5000CCA04DB1AE4B
Reserved Size : 0 KB
Total Size : 190782 MB
Write Cache : Disabled (write-through)
FRU : None
S.M.A.R.T. : No
S.M.A.R.T. warnings : 0
Power State : Full rpm
Supported Power States : Full power,Powered off
SSD : Yes
Temperature : 30 C/ 86 F
Cause
The node reporting the warning has an outdated version of dpnavsys that does not support Gen4T nodes.
This can be confirmed by running the following command (with keys loaded per Avamar: How to Log in to an Avamar Server and Load Various Keys):
mapall --all+ --noerror 'rpm -qa | grep dpnavsys'
Example output:
(0.s) ssh -q -x -o GSSAPIAuthentication=no admin@192.168.255.1 'rpm -qa'
dpnavsys-1.1.0-24
(0.0) ssh -q -x -o GSSAPIAuthentication=no admin@192.168.255.2 'rpm -qa'
dpnavsys-1.1.0-24
(0.1) ssh -q -x -o GSSAPIAuthentication=no admin@192.168.255.3 'rpm -qa'
dpnavsys-1.1.0-24
(0.2) ssh -q -x -o GSSAPIAuthentication=no admin@192.168.255.4 'rpm -qa'
dpnavsys-1.1.0-24
(0.3) ssh -q -x -o GSSAPIAuthentication=no admin@192.168.255.5 'rpm -qa'
dpnavsys-1.1.0-24
(0.4) ssh -q -x -o GSSAPIAuthentication=no admin@192.168.255.6 'rpm -qa'
dpnavsys-1.1.0-24
(0.5) ssh -q -x -o GSSAPIAuthentication=no admin@192.168.255.7 'rpm -qa'
dpnavsys-1.1.0-24
(0.6) ssh -q -x -o GSSAPIAuthentication=no admin@192.168.255.8 'rpm -qa'
dpnavsys-1.1.0-17
(0.7) ssh -q -x -o GSSAPIAuthentication=no admin@192.168.255.9 'rpm -qa'
dpnavsys-1.1.0-24
All nodes should be running the same version of dpnavsys.
From the output above, node 0.6 is not running the same version of dpnavsys.
Resolution
1. Log in to the Avamar Utility Node as admin.
2. Elevate to root.
3. Load the keys per Avamar: How to Log in to an Avamar Server and Load Various Keys):
4. If unknown, verify which node, or nodes, have the outdated copy of dpnavsys:
mapall --all+ --noerror 'rpm -qa | grep dpnavsys'
Example output:
(0.s) ssh -q -x -o GSSAPIAuthentication=no admin@192.168.255.1 'rpm -qa'
dpnavsys-1.1.0-24
(0.0) ssh -q -x -o GSSAPIAuthentication=no admin@192.168.255.2 'rpm -qa'
dpnavsys-1.1.0-24
(0.1) ssh -q -x -o GSSAPIAuthentication=no admin@192.168.255.3 'rpm -qa'
dpnavsys-1.1.0-24
(0.2) ssh -q -x -o GSSAPIAuthentication=no admin@192.168.255.4 'rpm -qa'
dpnavsys-1.1.0-24
(0.3) ssh -q -x -o GSSAPIAuthentication=no admin@192.168.255.5 'rpm -qa'
dpnavsys-1.1.0-24
(0.4) ssh -q -x -o GSSAPIAuthentication=no admin@192.168.255.6 'rpm -qa'
dpnavsys-1.1.0-24
(0.5) ssh -q -x -o GSSAPIAuthentication=no admin@192.168.255.7 'rpm -qa'
dpnavsys-1.1.0-24
(0.6) ssh -q -x -o GSSAPIAuthentication=no admin@192.168.255.8 'rpm -qa'
dpnavsys-1.1.0-17
(0.7) ssh -q -x -o GSSAPIAuthentication=no admin@192.168.255.9 'rpm -qa'
dpnavsys-1.1.0-24
5. Copy the correct version of the package to the problematic node or nodes:
a. Run the following command to determine which node (or nodes) has the updated copy of dpnavsys:
mapall --all+ --noerror 'ls -al /usr/local/avamar/src/dpnavsys*'
The output is similar to the following:
Using /usr/local/avamar/var/probe.xml
(0.s) ssh -q -x -o GSSAPIAuthentication=no admin@192.168.255.1 'ls -al /usr/local/avamar/src/dpnavsys*'
ls: cannot access '/usr/local/avamar/src/dpnavsys*': No such file or directory
(0.0) ssh -q -x -o GSSAPIAuthentication=no admin@192.168.255.2 'ls -al /usr/local/avamar/src/dpnavsys*'
-rw-r----- 1 root root 576658 Dec 8 2018 /usr/local/avamar/src/dpnavsys-1.1.0-17.x86_64.rpm
-rw-r----- 1 root root 579210 Oct 4 2019 /usr/local/avamar/src/dpnavsys-1.1.0-24.x86_64.rpm
(0.1) ssh -q -x -o GSSAPIAuthentication=no admin@192.168.255.3 'ls -al /usr/local/avamar/src/dpnavsys*'
-rw-r----- 1 root root 576658 Dec 8 2018 /usr/local/avamar/src/dpnavsys-1.1.0-17.x86_64.rpm
-rw-r----- 1 root root 579210 Oct 4 2019 /usr/local/avamar/src/dpnavsys-1.1.0-24.x86_64.rpm
(0.2) ssh -q -x -o GSSAPIAuthentication=no admin@192.168.255.4 'ls -al /usr/local/avamar/src/dpnavsys*'
-rw-r----- 1 root root 576658 Dec 8 2018 /usr/local/avamar/src/dpnavsys-1.1.0-17.x86_64.rpm
-rw-r----- 1 root root 579210 Oct 4 2019 /usr/local/avamar/src/dpnavsys-1.1.0-24.x86_64.rpm
(0.3) ssh -q -x -o GSSAPIAuthentication=no admin@192.168.255.5 'ls -al /usr/local/avamar/src/dpnavsys*'
-rw-r----- 1 root root 576658 Dec 8 2018 /usr/local/avamar/src/dpnavsys-1.1.0-17.x86_64.rpm
-rw-r----- 1 root root 579210 Oct 4 2019 /usr/local/avamar/src/dpnavsys-1.1.0-24.x86_64.rpm
(0.4) ssh -q -x -o GSSAPIAuthentication=no admin@192.168.255.6 'ls -al /usr/local/avamar/src/dpnavsys*'
-rw-r----- 1 root root 576658 Dec 8 2018 /usr/local/avamar/src/dpnavsys-1.1.0-17.x86_64.rpm
-rw-r----- 1 root root 579210 Oct 4 2019 /usr/local/avamar/src/dpnavsys-1.1.0-24.x86_64.rpm
(0.5) ssh -q -x -o GSSAPIAuthentication=no admin@192.168.255.7 'ls -al /usr/local/avamar/src/dpnavsys*'
-rw-r----- 1 root root 576658 Dec 8 2018 /usr/local/avamar/src/dpnavsys-1.1.0-17.x86_64.rpm
-rw-r----- 1 root root 579210 Oct 4 2019 /usr/local/avamar/src/dpnavsys-1.1.0-24.x86_64.rpm
(0.6) ssh -q -x -o GSSAPIAuthentication=no admin@192.168.255.8 'ls -al /usr/local/avamar/src/dpnavsys*'
-rw-r----- 1 root root 576658 Dec 8 2018 /usr/local/avamar/src/dpnavsys-1.1.0-17.x86_64.rpm
(0.7) ssh -q -x -o GSSAPIAuthentication=no admin@192.168.255.9 'ls -al /usr/local/avamar/src/dpnavsys*'
-rw-r----- 1 root root 576658 Dec 8 2018 /usr/local/avamar/src/dpnavsys-1.1.0-17.x86_64.rpm
-rw-r----- 1 root root 579210 Oct 4 2019 /usr/local/avamar/src/dpnavsys-1.1.0-24.x86_64.rpm
b. Record the IP address of a node with the required version (this would be 192.168.255.2, 192.168.255.3, 192.168.255.4, 192.168.255.5, 192.168.255.6, 192.168.255.7, or 192.168.255.9 based on the output above).
dpnavsys version is not present on any nodes, download it from the Dell Support Website to the Avamar Utility Nodes /tmp directory and continue from step d below.
c. Copy dpnavsys from the IP address noted in (b) above, to the /tmp directory on the Utility Node:
cd /tmp
scp root@<Ip-address-from-b-above>:/usr/local/avamar/src/dpnavsys-<version>.rpm .
Example:
scp root@192.168.255.5:/usr/local/avamar/src/dpnavsys-1.1.0-24_x86_64.rpm .
d. Copy the updated dpnavsys package from the Utility Node to the problematic node with the outdated dpnavsys:
scp dpnavsys-<version>.rpm root@<problematic-node>:/usr/local/avamar/src/
Example:
scp dpnavsys-1.1.0-24_x86_64.rpm root@192.168.255.3:/usr/local/avamar/src/
6. Connect to the problematic node with the outdated dpnavsys as root:
ssn --user=root 0.6
7. Update the software:
rpm -U /usr/local/avamar/src/dpnavsys-<version>.rpm
Example:
rpm -U /usr/local/avamar/src/dpnavsys-1.1.0-24_x86_64.rpm
8. Perform validation checks:
a. Verify that the dpnavsys version is now correct:
rpm -qa | grep dpnavsys
b. Verify that the avsysreport command runs:
avsysreport pdisk
9. Type "exit" to return to the utility node.
10. Perform steps 5-9 on any other nodes that must be updated.
11. Once all nodes have been updated, rerun the health check script. This should complete without reporting the error.