Demo/Lab Upgrade Issue

Question

Hello All,

This is a Demo/Lab System that I am working on. I am upgrading from 4.1 to 5.0. I had an issue with the upgrade failing at one point which was resolved by upgrading firmware on a few of the drives. I have renamed probe.xml.1 and probe.out.1 back to their original names and have tried to restart the upgrade. I have also had to relax some of the settings in the upgrade.conf file.

Now when I run the upgrade I get this error message and can't figure it out. The health check comes back with all systems ready. Has anyone seen this error before or know how to resolve it. It is driving me crazy and I can't seem to find any help on the issue on Primus, KB, or forums.

If I run the dnctl status it shows everything is ok but I do find a few anomolies in the upgrade. log file

admin@avamar-utility:/usr/local/avamar/var/log/>: dpnctl status

Identity added: /home/admin/.ssh/dpnid (/home/admin/.ssh/dpnid)

dpnctl: INFO: gsan status: ready

dpnctl: INFO: MCS status: up.

dpnctl: INFO: EMS status: up.

dpnctl: INFO: Backup scheduler status: up.

dpnctl: INFO: dtlt status: up.

dpnctl: INFO: Maintenance windows scheduler status: enabled.

dpnctl: INFO: Maintenance cron jobs status: enabled.

dpnctl: INFO: Unattended startup status: disabled.

admin@avamar-utility:/usr/local/avamar/var/log/>:

==============================================
==== Health Check Summary ====
==============================================
Logfile: /usr/local/avamar/var/health_check.log
Total Number of Errors: 2

Errors to be reviewed:
ERROR: There are currently no hfschecked checkpoints!
ERROR: Garbage collecting less than 10GB per pass! (averaging 0.00 GB)
2011/01/19-23:31:56 - - - - - - END
2011/01/19-23:31:56 avupswaux: INFO: using configured relax_gc_complaints setting (1)
2011/01/19-23:31:56 avupswaux: INFO: using configured relax_disk_usage_complaints setting (1)
2011/01/19-23:31:56 avupswaux: WARNING: Avamar server health check failed
2011/01/19-23:31:56 avupswaux: INFO: checking scheduler status
2011/01/19-23:31:56 avupswaux: INFO: component "sched" status is empty
2011/01/19-23:31:56 avupswaux: INFO: backup scheduler is already stopped
2011/01/19-23:31:56 avupswaux: INFO: component "gsan" status is empty
2011/01/19-23:31:56 avupswaux: INFO: using default terminate_active_backup_sessions_pre_upgrade setting (1)
2011/01/19-23:31:56 avupswaux: INFO: terminating active backup sessions
2011/01/19-23:31:56 avupswaux: INFO: terminating sessions
2011/01/19-23:31:56 contents of temporary script "/tmp/avupswaux-run-command-26539":

Output of the end of the upgrade.log

2011/01/19-23:31:57 - - - - - - BEGIN

ERROR: no database type or no node resource database could be found with search list /usr/local/avamar/var/probe.xml, stopped at

/usr/local/avamar/bin/../lib/dpnutils/node_util.pm line 748.

2011/01/19-23:31:57 - - - - - - END

2011/01/19-23:31:57 avupswaux: INFO: exit status of "/tmp/avupswaux-run-command-26539" script execution: 2

2011/01/19-23:31:57 avupswaux: ERROR: a problem occurred while stopping maintenance cron jobs

*** See log file for additional details: "/usr/local/avamar/var/log/upgrade.log"

2011/01/19-23:31:57 rm -f /tmp/avupswaux.26539.tgz

2011/01/19-23:31:57 ======= avupswaux 5.0.0-38, running as root, ENDING at 2011-01-19 16:31:57 MST =======

2011/01/19-23:31:57 avupswaux: INFO: applying "rm -rf '/tmp/avupswaux-tmp-26539-probedir' >/tmp/avupswaux-command-output.26539 2>&1"

2011/01/19-23:31:57 [user "root"] program (pid 26539) exit status = 1 (error)

Any point in the correct direction would be awesome!

Thanks in advance!

Jerry

rpervan · Answer

Hello, > avupswaux: ERROR: a problem occurred while stopping maintenance cron jobsHave you maybe tried to stop maintenance cron jobs manually and then re-run the upgrade procedure ....  ? Rej

jerry.reddick · Answer

Rej,

<<<<<<<<<<<

Jerry,

please check the AVAMAR 5.0 • SYSTEM UPGRADE GUIDE on page 95.

<< snip >>

The avupswaux program appears to be hung while waiting for the install_os_security_errata program to complete.

regards,

Rej>>>>>>>>>>>>>>

Will do.

Thanks!

jerry.reddick · Answer

Rej,

These are the only options with the avupsw command.

root@avamar-utility:/usr/local/avamar/src/#: avupsw --help
avupsw - orchestrate Avamar server software upgrade
Version: 5.0.0-38
Usage: avupsw [--(no)connectemc] [--(no)force_version[=N]] [--help]
[--help_milestones] [--hooks=PATH] [--lockbox_password=PASSWORD]

I am going to clear the upgrade.log and run it again and capture the output and attach as a file. I have a plane in about 30 minutes so may not answer until tomorrow. I do appreciate the help.

jerry.reddick · Answer

Ok, See below for some output. Sorry it is long. No way to attach files.

Here is the avupswaux output from when the avupsw command is run.

"
avupswaux: INFO: = = = = = 2011/01/24-14:37:19 MST (UTC -0700) START: avupswaux 5.0.0-38 = = = = = =
avupswaux: INFO: logging to "/usr/local/avamar/var/log/upgrade.log"
avupswaux: WARNING: renamed existing default XML node resource database file "/usr/local/avamar/var/probe.xml" to "/usr/local/avamar/var/probe.xml.1"
avupswaux: INFO: checking node connectivity and authorizations
avupswaux: INFO: checking for OS version uniformity on node set "all"
avupswaux: INFO: checking storage status
avupswaux: INFO: checking node types
avupswaux: INFO: checking that required Avamar server components are all up/enabled
avupswaux: ERROR: unexpected empty output received from "dpnctl status --xml"
avupswaux: WARNING: required Avamar server components are not all up/enabled at this time
avupswaux: ERROR: unexpected empty output received from "dpnctl status --xml"
avupswaux: INFO: expect Avamar health check to fail because Avamar subsystems are not up
avupswaux: INFO: checking Avamar server health
avupswaux: WARNING: Avamar server health check failed
avupswaux: INFO: checking scheduler status
avupswaux: INFO: terminating active backup sessions
avupswaux: INFO: skipping checkpoint creation, per configuration setting
avupswaux: INFO: skipping checkpoint validation, per configuration setting
avupswaux: INFO: skipping replication, per configuration setting
avupswaux: INFO: inhibiting maintenance cron jobs
avupswaux: ERROR: a problem occurred while stopping maintenance cron jobs
*** See log file for additional details: "/usr/local/avamar/var/log/upgrade.log"

Attached is the relaxed upgrade.conf file. The only error I am getting when I run the health_check.pl is about Garbage Collection. I also ran mccli checkpoint show and there is a validated checkpoint.

root@avamar-utility:/usr/local/avamar/var/log/#: mccli checkpoint show

0,23000,CLI command completed successfully.

Tag Time Validated

----------------- ----------------------- ---------

cp.20110124024444 2011-01-23 19:44:44 MST

cp.20110124170017 2011-01-24 10:00:17 MST Validated

root@avamar-utility:/usr/local/avamar/var/log/#:

Here is the latestes dpnctl status after renaming probe.xml.1 back to probe.xml

root@avamar-utility:/usr/local/avamar/var/#: dpnctl status

Identity added: /home/dpn/.ssh/dpnid (/home/dpn/.ssh/dpnid)

dpnctl: INFO: gsan status: ready

dpnctl: INFO: MCS status: up.

dpnctl: INFO: EMS status: up.

dpnctl: INFO: Backup scheduler status: up.

dpnctl: INFO: dtlt status: up.

dpnctl: INFO: Maintenance windows scheduler status: enabled.

dpnctl: INFO: Maintenance cron jobs status: enabled.

dpnctl: INFO: Unattended startup status: disabled.

root@avamar-utility:/usr/local/avamar/var/#:

root@avamar-utility:/usr/local/avamar/var/log/#: cat upgrade.log

2011/01/24-21:37:19 avupswaux: INFO: = = = = = 2011/01/24-14:37:19 MST (UTC -0700) START: avupswaux 5.0.0-38 = = = = = =

2011/01/24-21:37:19 avupswaux: INFO: logging to "/usr/local/avamar/var/log/upgrade.log"

2011/01/24-21:37:19 avupswaux: INFO: log file time stamps are in UTC; local time is 2011/01/24-14:37:19 MST (UTC -0700)

2011/01/24-21:37:19 avupswaux: INFO: no command line arguments were supplied to avupswaux

2011/01/24-21:37:19 avupswaux: INFO: found required default legacy node resource database file "/usr/local/avamar/var/probe.out"

2011/01/24-21:37:19 avupswaux: INFO: using default convert_probe setting (1)

2011/01/24-21:37:19 avupswaux: INFO: applying "mv "/usr/local/avamar/var/probe.xml" "/usr/local/avamar/var/probe.xml.1"

>/tmp/avupswaux-command-output.12707 2>&1"

2011/01/24-21:37:19 avupswaux: WARNING: renamed existing default XML node resource database file "/usr/local/avamar/var/probe.xml" to

"/usr/local/avamar/var/probe.xml.1"

2011/01/24-21:37:19 avupswaux: INFO: found local system time zone "America/Phoenix"

2011/01/24-21:37:20 avupswaux: INFO: selected subset of Avamar packages seen to be currently installed:

connectemc-2.6.1.2-2

download-AvamarBackupSystemState-windows-x86-5.0.106-28

download-AvamarBackupSystemState-windows-x86_64-5.0.106-28

download-AvamarClient-aix5-ppc-5.0.106-28

download-AvamarClient-debian4.0-x86-5.0.106-28

download-AvamarClient-debian4.0-x86_64-5.0.106-28

download-AvamarClient-freebsd6-x86-5.0.106-28

download-AvamarClient-freebsd6-x86_64-5.0.106-28

download-AvamarClient-hpux11-pa_risc-5.0.106-28

download-AvamarClient-hpux11iv2-itan-5.0.106-28

download-AvamarClient-linux-centos_32-x86-5.0.106-28

download-AvamarClient-linux-centos_64-5.0.106-28

download-AvamarClient-linux-el5-s390x-5.0.106-28

download-AvamarClient-linux-oulinux5_32-x86-5.0.106-28

download-AvamarClient-linux-oulinux5_64-5.0.106-28

download-AvamarClient-linux-rh9-x86-5.0.106-28

download-AvamarClient-linux-rhel3-x86-5.0.106-28

download-AvamarClient-linux-rhel3-x86_64-5.0.106-28

download-AvamarClient-linux-rhel4-x86-5.0.106-28

download-AvamarClient-linux-rhel4-x86_64-5.0.106-28

download-AvamarClient-linux-rhel5-x86-5.0.106-28

download-AvamarClient-linux-rhel5-x86_64-5.0.106-28

download-AvamarClient-linux-sles10-x86-5.0.106-28

download-AvamarClient-linux-sles10-x86_64-5.0.106-28

download-AvamarClient-linux-sles11-x86_64-5.0.106-28

download-AvamarClient-linux-sles8-x86-5.0.106-28

download-AvamarClient-linux-sles8-x86_64-5.0.106-28

download-AvamarClient-linux-sles9-x86-5.0.106-28

download-AvamarClient-linux-sles9-x86_64-5.0.106-28

download-AvamarClient-linux-suse10-s390x-5.0.106-28

download-AvamarClient-linux-vmware_esx3-x86-5.0.106-28

download-AvamarClient-macosx-uni-5.0.106-28

download-AvamarClient-netware6-x86-5.0.106-28

download-AvamarClient-sco5.0.5-i386-5.0.106-28

download-AvamarClient-solaris10-sparc-5.0.106-28

download-AvamarClient-solaris10-x86-5.0.106-28

download-AvamarClient-solaris10-x86_64-5.0.106-28

download-AvamarClient-solaris8-sparc-5.0.106-28

download-AvamarClient-solaris9-sparc-5.0.106-28

download-AvamarClient-unixware7.1.1-i386-5.0.106-28

download-AvamarClient-windows-x86-5.0.106-28

download-AvamarClient-windows-x86_64-5.0.106-28

download-AvamarClientCluster-windows-x86-4.1.106-27

download-AvamarClientCluster-windows-x86_64-4.1.106-27

download-AvamarClusterClient-solaris10-sparc-5.0.106-28

download-AvamarClusterClient-solaris10-x86-5.0.106-28

download-AvamarClusterClient-solaris10-x86_64-5.0.106-28

download-AvamarClusterClient-solaris8-sparc-5.0.106-28

download-AvamarDB2-aix5-ppc-5.0.106-28

download-AvamarDB2-aix5-ppc64-5.0.106-28

download-AvamarDB2-linux-rhel4-x86_64-5.0.106-28

download-AvamarDB2-linux-rhel5-x86_64-5.0.106-28

download-AvamarDB2-windows-x86_64-5.0.106-28

download-AvamarExchange-windows-x86-4.1.106-27

download-AvamarExchange2003-windows-x86-5.0.106-28

download-AvamarExchange2007-windows-x86_64-5.0.106-28

download-AvamarExchangeVSS-windows-x86_64-5.0.106-28

download-AvamarLotus-aix5-ppc-5.0.106-28

download-AvamarLotus-windows-x86-5.0.106-28

download-AvamarLotus-windows-x86_64-5.0.106-28

download-AvamarMoss2007-windows-x86-5.0.106-28

download-AvamarMoss2007-windows-x86_64-5.0.106-28

download-AvamarNDMP-linux-rhel4-x86_64-5.0.106-28

download-AvamarRMAN-aix5-ppc-5.0.106-28

download-AvamarRMAN-hpux11-pa_risc-4.1.106-27

download-AvamarRMAN-linux-rhel3-x86-5.0.106-28

download-AvamarRMAN-linux-rhel3-x86_64-5.0.106-28

download-AvamarRMAN-linux-rhel4-x86-5.0.106-28

download-AvamarRMAN-linux-rhel4-x86_64-5.0.106-28

download-AvamarRMAN-linux-rhel5-x86-5.0.106-28

download-AvamarRMAN-linux-rhel5-x86_64-5.0.106-28

download-AvamarRMAN-linux-sles10-x86-5.0.106-28

download-AvamarRMAN-linux-sles10-x86_64-5.0.106-28

download-AvamarRMAN-linux-sles9-x86-5.0.106-28

download-AvamarRMAN-linux-sles9-x86_64-5.0.106-28

download-AvamarRMAN-solaris10-sparc-5.0.106-28

download-AvamarRMAN-solaris10-x86-5.0.106-28

download-AvamarRMAN-solaris10-x86_64-5.0.106-28

download-AvamarRMAN-solaris8-sparc-5.0.106-28

download-AvamarRMAN-solaris9-sparc-5.0.106-28

download-AvamarRMAN-windows-x86-5.0.106-28

download-AvamarRMAN-windows-x86_64-5.0.106-28

download-AvamarSQL-windows-x86-5.0.106-28

download-AvamarSQL-windows-x86_64-5.0.106-28

download-AvamarVmImage-linux-x86-iso-5.0.106-28

download-AvamarVmImage-windows-x86-5.0.106-28

download-AvamarVmImageProxy-linux-ova-5.0.106-28

download-AvamarWindowsSystemRecover_WINDRW32.iso-5.0.106-28

download-AvamarWindowsSystemRecover_WINDRW64.iso-5.0.106-28

dpnavclient-5.0.106-28

dpnavqinstall-5.0.0-38

dpnavupsw-5.0.0-38

dpnlmauth-5.0.3-29

dpnloginmgr-5.0.3-29

dpnmcs-5.0.3-29

dpnnetutil-5.0.0-32

dpnresite-1.0-13

dpnserver-5.0.3-29

dpnutils-5.0.3-29

dpnwebdoc-5.0.3-29

dpnwebdoc-AvAdministration-5.0.106-28

dpnwebdoc-AvClients-5.0.106-28

dpnwebdoc-AvDB2-5.0.106-28

dpnwebdoc-AvExchange-5.0.106-28

dpnwebdoc-AvExchangeVSS-5.0.106-28

dpnwebdoc-AvLotusNotes-5.0.106-28

dpnwebdoc-AvNDMP-5.0.106-28

dpnwebdoc-AvOracle-5.0.106-28

dpnwebdoc-AvReleaseNotes-5.0.106-28

dpnwebdoc-AvSQLServer-5.0.106-28

dpnwebdoc-AvSharePoint-5.0.106-28

dpnwebdoc-AvWindowsServer-5.0.106-28

dpnwebdownloads-5.0.3-29

dpnwebsite-5.0.3-29

jre-1.6.0_12-fcs

perl-DBD-Pg-1.49-0

perl-DBI-1.53-0

postgresql-8.2.3-1PGDG

postgresql-devel-8.2.3-1PGDG

postgresql-libs-8.2.3-1PGDG

postgresql-server-8.2.3-1PGDG

samba-3.0.33-0.18.el4_8

samba-client-3.0.33-0.18.el4_8

samba-common-3.0.33-0.18.el4_8

samba-swat-3.0.33-0.18.el4_8

zip-3.0-0

2011/01/24-21:37:20 avupswaux: INFO: selected subset of Avamar packages NOT seen to be currently installed (acceptable in some cases):

dpnaxionfs

dpnhotfix

dpnmccli

dpnmetadata

dpnugprep

tzupdater

2011/01/24-21:37:20 avupswaux: DEBUG: rsw_generic_text_file::exists(): file "/usr/local/avamar/var/probe.out" exists

2011/01/24-21:37:20 avupswaux: DEBUG: rsw_generic_text_file::is_readable(): file "/usr/local/avamar/var/probe.out" is readable

2011/01/24-21:37:20 avupswaux: DEBUG: rsw_generic_text_file::exists(): file "/usr/local/avamar/var/probe.out" exists

2011/01/24-21:37:20 avupswaux: DEBUG: read probe file line "NONAT"

2011/01/24-21:37:20 avupswaux: DEBUG: saw required 'NONAT' line

2011/01/24-21:37:20 avupswaux: DEBUG: read probe file line "avamar-

utility:192.168.10.150,192.168.10.150:192.168.10.151,192.168.10.153,192.168.10.154,192.168.10.155,192.168.10.156"

2011/01/24-21:37:20 avupswaux: DEBUG: probe file: module id = "avamar-utility", utility nodes = "192.168.10.150,192.168.10.150",

storage nodes = "192.168.10.151,192.168.10.153,192.168.10.154,192.168.10.155,192.168.10.156"

2011/01/24-21:37:20 avupswaux: INFO: no accelerator node IP address list configured

2011/01/24-21:37:20 avupswaux: INFO: no access node IP address list configured

2011/01/24-21:37:20 avupswaux: INFO: no spare node IP address list configured

2011/01/24-21:37:20 avupswaux: INFO: no optional node addresses found for addition to temporary node resource list

2011/01/24-21:37:20 avupswaux: INFO: applying "mkdir /tmp/avupswaux-tmp-12707-probedir >/tmp/avupswaux-command-output.12707 2>&1"

2011/01/24-21:37:20 avupswaux: DEBUG: writing to /tmp/avupswaux-tmp-12707-probedir/probe.out:

2011/01/24-21:37:20 avupswaux: DEBUG: rsw_generic_text_file::exists(): file "/tmp/avupswaux-tmp-12707-probedir/probe.out" does not

exist

2011/01/24-21:37:20 /bin/mv '/tmp/avupswaux-tmp-12707-probedir/probe.out.avupswaux_new' '/tmp/avupswaux-tmp-12707-probedir/probe.out'

2011/01/24-21:37:20 avupswaux: DEBUG: new contents of "/tmp/avupswaux-tmp-12707-probedir/probe.out":

2011/01/24-21:37:20 avupswaux: DEBUG: - - - - - - BEGIN

NONAT

avamar-utility:192.168.10.150,192.168.10.150:192.168.10.151,192.168.10.153,192.168.10.154,192.168.10.155,192.168.10.156

2011/01/24-21:37:20 avupswaux: DEBUG: - - - - - - END

2011/01/24-21:37:20 avupswaux: INFO: found conversion program /usr/local/avamar/bin/convert-probe - creating temporary probe.xml file

2011/01/24-21:37:20 avupswaux: INFO: applying "/usr/local/avamar/bin/convert-probe --in="/tmp/avupswaux-tmp-12707-probedir/probe.out"

--out="/tmp/avupswaux-tmp-12707-probedir/probe.xml" >/tmp/avupswaux-command-output.12707 2>&1"

2011/01/24-21:37:20 avupswaux: INFO: checking node connectivity and authorizations

2011/01/24-21:37:20 avupswaux: INFO: using default per_node_ssh_connectivity_test_timeout setting (180)

2011/01/24-21:37:20 avupswaux: INFO: applying "/usr/local/avamar/lib/avupsw/avtest-auth --probedir='/tmp/avupswaux-tmp-12707-probedir'

--timeout=180 --key='~admin/.ssh/dpnid' --verbose >/tmp/avupswaux-command-output.12707 2>&1"

2011/01/24-21:37:20 - - - - - - - - - - - - BEGIN output of "/usr/local/avamar/lib/avupsw/avtest-auth --probedir='/tmp/avupswaux-tmp-

12707-probedir' --timeout=180 --key='~admin/.ssh/dpnid' --verbose":

Using /tmp/avupswaux-tmp-12707-probedir/probe.xml

avtest-auth: INFO: output of connectivity and authorization check:

- - - - - - - - - - BEGIN

Identity added: /home/admin/.ssh/dpnid (/home/admin/.ssh/dpnid)

Using /tmp/avupswaux-tmp-12707-probedir/probe.xml

avtest-auth: INFO: testing ssh authorizations for user "admin" using passphrase-less private key "~admin/.ssh/dpnid"

avtest-auth: INFO: testing ssh authorizations for user "root" using passphrase-less private key "~admin/.ssh/dpnid"

avtest-auth: INFO: [user "root"] program exit status = 0 (normal)

- - - - - - - - - - END

avtest-auth: INFO: ssh connectivity and authorization test succeeded

avtest-auth: INFO: [user "root"] program exit status = 0 (normal)

2011/01/24-21:37:20 - - - - - - - - - - - - END output of "/usr/local/avamar/lib/avupsw/avtest-auth --probedir='/tmp/avupswaux-tmp-

12707-probedir' --timeout=180 --key='~admin/.ssh/dpnid' --verbose":

2011/01/24-21:37:20 avupswaux: INFO: applying "/usr/local/avamar/lib/avupsw/avtest-auth --probedir='/tmp/avupswaux-tmp-12707-probedir'

--timeout=180 --key='~dpn/.ssh/dpnid' --verbose >/tmp/avupswaux-command-output.12707 2>&1"

2011/01/24-21:37:21 - - - - - - - - - - - - BEGIN output of "/usr/local/avamar/lib/avupsw/avtest-auth --probedir='/tmp/avupswaux-tmp-

12707-probedir' --timeout=180 --key='~dpn/.ssh/dpnid' --verbose":

Using /tmp/avupswaux-tmp-12707-probedir/probe.xml

avtest-auth: INFO: output of connectivity and authorization check:

- - - - - - - - - - BEGIN

Identity added: /home/dpn/.ssh/dpnid (/home/dpn/.ssh/dpnid)

Using /tmp/avupswaux-tmp-12707-probedir/probe.xml

avtest-auth: INFO: testing ssh authorizations for user "admin" using passphrase-less private key "~dpn/.ssh/dpnid"

avtest-auth: INFO: testing ssh authorizations for user "root" using passphrase-less private key "~dpn/.ssh/dpnid"

avtest-auth: INFO: [user "root"] program exit status = 0 (normal)

- - - - - - - - - - END

avtest-auth: INFO: ssh connectivity and authorization test succeeded

avtest-auth: INFO: [user "root"] program exit status = 0 (normal)

2011/01/24-21:37:21 - - - - - - - - - - - - END output of "/usr/local/avamar/lib/avupsw/avtest-auth --probedir='/tmp/avupswaux-tmp-

12707-probedir' --timeout=180 --key='~dpn/.ssh/dpnid' --verbose":

2011/01/24-21:37:21 avupswaux: INFO: checking for OS version uniformity on node set "all"

2011/01/24-21:37:21 avupswaux: INFO: using default relax_os_version_check setting (0)

2011/01/24-21:37:21 contents of temporary script "/tmp/avupswaux-mapall-12707":

2011/01/24-21:37:21 - - - - - - BEGIN

#!/bin/bash

ssh-add /home/dpn/.ssh/dpnid

[ -r /etc/profile ] && . /etc/profile

(export SYSPROBEUSER=root SYSPROBEDIR=/tmp/avupswaux-tmp-12707-probedir ; mapall --nodes=all --noerror 'cat /etc/redhat-release')

/tmp/avupswaux-mapall-apply.12707 2>&1

2011/01/24-21:37:21 - - - - - - END

2011/01/24-21:37:21 chmod 700 /tmp/avupswaux-mapall-12707

2011/01/24-21:37:21 ssh-agent bash -c /tmp/avupswaux-mapall-12707 /tmp/avupswaux-ssh-agent-mapall-output-12707 2>&1

2011/01/24-21:37:22 'outer' output from "ssh-agent bash -c /tmp/avupswaux-mapall-12707 /tmp/avupswaux-ssh-agent-mapall-

output-12707 2>&1":

2011/01/24-21:37:22 - - - - - - BEGIN

Identity added: /home/dpn/.ssh/dpnid (/home/dpn/.ssh/dpnid)

2011/01/24-21:37:22 - - - - - - END

2011/01/24-21:37:22 mapall command output from "ssh-agent bash -c /tmp/avupswaux-mapall-12707 /tmp/avupswaux-ssh-agent-

mapall-output-12707 2>&1":

2011/01/24-21:37:22 - - - - - - BEGIN

Using /tmp/avupswaux-tmp-12707-probedir/probe.xml

(0.s) ssh -x root@192.168.10.150 'cat /etc/redhat-release'

Red Hat Enterprise Linux ES release 4 (Nahant Update 6)

(0.0) ssh -x root@192.168.10.151 'cat /etc/redhat-release'

Red Hat Enterprise Linux ES release 4 (Nahant Update 6)

(0.1) ssh -x root@192.168.10.153 'cat /etc/redhat-release'

Red Hat Enterprise Linux ES release 4 (Nahant Update 6)

(0.2) ssh -x root@192.168.10.154 'cat /etc/redhat-release'

Red Hat Enterprise Linux ES release 4 (Nahant Update 6)

(0.3) ssh -x root@192.168.10.155 'cat /etc/redhat-release'

Red Hat Enterprise Linux ES release 4 (Nahant Update 6)

(0.4) ssh -x root@192.168.10.156 'cat /etc/redhat-release'

Red Hat Enterprise Linux ES release 4 (Nahant Update 6)

2011/01/24-21:37:22 - - - - - - END

2011/01/24-21:37:22 avupswaux: INFO: OS versions seen on node set "all": "4.6"

2011/01/24-21:37:22 avupswaux: INFO: OS version information for node 0.s: 4.6 ("Red Hat Enterprise Linux ES release 4 (Nahant Update

6)"; major version = 4)

2011/01/24-21:37:22 avupswaux: INFO: OS version information for node 0.0: 4.6 ("Red Hat Enterprise Linux ES release 4 (Nahant Update

6)"; major version = 4)

2011/01/24-21:37:22 avupswaux: INFO: OS version information for node 0.1: 4.6 ("Red Hat Enterprise Linux ES release 4 (Nahant Update

6)"; major version = 4)

2011/01/24-21:37:22 avupswaux: INFO: OS version information for node 0.2: 4.6 ("Red Hat Enterprise Linux ES release 4 (Nahant Update

6)"; major version = 4)

2011/01/24-21:37:22 avupswaux: INFO: OS version information for node 0.3: 4.6 ("Red Hat Enterprise Linux ES release 4 (Nahant Update

6)"; major version = 4)

2011/01/24-21:37:22 avupswaux: INFO: OS version information for node 0.4: 4.6 ("Red Hat Enterprise Linux ES release 4 (Nahant Update

6)"; major version = 4)

2011/01/24-21:37:22 avupswaux: INFO: checking storage status

2011/01/24-21:37:22 avupswaux: INFO: using default disable_perc_disk_drive_caches setting (1)

2011/01/24-21:37:22 tar Cczf /usr/local/avamar /tmp/avupswaux.12707.tgz 'lib/avupsw/check_storage_health'

2011/01/24-21:37:22 contents of temporary script "/tmp/avupswaux-mapall-12707":

2011/01/24-21:37:22 - - - - - - BEGIN

#!/bin/bash

ssh-add /home/dpn/.ssh/dpnid

[ -r /etc/profile ] && . /etc/profile

(export SYSPROBEUSER=root SYSPROBEDIR=/tmp/avupswaux-tmp-12707-probedir ; mapall --nodes=all --noerror 'if [ -e tmp -a ! -d tmp ] ;

then rm -f tmp ; fi') /tmp/avupswaux-mapall-command-output-12707 2>&1

2011/01/24-21:37:22 - - - - - - END

2011/01/24-21:37:22 chmod 700 /tmp/avupswaux-mapall-12707

2011/01/24-21:37:22 ssh-agent bash -c /tmp/avupswaux-mapall-12707 /tmp/avupswaux-ssh-agent-mapall-output-12707 2>&1

2011/01/24-21:37:24 'outer' output from "ssh-agent bash -c /tmp/avupswaux-mapall-12707 /tmp/avupswaux-ssh-agent-mapall-

output-12707 2>&1":

2011/01/24-21:37:24 - - - - - - BEGIN

Identity added: /home/dpn/.ssh/dpnid (/home/dpn/.ssh/dpnid)

2011/01/24-21:37:24 - - - - - - END

2011/01/24-21:37:24 mapall command output from "ssh-agent bash -c /tmp/avupswaux-mapall-12707 /tmp/avupswaux-ssh-agent-

mapall-output-12707 2>&1":

2011/01/24-21:37:24 - - - - - - BEGIN

Using /tmp/avupswaux-tmp-12707-probedir/probe.xml

(0.s) ssh -x root@192.168.10.150 'if [ -e tmp -a ! -d tmp ] ; then rm -f tmp ; fi'

(0.0) ssh -x root@192.168.10.151 'if [ -e tmp -a ! -d tmp ] ; then rm -f tmp ; fi'

(0.1) ssh -x root@192.168.10.153 'if [ -e tmp -a ! -d tmp ] ; then rm -f tmp ; fi'

(0.2) ssh -x root@192.168.10.154 'if [ -e tmp -a ! -d tmp ] ; then rm -f tmp ; fi'

(0.3) ssh -x root@192.168.10.155 'if [ -e tmp -a ! -d tmp ] ; then rm -f tmp ; fi'

(0.4) ssh -x root@192.168.10.156 'if [ -e tmp -a ! -d tmp ] ; then rm -f tmp ; fi'

2011/01/24-21:37:24 - - - - - - END

2011/01/24-21:37:24 contents of temporary script "/tmp/avupswaux-mapall-12707":

2011/01/24-21:37:24 - - - - - - BEGIN

#!/bin/bash

ssh-add /home/dpn/.ssh/dpnid

[ -r /etc/profile ] && . /etc/profile

(export SYSPROBEUSER=root SYSPROBEDIR=/tmp/avupswaux-tmp-12707-probedir ; mapall --nodes=all --noerror copy /tmp/avupswaux.12707.tgz)

/tmp/avupswaux-mapall-command-output-12707 2>&1

2011/01/24-21:37:24 - - - - - - END

2011/01/24-21:37:24 chmod 700 /tmp/avupswaux-mapall-12707

2011/01/24-21:37:24 ssh-agent bash -c /tmp/avupswaux-mapall-12707 /tmp/avupswaux-ssh-agent-mapall-output-12707 2>&1

2011/01/24-21:37:26 'outer' output from "ssh-agent bash -c /tmp/avupswaux-mapall-12707 /tmp/avupswaux-ssh-agent-mapall-

output-12707 2>&1":

2011/01/24-21:37:26 - - - - - - BEGIN

Identity added: /home/dpn/.ssh/dpnid (/home/dpn/.ssh/dpnid)

2011/01/24-21:37:26 - - - - - - END

2011/01/24-21:37:26 mapall command output from "ssh-agent bash -c /tmp/avupswaux-mapall-12707 /tmp/avupswaux-ssh-agent-

mapall-output-12707 2>&1":

2011/01/24-21:37:26 - - - - - - BEGIN

Using /tmp/avupswaux-tmp-12707-probedir/probe.xml

/bin/tar czhf /tmp/.mapall12999 /tmp/avupswaux.12707.tgz

/bin/tar: Removing leading `/' from member names

(0.s) scp -q /tmp/.mapall12999 root@192.168.10.150:.

(0.s) ssh -x root@192.168.10.150 'tar xzf .mapall12999; rm -f .mapall12999'

(0.0) scp -q /tmp/.mapall12999 root@192.168.10.151:.

(0.0) ssh -x root@192.168.10.151 'tar xzf .mapall12999; rm -f .mapall12999'

(0.1) scp -q /tmp/.mapall12999 root@192.168.10.153:.

(0.1) ssh -x root@192.168.10.153 'tar xzf .mapall12999; rm -f .mapall12999'

(0.2) scp -q /tmp/.mapall12999 root@192.168.10.154:.

(0.2) ssh -x root@192.168.10.154 'tar xzf .mapall12999; rm -f .mapall12999'

(0.3) scp -q /tmp/.mapall12999 root@192.168.10.155:.

(0.3) ssh -x root@192.168.10.155 'tar xzf .mapall12999; rm -f .mapall12999'

(0.4) scp -q /tmp/.mapall12999 root@192.168.10.156:.

(0.4) ssh -x root@192.168.10.156 'tar xzf .mapall12999; rm -f .mapall12999'

2011/01/24-21:37:26 - - - - - - END

2011/01/24-21:37:26 contents of temporary script "/tmp/avupswaux-mapall-12707":

2011/01/24-21:37:26 - - - - - - BEGIN

#!/bin/bash

ssh-add /home/dpn/.ssh/dpnid

[ -r /etc/profile ] && . /etc/profile

(export SYSPROBEUSER=root SYSPROBEDIR=/tmp/avupswaux-tmp-12707-probedir ; mapall --nodes=all --noerror 'mkdir avupswaux-work.12707 &&

cd avupswaux-work.12707 && tar xzf ../tmp/avupswaux.12707.tgz ; /bin/rm -f ../tmp/avupswaux.12707.tgz') /tmp/avupswaux-

mapall-command-output-12707 2>&1

2011/01/24-21:37:26 - - - - - - END

2011/01/24-21:37:26 chmod 700 /tmp/avupswaux-mapall-12707

2011/01/24-21:37:26 ssh-agent bash -c /tmp/avupswaux-mapall-12707 /tmp/avupswaux-ssh-agent-mapall-output-12707 2>&1

2011/01/24-21:37:27 'outer' output from "ssh-agent bash -c /tmp/avupswaux-mapall-12707 /tmp/avupswaux-ssh-agent-mapall-

output-12707 2>&1":

2011/01/24-21:37:27 - - - - - - BEGIN

Identity added: /home/dpn/.ssh/dpnid (/home/dpn/.ssh/dpnid)

2011/01/24-21:37:27 - - - - - - END

2011/01/24-21:37:27 mapall command output from "ssh-agent bash -c /tmp/avupswaux-mapall-12707 /tmp/avupswaux-ssh-agent-

mapall-output-12707 2>&1":

2011/01/24-21:37:27 - - - - - - BEGIN

Using /tmp/avupswaux-tmp-12707-probedir/probe.xml

(0.s) ssh -x root@192.168.10.150 'mkdir avupswaux-work.12707 && cd avupswaux-work.12707 && tar xzf ../tmp/avupswaux.12707.tgz ;

/bin/rm -f ../tmp/avupswaux.12707.tgz'

(0.0) ssh -x root@192.168.10.151 'mkdir avupswaux-work.12707 && cd avupswaux-work.12707 && tar xzf ../tmp/avupswaux.12707.tgz ;

/bin/rm -f ../tmp/avupswaux.12707.tgz'

(0.1) ssh -x root@192.168.10.153 'mkdir avupswaux-work.12707 && cd avupswaux-work.12707 && tar xzf ../tmp/avupswaux.12707.tgz ;

/bin/rm -f ../tmp/avupswaux.12707.tgz'

(0.2) ssh -x root@192.168.10.154 'mkdir avupswaux-work.12707 && cd avupswaux-work.12707 && tar xzf ../tmp/avupswaux.12707.tgz ;

/bin/rm -f ../tmp/avupswaux.12707.tgz'

(0.3) ssh -x root@192.168.10.155 'mkdir avupswaux-work.12707 && cd avupswaux-work.12707 && tar xzf ../tmp/avupswaux.12707.tgz ;

/bin/rm -f ../tmp/avupswaux.12707.tgz'

(0.4) ssh -x root@192.168.10.156 'mkdir avupswaux-work.12707 && cd avupswaux-work.12707 && tar xzf ../tmp/avupswaux.12707.tgz ;

/bin/rm -f ../tmp/avupswaux.12707.tgz'

2011/01/24-21:37:27 - - - - - - END

2011/01/24-21:37:27 contents of temporary script "/tmp/avupswaux-mapall-12707":

2011/01/24-21:37:27 - - - - - - BEGIN

#!/bin/bash

ssh-add /home/dpn/.ssh/dpnid

[ -r /etc/profile ] && . /etc/profile

(export SYSPROBEUSER=root SYSPROBEDIR=/tmp/avupswaux-tmp-12707-probedir ; mapall --nodes=all --noerror 'avupswaux-

work.12707/lib/avupsw/check_storage_health --disable_perc_disk_drive_caches ; /bin/rm -rf avupswaux-work.12707')

>/tmp/avupswaux-mapall-check_storage_health-output.12707 2>&1

2011/01/24-21:37:27 - - - - - - END

2011/01/24-21:37:27 chmod 700 /tmp/avupswaux-mapall-12707

2011/01/24-21:37:27 ssh-agent bash -c /tmp/avupswaux-mapall-12707 /tmp/avupswaux-ssh-agent-mapall-output-12707 2>&1

2011/01/24-21:37:35 'outer' output from "ssh-agent bash -c /tmp/avupswaux-mapall-12707 /tmp/avupswaux-ssh-agent-mapall-

output-12707 2>&1":

2011/01/24-21:37:35 - - - - - - BEGIN

Identity added: /home/dpn/.ssh/dpnid (/home/dpn/.ssh/dpnid)

2011/01/24-21:37:35 - - - - - - END

2011/01/24-21:37:35 mapall command output from "ssh-agent bash -c /tmp/avupswaux-mapall-12707 /tmp/avupswaux-ssh-agent-

mapall-output-12707 2>&1":

2011/01/24-21:37:35 - - - - - - BEGIN

Using /tmp/avupswaux-tmp-12707-probedir/probe.xml

(0.s) ssh -x root@192.168.10.150 'avupswaux-work.12707/lib/avupsw/check_storage_health --disable_perc_disk_drive_caches ; /bin/rm -rf

avupswaux-work.12707'

check_storage_health: INFO: [avamar-utility, 192.168.10.150] PERC storage controllers found: 1

check_storage_health: INFO: [avamar-utility, 192.168.10.150] on storage controller "0" found installed firmware version "6.0.2-0002"

check_storage_health: INFO: [avamar-utility, 192.168.10.150] on storage controller "0" found minimum required firmware version "Not

Applicable"

check_storage_health: INFO: [avamar-utility, 192.168.10.150] on storage controller "0" found installed driver version "00.00.03.16"

check_storage_health: INFO: [avamar-utility, 192.168.10.150] on storage controller "0" found minimum required driver version "Not

Applicable"

check_storage_health: INFO: [avamar-utility, 192.168.10.150] storage controller "0" shows state "Ready"

check_storage_health: INFO: [avamar-utility, 192.168.10.150] storage controller "0" shows alarm state "Not Applicable"

check_storage_health: INFO: [avamar-utility, 192.168.10.150] disabling patrol read via "omconfig storage controller

action=setpatrolreadmode controller="0" mode=manual ; omconfig storage controller action=stoppatrolread controller="0" ; omconfig

storage controller action=setpatrolreadmode controller="0" mode=disable"

Command successful!

Operation disabled. Read, action=stoppatrolread

.Refer to the documentation for more information.

Command successful!

check_storage_health: INFO: [avamar-utility, 192.168.10.150] disk cache policy setting for vdisk "0" on storage controller "0" is set

correctly as "Disabled"

check_storage_health: INFO: [avamar-utility, 192.168.10.150] storage controller "0" shows patrol read state "Stopped"

check_storage_health: INFO: [avamar-utility, 192.168.10.150] disk cache policy setting for vdisk "0" on storage controller "0" is set

correctly as "Disabled"

(0.0) ssh -x root@192.168.10.151 'avupswaux-work.12707/lib/avupsw/check_storage_health --disable_perc_disk_drive_caches ; /bin/rm -rf

avupswaux-work.12707'

check_storage_health: INFO: [avamar2, 192.168.10.151] PERC storage controllers found: 1

check_storage_health: INFO: [avamar2, 192.168.10.151] on storage controller "0" found installed firmware version "6.0.2-0002"

check_storage_health: INFO: [avamar2, 192.168.10.151] on storage controller "0" found minimum required firmware version "Not

Applicable"

check_storage_health: INFO: [avamar2, 192.168.10.151] on storage controller "0" found installed driver version "00.00.03.16"

check_storage_health: INFO: [avamar2, 192.168.10.151] on storage controller "0" found minimum required driver version "Not Applicable"

check_storage_health: INFO: [avamar2, 192.168.10.151] storage controller "0" shows state "Ready"

check_storage_health: INFO: [avamar2, 192.168.10.151] storage controller "0" shows alarm state "Not Applicable"

check_storage_health: INFO: [avamar2, 192.168.10.151] disabling patrol read via "omconfig storage controller action=setpatrolreadmode

controller="0" mode=manual ; omconfig storage controller action=stoppatrolread controller="0" ; omconfig storage controller

action=setpatrolreadmode controller="0" mode=disable"

Command successful!

Operation disabled. Read, action=stoppatrolread

.Refer to the documentation for more information.

Command successful!

check_storage_health: INFO: [avamar2, 192.168.10.151] disk cache policy setting for vdisk "0" on storage controller "0" is set

correctly as "Disabled"

check_storage_health: INFO: [avamar2, 192.168.10.151] disk cache policy setting for vdisk "1" on storage controller "0" is set

correctly as "Disabled"

check_storage_health: INFO: [avamar2, 192.168.10.151] disk cache policy setting for vdisk "2" on storage controller "0" is set

correctly as "Disabled"

check_storage_health: INFO: [avamar2, 192.168.10.151] disk cache policy setting for vdisk "3" on storage controller "0" is set

correctly as "Disabled"

check_storage_health: INFO: [avamar2, 192.168.10.151] storage controller "0" shows patrol read state "Stopped"

check_storage_health: INFO: [avamar2, 192.168.10.151] disk cache policy setting for vdisk "0" on storage controller "0" is set

correctly as "Disabled"

check_storage_health: INFO: [avamar2, 192.168.10.151] disk cache policy setting for vdisk "1" on storage controller "0" is set

correctly as "Disabled"

check_storage_health: INFO: [avamar2, 192.168.10.151] disk cache policy setting for vdisk "2" on storage controller "0" is set

correctly as "Disabled"

check_storage_health: INFO: [avamar2, 192.168.10.151] disk cache policy setting for vdisk "3" on storage controller "0" is set

correctly as "Disabled"

check_storage_health: INFO: [avamar2, 192.168.10.151] drive "0:0:0" with product id "ST3300655SS" and revision "S528" on storage

controller "0" is not on the watch list

check_storage_health: INFO: [avamar2, 192.168.10.151] drive "0:0:1" with product id "HUS156030VLS600" and revision "E516" on storage

controller "0" is not on the watch list

check_storage_health: INFO: [avamar2, 192.168.10.151] drive "0:0:2" with product id "HUS156030VLS600" and revision "E516" on storage

controller "0" is not on the watch list

check_storage_health: INFO: [avamar2, 192.168.10.151] drive "0:0:3" with product id "ST3300655SS" and revision "S528" on storage

controller "0" is not on the watch list

check_storage_health: INFO: [avamar2, 192.168.10.151] drive "1:0:4" with product id "ST3300655SS" and revision "S528" on storage

controller "0" is not on the watch list

check_storage_health: INFO: [avamar2, 192.168.10.151] drive "1:0:5" with product id "ST3300655SS" and revision "S528" on storage

controller "0" is not on the watch list

(0.1) ssh -x root@192.168.10.153 'avupswaux-work.12707/lib/avupsw/check_storage_health --disable_perc_disk_drive_caches ; /bin/rm -rf

avupswaux-work.12707'

check_storage_health: INFO: [avamar3, 192.168.10.153] PERC storage controllers found: 1

check_storage_health: INFO: [avamar3, 192.168.10.153] on storage controller "0" found installed firmware version "6.0.2-0002"

check_storage_health: INFO: [avamar3, 192.168.10.153] on storage controller "0" found minimum required firmware version "Not

Applicable"

check_storage_health: INFO: [avamar3, 192.168.10.153] on storage controller "0" found installed driver version "00.00.03.16"

check_storage_health: INFO: [avamar3, 192.168.10.153] on storage controller "0" found minimum required driver version "Not Applicable"

check_storage_health: INFO: [avamar3, 192.168.10.153] storage controller "0" shows state "Ready"

check_storage_health: INFO: [avamar3, 192.168.10.153] storage controller "0" shows alarm state "Not Applicable"

check_storage_health: INFO: [avamar3, 192.168.10.153] disabling patrol read via "omconfig storage controller action=setpatrolreadmode

controller="0" mode=manual ; omconfig storage controller action=stoppatrolread controller="0" ; omconfig storage controller

action=setpatrolreadmode controller="0" mode=disable"

Command successful!

Operation disabled. Read, action=stoppatrolread

.Refer to the documentation for more information.

Command successful!

check_storage_health: INFO: [avamar3, 192.168.10.153] disk cache policy setting for vdisk "0" on storage controller "0" is set

correctly as "Disabled"

check_storage_health: INFO: [avamar3, 192.168.10.153] disk cache policy setting for vdisk "1" on storage controller "0" is set

correctly as "Disabled"

check_storage_health: INFO: [avamar3, 192.168.10.153] disk cache policy setting for vdisk "2" on storage controller "0" is set

correctly as "Disabled"

check_storage_health: INFO: [avamar3, 192.168.10.153] disk cache policy setting for vdisk "3" on storage controller "0" is set

correctly as "Disabled"

check_storage_health: INFO: [avamar3, 192.168.10.153] storage controller "0" shows patrol read state "Stopped"

check_storage_health: INFO: [avamar3, 192.168.10.153] disk cache policy setting for vdisk "0" on storage controller "0" is set

correctly as "Disabled"

check_storage_health: INFO: [avamar3, 192.168.10.153] disk cache policy setting for vdisk "1" on storage controller "0" is set

correctly as "Disabled"

check_storage_health: INFO: [avamar3, 192.168.10.153] disk cache policy setting for vdisk "2" on storage controller "0" is set

correctly as "Disabled"

check_storage_health: INFO: [avamar3, 192.168.10.153] disk cache policy setting for vdisk "3" on storage controller "0" is set

correctly as "Disabled"

(0.2) ssh -x root@192.168.10.154 'avupswaux-work.12707/lib/avupsw/check_storage_health --disable_perc_disk_drive_caches ; /bin/rm -rf

avupswaux-work.12707'

check_storage_health: INFO: [avamar4, 192.168.10.154] PERC storage controllers found: 1

check_storage_health: INFO: [avamar4, 192.168.10.154] on storage controller "0" found installed firmware version "6.0.2-0002"

check_storage_health: INFO: [avamar4, 192.168.10.154] on storage controller "0" found minimum required firmware version "Not

Applicable"

check_storage_health: INFO: [avamar4, 192.168.10.154] on storage controller "0" found installed driver version "00.00.03.16"

check_storage_health: INFO: [avamar4, 192.168.10.154] on storage controller "0" found minimum required driver version "Not Applicable"

check_storage_health: INFO: [avamar4, 192.168.10.154] storage controller "0" shows state "Ready"

check_storage_health: INFO: [avamar4, 192.168.10.154] storage controller "0" shows alarm state "Not Applicable"

check_storage_health: INFO: [avamar4, 192.168.10.154] disabling patrol read via "omconfig storage controller action=setpatrolreadmode

controller="0" mode=manual ; omconfig storage controller action=stoppatrolread controller="0" ; omconfig storage controller

action=setpatrolreadmode controller="0" mode=disable"

Command successful!

Operation disabled. Read, action=stoppatrolread

.Refer to the documentation for more information.

Command successful!

check_storage_health: INFO: [avamar4, 192.168.10.154] disk cache policy setting for vdisk "0" on storage controller "0" is set

correctly as "Disabled"

check_storage_health: INFO: [avamar4, 192.168.10.154] disk cache policy setting for vdisk "1" on storage controller "0" is set

correctly as "Disabled"

check_storage_health: INFO: [avamar4, 192.168.10.154] disk cache policy setting for vdisk "2" on storage controller "0" is set

correctly as "Disabled"

check_storage_health: INFO: [avamar4, 192.168.10.154] disk cache policy setting for vdisk "3" on storage controller "0" is set

correctly as "Disabled"

check_storage_health: INFO: [avamar4, 192.168.10.154] storage controller "0" shows patrol read state "Stopped"

check_storage_health: INFO: [avamar4, 192.168.10.154] disk cache policy setting for vdisk "0" on storage controller "0" is set

correctly as "Disabled"

check_storage_health: INFO: [avamar4, 192.168.10.154] disk cache policy setting for vdisk "1" on storage controller "0" is set

correctly as "Disabled"

check_storage_health: INFO: [avamar4, 192.168.10.154] disk cache policy setting for vdisk "2" on storage controller "0" is set

correctly as "Disabled"

check_storage_health: INFO: [avamar4, 192.168.10.154] disk cache policy setting for vdisk "3" on storage controller "0" is set

correctly as "Disabled"

check_storage_health: INFO: [avamar4, 192.168.10.154] drive "0:0:0" with product id "HUS156030VLS600" and revision "E516" on storage

controller "0" is not on the watch list

check_storage_health: INFO: [avamar4, 192.168.10.154] drive "0:0:1" with product id "ST3300655SS" and revision "S528" on storage

controller "0" is not on the watch list

check_storage_health: INFO: [avamar4, 192.168.10.154] drive "0:0:2" with product id "ST3300655SS" and revision "S528" on storage

controller "0" is not on the watch list

check_storage_health: INFO: [avamar4, 192.168.10.154] drive "0:0:3" with product id "ST3300655SS" and revision "S528" on storage

controller "0" is not on the watch list

check_storage_health: INFO: [avamar4, 192.168.10.154] drive "1:0:4" with product id "ST3300655SS" and revision "S528" on storage

controller "0" is not on the watch list

check_storage_health: INFO: [avamar4, 192.168.10.154] drive "1:0:5" with product id "ST3300655SS" and revision "S528" on storage

controller "0" is not on the watch list

(0.3) ssh -x root@192.168.10.155 'avupswaux-work.12707/lib/avupsw/check_storage_health --disable_perc_disk_drive_caches ; /bin/rm -rf

avupswaux-work.12707'

check_storage_health: INFO: [avamar5, 192.168.10.155] PERC storage controllers found: 1

check_storage_health: INFO: [avamar5, 192.168.10.155] on storage controller "0" found installed firmware version "6.0.2-0002"

check_storage_health: INFO: [avamar5, 192.168.10.155] on storage controller "0" found minimum required firmware version "Not

Applicable"

check_storage_health: INFO: [avamar5, 192.168.10.155] on storage controller "0" found installed driver version "00.00.03.16"

check_storage_health: INFO: [avamar5, 192.168.10.155] on storage controller "0" found minimum required driver version "Not Applicable"

check_storage_health: INFO: [avamar5, 192.168.10.155] storage controller "0" shows state "Ready"

check_storage_health: INFO: [avamar5, 192.168.10.155] storage controller "0" shows alarm state "Not Applicable"

check_storage_health: INFO: [avamar5, 192.168.10.155] disabling patrol read via "omconfig storage controller action=setpatrolreadmode

controller="0" mode=manual ; omconfig storage controller action=stoppatrolread controller="0" ; omconfig storage controller

action=setpatrolreadmode controller="0" mode=disable"

Command successful!

Operation disabled. Read, action=stoppatrolread

.Refer to the documentation for more information.

Command successful!

check_storage_health: INFO: [avamar5, 192.168.10.155] disk cache policy setting for vdisk "0" on storage controller "0" is set

correctly as "Disabled"

check_storage_health: INFO: [avamar5, 192.168.10.155] disk cache policy setting for vdisk "1" on storage controller "0" is set

correctly as "Disabled"

check_storage_health: INFO: [avamar5, 192.168.10.155] disk cache policy setting for vdisk "2" on storage controller "0" is set

correctly as "Disabled"

check_storage_health: INFO: [avamar5, 192.168.10.155] disk cache policy setting for vdisk "3" on storage controller "0" is set

correctly as "Disabled"

check_storage_health: INFO: [avamar5, 192.168.10.155] storage controller "0" shows patrol read state "Stopped"

check_storage_health: INFO: [avamar5, 192.168.10.155] disk cache policy setting for vdisk "0" on storage controller "0" is set

correctly as "Disabled"

check_storage_health: INFO: [avamar5, 192.168.10.155] disk cache policy setting for vdisk "1" on storage controller "0" is set

correctly as "Disabled"

check_storage_health: INFO: [avamar5, 192.168.10.155] disk cache policy setting for vdisk "2" on storage controller "0" is set

correctly as "Disabled"

check_storage_health: INFO: [avamar5, 192.168.10.155] disk cache policy setting for vdisk "3" on storage controller "0" is set

correctly as "Disabled"

check_storage_health: INFO: [avamar5, 192.168.10.155] drive "0:0:0" with product id "ST3300655SS" and revision "S528" on storage

controller "0" is not on the watch list

check_storage_health: INFO: [avamar5, 192.168.10.155] drive "0:0:1" with product id "ST3300655SS" and revision "S528" on storage

controller "0" is not on the watch list

check_storage_health: INFO: [avamar5, 192.168.10.155] drive "0:0:2" with product id "ST3300655SS" and revision "S528" on storage

controller "0" is not on the watch list

check_storage_health: INFO: [avamar5, 192.168.10.155] drive "0:0:3" with product id "HUS156030VLS600" and revision "E516" on storage

controller "0" is not on the watch list

check_storage_health: INFO: [avamar5, 192.168.10.155] drive "1:0:4" with product id "ST3300655SS" and revision "S528" on storage

controller "0" is not on the watch list

check_storage_health: INFO: [avamar5, 192.168.10.155] drive "1:0:5" with product id "ST3300655SS" and revision "S528" on storage

controller "0" is not on the watch list

(0.4) ssh -x root@192.168.10.156 'avupswaux-work.12707/lib/avupsw/check_storage_health --disable_perc_disk_drive_caches ; /bin/rm -rf

avupswaux-work.12707'

check_storage_health: INFO: [avamar6, 192.168.10.156] PERC storage controllers found: 1

check_storage_health: INFO: [avamar6, 192.168.10.156] on storage controller "0" found installed firmware version "6.0.2-0002"

check_storage_health: INFO: [avamar6, 192.168.10.156] on storage controller "0" found minimum required firmware version "Not

Applicable"

check_storage_health: INFO: [avamar6, 192.168.10.156] on storage controller "0" found installed driver version "00.00.03.16"

check_storage_health: INFO: [avamar6, 192.168.10.156] on storage controller "0" found minimum required driver version "Not Applicable"

check_storage_health: INFO: [avamar6, 192.168.10.156] storage controller "0" shows state "Ready"

check_storage_health: INFO: [avamar6, 192.168.10.156] storage controller "0" shows alarm state "Not Applicable"

check_storage_health: INFO: [avamar6, 192.168.10.156] disabling patrol read via "omconfig storage controller action=setpatrolreadmode

controller="0" mode=manual ; omconfig storage controller action=stoppatrolread controller="0" ; omconfig storage controller

action=setpatrolreadmode controller="0" mode=disable"

Command successful!

Operation disabled. Read, action=stoppatrolread

.Refer to the documentation for more information.

Command successful!

check_storage_health: INFO: [avamar6, 192.168.10.156] disk cache policy setting for vdisk "0" on storage controller "0" is set

correctly as "Disabled"

check_storage_health: INFO: [avamar6, 192.168.10.156] disk cache policy setting for vdisk "1" on storage controller "0" is set

correctly as "Disabled"

check_storage_health: INFO: [avamar6, 192.168.10.156] disk cache policy setting for vdisk "2" on storage controller "0" is set

correctly as "Disabled"

check_storage_health: INFO: [avamar6, 192.168.10.156] disk cache policy setting for vdisk "3" on storage controller "0" is set

correctly as "Disabled"

check_storage_health: INFO: [avamar6, 192.168.10.156] storage controller "0" shows patrol read state "Stopped"

check_storage_health: INFO: [avamar6, 192.168.10.156] disk cache policy setting for vdisk "0" on storage controller "0" is set

correctly as "Disabled"

check_storage_health: INFO: [avamar6, 192.168.10.156] disk cache policy setting for vdisk "1" on storage controller "0" is set

correctly as "Disabled"

check_storage_health: INFO: [avamar6, 192.168.10.156] disk cache policy setting for vdisk "2" on storage controller "0" is set

correctly as "Disabled"

check_storage_health: INFO: [avamar6, 192.168.10.156] disk cache policy setting for vdisk "3" on storage controller "0" is set

correctly as "Disabled"

check_storage_health: INFO: [avamar6, 192.168.10.156] drive "0:0:0" with product id "ST3300655SS" and revision "S528" on storage

controller "0" is not on the watch list

check_storage_health: INFO: [avamar6, 192.168.10.156] drive "0:0:1" with product id "ST3300655SS" and revision "S528" on storage

controller "0" is not on the watch list

check_storage_health: INFO: [avamar6, 192.168.10.156] drive "0:0:2" with product id "ST3300655SS" and revision "S528" on storage

controller "0" is not on the watch list

check_storage_health: INFO: [avamar6, 192.168.10.156] drive "0:0:3" with product id "HUS156030VLS600" and revision "E516" on storage

controller "0" is not on the watch list

check_storage_health: INFO: [avamar6, 192.168.10.156] drive "1:0:4" with product id "ST3300655SS" and revision "S528" on storage

controller "0" is not on the watch list

check_storage_health: INFO: [avamar6, 192.168.10.156] drive "1:0:5" with product id "ST3300655SS" and revision "S528" on storage

controller "0" is not on the watch list

2011/01/24-21:37:35 - - - - - - END

2011/01/24-21:37:35 avupswaux: INFO: using configured relax_storage_health_check setting (1)

2011/01/24-21:37:35 avupswaux: INFO: checking node types

2011/01/24-21:37:35 avupswaux: INFO: no accelerator node IP address list configured

2011/01/24-21:37:35 avupswaux: INFO: no nodes of type "accelerator" found

2011/01/24-21:37:35 avupswaux: INFO: no access node IP address list configured

2011/01/24-21:37:35 avupswaux: INFO: no nodes of type "access" found

2011/01/24-21:37:35 avupswaux: INFO: no spare node IP address list configured

2011/01/24-21:37:35 avupswaux: INFO: no nodes of type "spare" found

2011/01/24-21:37:35 avupswaux: DEBUG: rsw_generic_text_file::exists(): file "/usr/local/avamar/var/probe.out" exists

2011/01/24-21:37:35 avupswaux: DEBUG: read probe file line "NONAT"

2011/01/24-21:37:35 avupswaux: DEBUG: saw required 'NONAT' line

2011/01/24-21:37:35 avupswaux: DEBUG: read probe file line "avamar-

utility:192.168.10.150,192.168.10.150:192.168.10.151,192.168.10.153,192.168.10.154,192.168.10.155,192.168.10.156"

2011/01/24-21:37:35 avupswaux: DEBUG: probe file: module id = "avamar-utility", utility nodes = "192.168.10.150,192.168.10.150",

storage nodes = "192.168.10.151,192.168.10.153,192.168.10.154,192.168.10.155,192.168.10.156"

2011/01/24-21:37:35 avupswaux: INFO: count of storage nodes found in "/usr/local/avamar/var/probe.out": 5

2011/01/24-21:37:35 contents of temporary script "/tmp/avupswaux-ssh-12707":

2011/01/24-21:37:35 - - - - - - BEGIN

#!/bin/bash

ssh-add /home/dpn/.ssh/dpnid

export TERM=${TERM:-dumb}

[ -r /etc/profile ] && . /etc/profile

echo '(192.168.10.151)' ssh -x -o PasswordAuthentication=no -o StrictHostKeyChecking=no root@192.168.10.151 'cat

/usr/local/avamar/etc/node.cfg' >>/tmp/avupswaux-ssh-apply.12707

(ssh -x -o PasswordAuthentication=no -o StrictHostKeyChecking=no root@192.168.10.151 'cat /usr/local/avamar/etc/node.cfg')

>>/tmp/avupswaux-ssh-apply.12707 2>&1

2011/01/24-21:37:35 - - - - - - END

2011/01/24-21:37:35 chmod 700 /tmp/avupswaux-ssh-12707

2011/01/24-21:37:35 export TERM=${TERM:-dumb} ; ssh-agent bash -c /tmp/avupswaux-ssh-12707 >/tmp/avupswaux-ssh-agent-ssh-

outer-output-12707 2>&1

2011/01/24-21:37:35 contents of temporary script "/tmp/avupswaux-ssh-12707":

2011/01/24-21:37:35 - - - - - - BEGIN

#!/bin/bash

ssh-add /home/dpn/.ssh/dpnid

export TERM=${TERM:-dumb}

[ -r /etc/profile ] && . /etc/profile

echo '(192.168.10.153)' ssh -x -o PasswordAuthentication=no -o StrictHostKeyChecking=no root@192.168.10.153 'cat

/usr/local/avamar/etc/node.cfg' >>/tmp/avupswaux-ssh-apply.12707

(ssh -x -o PasswordAuthentication=no -o StrictHostKeyChecking=no root@192.168.10.153 'cat /usr/local/avamar/etc/node.cfg')

>>/tmp/avupswaux-ssh-apply.12707 2>&1

2011/01/24-21:37:35 - - - - - - END

2011/01/24-21:37:35 chmod 700 /tmp/avupswaux-ssh-12707

2011/01/24-21:37:35 export TERM=${TERM:-dumb} ; ssh-agent bash -c /tmp/avupswaux-ssh-12707 >/tmp/avupswaux-ssh-agent-ssh-

outer-output-12707 2>&1

2011/01/24-21:37:35 contents of temporary script "/tmp/avupswaux-ssh-12707":

2011/01/24-21:37:35 - - - - - - BEGIN

#!/bin/bash

ssh-add /home/dpn/.ssh/dpnid

export TERM=${TERM:-dumb}

[ -r /etc/profile ] && . /etc/profile

echo '(192.168.10.154)' ssh -x -o PasswordAuthentication=no -o StrictHostKeyChecking=no root@192.168.10.154 'cat

/usr/local/avamar/etc/node.cfg' >>/tmp/avupswaux-ssh-apply.12707

(ssh -x -o PasswordAuthentication=no -o StrictHostKeyChecking=no root@192.168.10.154 'cat /usr/local/avamar/etc/node.cfg')

>>/tmp/avupswaux-ssh-apply.12707 2>&1

2011/01/24-21:37:35 - - - - - - END

2011/01/24-21:37:35 chmod 700 /tmp/avupswaux-ssh-12707

2011/01/24-21:37:35 export TERM=${TERM:-dumb} ; ssh-agent bash -c /tmp/avupswaux-ssh-12707 >/tmp/avupswaux-ssh-agent-ssh-

outer-output-12707 2>&1

2011/01/24-21:37:35 contents of temporary script "/tmp/avupswaux-ssh-12707":

2011/01/24-21:37:35 - - - - - - BEGIN

#!/bin/bash

ssh-add /home/dpn/.ssh/dpnid

export TERM=${TERM:-dumb}

[ -r /etc/profile ] && . /etc/profile

echo '(192.168.10.155)' ssh -x -o PasswordAuthentication=no -o StrictHostKeyChecking=no root@192.168.10.155 'cat

/usr/local/avamar/etc/node.cfg' >>/tmp/avupswaux-ssh-apply.12707

(ssh -x -o PasswordAuthentication=no -o StrictHostKeyChecking=no root@192.168.10.155 'cat /usr/local/avamar/etc/node.cfg')

>>/tmp/avupswaux-ssh-apply.12707 2>&1

2011/01/24-21:37:35 - - - - - - END

2011/01/24-21:37:35 chmod 700 /tmp/avupswaux-ssh-12707

2011/01/24-21:37:35 export TERM=${TERM:-dumb} ; ssh-agent bash -c /tmp/avupswaux-ssh-12707 >/tmp/avupswaux-ssh-agent-ssh-

outer-output-12707 2>&1

2011/01/24-21:37:35 contents of temporary script "/tmp/avupswaux-ssh-12707":

2011/01/24-21:37:35 - - - - - - BEGIN

#!/bin/bash

ssh-add /home/dpn/.ssh/dpnid

export TERM=${TERM:-dumb}

[ -r /etc/profile ] && . /etc/profile

echo '(192.168.10.156)' ssh -x -o PasswordAuthentication=no -o StrictHostKeyChecking=no root@192.168.10.156 'cat

/usr/local/avamar/etc/node.cfg' >>/tmp/avupswaux-ssh-apply.12707

(ssh -x -o PasswordAuthentication=no -o StrictHostKeyChecking=no root@192.168.10.156 'cat /usr/local/avamar/etc/node.cfg')

>>/tmp/avupswaux-ssh-apply.12707 2>&1

2011/01/24-21:37:35 - - - - - - END

2011/01/24-21:37:35 chmod 700 /tmp/avupswaux-ssh-12707

2011/01/24-21:37:35 export TERM=${TERM:-dumb} ; ssh-agent bash -c /tmp/avupswaux-ssh-12707 >/tmp/avupswaux-ssh-agent-ssh-

outer-output-12707 2>&1

2011/01/24-21:37:36 collected 'outer' output from preceding ssh operation:

2011/01/24-21:37:36 - - - - - - BEGIN

Identity added: /home/dpn/.ssh/dpnid (/home/dpn/.ssh/dpnid)

2011/01/24-21:37:36 - - - - - - END

2011/01/24-21:37:36 collected output from preceding ssh operation:

2011/01/24-21:37:36 - - - - - - BEGIN

(192.168.10.151) ssh -x -o PasswordAuthentication=no -o StrictHostKeyChecking=no root@192.168.10.151 cat /usr/local/avamar/etc/node.cfg

node type=data

os version=5.0.0

install version=4.0.9

(192.168.10.153) ssh -x -o PasswordAuthentication=no -o StrictHostKeyChecking=no root@192.168.10.153 cat /usr/local/avamar/etc/node.cfg

node type=data

os version=5.0.0

install version=4.0.9

(192.168.10.154) ssh -x -o PasswordAuthentication=no -o StrictHostKeyChecking=no root@192.168.10.154 cat /usr/local/avamar/etc/node.cfg

node type=data

os version=5.0.0

install version=4.0.9

(192.168.10.155) ssh -x -o PasswordAuthentication=no -o StrictHostKeyChecking=no root@192.168.10.155 cat /usr/local/avamar/etc/node.cfg

node type=data

os version=5.0.0

install version=4.0.9

(192.168.10.156) ssh -x -o PasswordAuthentication=no -o StrictHostKeyChecking=no root@192.168.10.156 cat /usr/local/avamar/etc/node.cfg

node type=data

os version=5.0.0

install version=4.0.9

2011/01/24-21:37:36 - - - - - - END

2011/01/24-21:37:36 avupswaux: INFO: using default relax_node_type_check setting (0)

2011/01/24-21:37:36 avupswaux: INFO: the following nodes had the expected types (apart from utility or single-node server):

node 192.168.10.151 has canonical type storage (configured type: data)

node 192.168.10.153 has canonical type storage (configured type: data)

node 192.168.10.154 has canonical type storage (configured type: data)

node 192.168.10.155 has canonical type storage (configured type: data)

node 192.168.10.156 has canonical type storage (configured type: data)

2011/01/24-21:37:36 avupswaux: INFO: bug #13216 should not be present in installed dpnmcs package version "5.0.3-29" - not checking for

bug #13216 hot fix compliance

2011/01/24-21:37:36 avupswaux: INFO: using configured relax_avamar_up_requirement setting (1)

2011/01/24-21:37:36 avupswaux: INFO: checking that required Avamar server components are all up/enabled

2011/01/24-21:37:36 avupswaux: INFO: applying "dpnctl status --xml >/tmp/avupswaux-command-output.12707 2>/dev/null"

2011/01/24-21:37:36 avupswaux: ERROR: unexpected empty output received from "dpnctl status --xml"

2011/01/24-21:37:36 avupswaux: WARNING: required Avamar server components are not all up/enabled at this time

2011/01/24-21:37:36 avupswaux: INFO: using configured relax_health_check setting (1)

2011/01/24-21:37:36 avupswaux: INFO: applying "dpnctl status --xml >/tmp/avupswaux-command-output.12707 2>/dev/null"

2011/01/24-21:37:36 avupswaux: ERROR: unexpected empty output received from "dpnctl status --xml"

2011/01/24-21:37:36 avupswaux: INFO: expect Avamar health check to fail because Avamar subsystems are not up

2011/01/24-21:37:36 avupswaux: INFO: checking Avamar server health

2011/01/24-21:37:36 contents of temporary script "/tmp/avupswaux-run-command-12707":

2011/01/24-21:37:36 - - - - - - BEGIN

#!/bin/bash

ssh-add /home/admin/.ssh/dpnid

[ -r /etc/profile ] && . /etc/profile

cd

export USERNAME="$LOGNAME"

( export PERL5LIB='/usr/local/avamar/lib/avupsw' ; /usr/local/avamar/lib/avupsw/health_check.pl --

log='/usr/local/avamar/var/health_check.log' --norequire_sched_up --norequire_unattended_startup ) /tmp/avupswaux-health-

check-output.12707 2>&1

2011/01/24-21:37:36 - - - - - - END

2011/01/24-21:37:36 chmod 700 /tmp/avupswaux-run-command-12707

2011/01/24-21:37:36 avupswaux: INFO: applying "chown 'admin' '/tmp/avupswaux-run-command-12707' >/tmp/avupswaux-command-output.12707

2>&1"

2011/01/24-21:37:36 avupswaux: INFO: applying "chmod 0700 '/tmp/avupswaux-run-command-12707' >/tmp/avupswaux-command-output.12707 2>&1"

2011/01/24-21:37:36 avupswaux: INFO: applying "( export PERL5LIB='/usr/local/avamar/lib/avupsw' ;

/usr/local/avamar/lib/avupsw/health_check.pl --log='/usr/local/avamar/var/health_check.log' --norequire_sched_up --

norequire_unattended_startup ) /tmp/avupswaux-health-check-output.12707 2>&1"

2011/01/24-21:37:43 'outer' output from "sudo -u 'admin' -H ssh-agent bash -c /tmp/avupswaux-run-command-12707

>/tmp/avupswaux-ssh-agent-command-output-12707 2>&1":

2011/01/24-21:37:43 - - - - - - BEGIN

Identity added: /home/admin/.ssh/dpnid (/home/admin/.ssh/dpnid)

2011/01/24-21:37:43 - - - - - - END

2011/01/24-21:37:43 command output from "sudo -u 'admin' -H ssh-agent bash -c /tmp/avupswaux-run-command-12707

>/tmp/avupswaux-ssh-agent-command-output-12707 2>&1":

2011/01/24-21:37:43 - - - - - - BEGIN

---------- BEGIN ----------

# Check system readiness

Getting current status...

ERROR: no database type or no node resource database could be found with search list /usr/local/avamar/var/probe.xml, stopped at

/usr/local/avamar/bin/../lib/dpnutils/node_util.pm line 748.

---------- END ----------

---------- BEGIN ----------

# Check GSAN readiness

Node: 0.0

Version: 5.0.3-29

State: ONLINE

Runlevel: fullaccess

Mode: mhpu+0hpu+0hpu

Suspended: false

Node: 0.1

Version: 5.0.3-29

State: ONLINE

Runlevel: fullaccess

Mode: mhpu+0hpu+0hpu

Suspended: false

Node: 0.2

Version: 5.0.3-29

State: ONLINE

Runlevel: fullaccess

Mode: mhpu+0hpu+0hpu

Suspended: false

Node: 0.3

Version: 5.0.3-29

State: ONLINE

Runlevel: fullaccess

Mode: mhpu+0hpu+0hpu

Suspended: false

Node: 0.4

Version: 5.0.3-29

State: ONLINE

Runlevel: fullaccess

Mode: mhpu+0hpu+0hpu

Suspended: false

---------- END ----------

---------- BEGIN ----------

# Validate the checkpoint status

Last good hfschecked checkpoint:

ERROR: There are currently no hfschecked checkpoints!

---------- END ----------

---------- BEGIN ----------

# Review last hfscheck

Last full hfscheck was successful (errcode=4003)

Number of nodes queried: 5

Number of nodes replied: 5

No errors in last hfscheck message (errors="0")

---------- END ----------

---------- BEGIN ----------

# Review gc.log file (previous 10 passes only)

ERROR: Garbage collecting less than 10GB per pass! (averaging 0.00 GB)

---------- END ----------

---------- BEGIN ----------

# Check key configuration paramaters

disknogc: 85

diskreadonly: 65

balancemin: 0

cpmostrecent: 2

cphfscheck: 1

---------- END ----------

---------- BEGIN ----------

# Verify overall capacity on all nodes

---------- END ----------

---------- BEGIN ----------

# Ensure all nodes have time server selected