NetWorker: NMDA MySQL Triage Guide

Getting Started:

Write down customer s problem description and/or screenshot  and  your own observation notes and errors:

• NetWorker server OS platform
• NetWorker server version and build number
• NetWorker server daemon.raw(preferably rendered)
• NetWorker client OS platform
• NetWorker client version and build number
• NetWorker client daemon.raw  (preferably rendered)
• Provide the OS platform,architecture and version of the client (uname aoutputonUnix/Linux).

NMDA information:

• Provide the version information of NMDA binaries.

• Provide the NMDA configuration file used in the Backup Command  field (e.g.nsrdasv z /nsr/apps/config/nmda_oracle<SID>.cfg) of the client resource configured for the scheduled backups, or provided in the command line for manual operations.
• Provide the cluster information (e.g.virtual host names involved in the cluster, cluster type), if this is a cluster environment.(Ask if the customer can provide a copy of the hosts file on the client)
• If the error message is pointing an issue in the NetWorker save or recover session, provide NetWorker specific info as follows:
• Device type (DDBoost, VTL,..)
• Is it a Client Direct or SN backup to DD, is it Local or Remote SN, is it an Avamar backup, ..?
• Provide the rendered daemon.raw from SN from the same time window when the failure occurred
• Provide the version information of NMDA binaries.
•  

Unix:

strings/usr/sbin/nsrdasv  | grep Build

strings  /usr/sbin/nsrdaprobe  | grep Build 

(only if probe fails)strings/usr/lib/libnsrora.so | grep  @(#)   (or libnsrora.a)

Windows:

Right click on the file %NW_install_path%\bin\nsrsbtcn.exe  -> Properties -> Details on Windows

• If NMDA Oracle backup is configured via the wizard,tar/zip up  a copy of thensrdbfolder

Enable Debugging:

Set  NSR_DEBUG_LEVEL=9 in the NMDA configuration file, or in the Advanced Option table of the wizard  (if the configuration is created by wizard).

Do not ask user to set NSR_DPRINTF=TRUE, to avoid debug logs growing to a big size, unless the error message is pointing to a issue in NMDA save or recover session with NetWorker, or this is explicitly requested by the engineering.(In other words, the NSR_DPRINTF=TRUE can generate "lower layer errors" like Data Domain Error code, but the downside is the verbosity of the logs)

Note that: The total debug log size might be concern for NMDA 1.2 or older version when enabling the debugging in big-size database environment where the failure occurs after a couple of hours.. The debug log size is expected to be reduced up to %50  in NMDA 1.5.

Information Gathering and Log gathering:

Tar/zip the whole directory of  /nsr/apps/logs

Note:  this directory contains operational log file nmda_<app>.messages.raw  and debug log files.  The default debug logs location can be changed by setting NSR_DIAGNOSTIC_DEST in the configuration file.Note:  Oracle Rman output and NW server Daemon.raw  are very important for troubleshooting Oracle backup(make sure you remember to collect these information)

MySQL  Information:

• Check the MySQL version and MEB version installed against the Release Notes for the version of NMDA you are using and ensure we have a Supported configuration.  If not recommend upgrade of NetWorker/NMDA on the client or installing supported version MySQL and/or MEB as directed.
• Provide a copy of the MySQL configuration

        E.g.

        /etc/my.cnf

       or CNF file (MYSQL_CFG_FILE in the NMDA cfgfile). See http://dev.mysql.com/doc/refman/5.5/en/option-files.html for details.

• Provide a copy of the MySQL binary log index file.

By default, this has the same basename as the binary log file, with the extension '.index' (e.g. /var/log/mysql/mysql-bin.index),

andits location defined by log-bin parameter in MYSQL configuration file. You can change the name of the binary log index file with the --log-bin-index[=file_name] option. You should not manually edit this file while mysqld is running; doing so would confuse mysqld.

• Provide a copy of the MySQL error log (instance.err).  Can specify wheremysqldwrites the error log with the --log-error[=file_name] option. If the option is given with no file_name value, mysqld uses the name host_name.err by default. The server creates the file in the data directory unless an absolute path name is given to specify a different directory.
• Dump of  mysqladmin variables  to a text file

          E.g.

          mysqladmin-u root -h 127.0.0.1 --password=football1 variable

          ( displaydatabase parameter settings)