NMDA: IBM DB2 snapshot backups are unresponsive
Summary: NetWorker Module for Databases and Applications (NMDA) IBM DB2 snapshot backups are unresponsive.
Symptoms
NMDA IBM DB2 snapshot backups are unresponsive.
DB2 database is write suspend mode.
Error in db2diag.log
Warning: ---------------------------------------- 2016-05-20-15.05.16.771114+000 E7423877A808 LEVEL: Warning PID : 7733668 TID : 47390 PROC : db2sysc 0 INSTANCE: <instance_name> NODE : 000 DB : APPHDL : 0-57647 APPID: <app_id> AUTHID : EDUID : 47390 EDUNAME: db2agent (instance) 0 FUNCTION: DB2 UDB, database monitor, sqm___sqlmonssagnt, probe:1234 DATA #1 : <preformatted> This occurs when any snapshot other than list applications is attempted on a database that is write suspended. This is necessary due to latching conflicts. This database is skipped in the snapshot, but other databases should still be processed. Skipping database '<db_name>'. For more information on write suspend, please see the 'db2 set write suspend for database' command. ------------------------------------------
Cause
DB2 workflow works as:
DB2 calls db2ACSPrepare() in the NMDA DB2 library to do initial preparation to get ready for the hardware snapshot. Once this is done, DB2 Database manager suspends the database and calls NMDA DB2 function db2ACSSnapshot(). In this case, DB2 does not call db2ACSSnapshot(). The workflow stops responding for the DB2 engine to continue the backup. IBM support PMR: 29538,004,000 is opened. IBM DB2 engineering needs more debug to determine the root cause of this hanging.
Resolution
Workaround:
- Cancel current backup
- Take database out of write suspend mode
In order to discover the root cause, IBM DB2 support requests following info for further investigation,
- Delete or relocate any stacks or dumps in the db2dump directory. It creates new stacks or dumps without mixing old ones. (UNIX/Linux path:
/$INSTHOME/sqllib/db2dump) - DB2 trace for a short while:
db2trc on -f trace.dmp wait about 30 seconds db2trc off db2trc fmt trace.dmp trace.fmt db2trc flw trace.dmp trace.flw
- Gather stacks:
db2pd -stack all -rep 5 5
- After the above step completes, run,
procstack <PID> > db2vend.pstack.out
The <PID> is the Process ID of thedb2vendprocess for the database. - Then get the snapshot backup session,
db2 get snapshot for applications on <dbname> > db2snap.out
- Now you can set write resume, and then try the snapshot backup again and see if it finishes.
- Gather a db2support.zip as follows:
db2support . -d <dbname> -c -s
- Zip and send to IBM PMR:
trace.dmptrace.fmttrace.flwdb2vend.pstack.outdb2snap.outdb2support.zip