PowerScale: NDMP Snapshot Fast Incremental taking longer than expected
Summary: Under certain conditions, NDMP may experience performance issues when attempting to run Snapshot faster incremental jobs.
Instructions
Understanding SNAPSHOT Mode
In SNAPSHOT mode, after receiving the NDMP_DATA_START_BACKUP with variables from the Data Management Application (DMA), PowerScale creates a new snapshot and uses the supplied BASE_DATE variable to find the associated snapshot. This snapshot is used to create the Change Database (cdb) between the two snapshots. The Change Database (cdb) is tracked by seconds, and the time it takes to build depends on the number of changes made since the last backup for the FS.
Impact of High Change Rates on NDMP Performance
When the change rate exceeds 2% for the path associated to the backup, NDMP performance can be impacted due to the large number of changes and the time it takes to build the Change Database (cdb).
Troubleshooting: Checking Backup Mode and Change Database (cdb) Build Time.
To determine if jobs are running in SNAPSHOT FI mode, use the following commands:
Live: grep BACKUP_MODE /ifs/.ifsvar/modules/ndmp/node*-ndmp.trace |lessLogs: grep BACKUP_MODE local/ifsvar_modules.tar/modules/ndmp/trace/node*-ndmpd.trace |less
Output may look similar to:
local/ifsvar_modules.tar/modules/ndmp/trace/node13-ndmpd.trace.9: Name (value) : BACKUP_MODE (SNAPSHOT)local/ifsvar_modules.tar/modules/ndmp/trace/node13-ndmpd.trace.9: Name (value) : BACKUP_MODE (SNAPSHOT)local/ifsvar_modules.tar/modules/ndmp/trace/node13-ndmpd.trace.9: Name (value) : BACKUP_MODE (SNAPSHOT)local/ifsvar_modules.tar/modules/ndmp/trace/node13-ndmpd.trace.9: Name (value) : BACKUP_MODE (SNAPSHOT)
To verify the amount of time spent building the Change Database (cdb), use the following commands:
Live: grep cdb /ifs/.ifsvar/modules/ndmp/node*-ndmp.trace | lessLogs: grep cdb local/ifsvar_modules.tar/modules/ndmp/trace/node*-ndmpd.trace | less
Output may look similar to:
Log : CPU user=3986.358010 sys=380527.011168 ft=244728.838385 cdb=76561.738506Log : CPU user=412.115524 sys=98946.219716 ft=14675.093316 cdb=8817.705529Log : CPU user=1971.577102 sys=380480.629420 ft=116.502309 cdb=93954.847617Log : CPU user=1768.215238 sys=380712.643814 ft=36.631554 cdb=73611.565402
Recommended Action
If NDMP performance impact is found to be related to the time spent building the Change Database (cdb), for better performance consider revising the BACKUP_MODE to TIMESTAMP for better performance.
For more information about SNAPSHOT mode and TIMESTAMP mode, refer to KB PowerScale (Isilon) OneFS 9 - NDMP Timestamp Backup Mode vs. Snapshot Backup Mode job types and requirements