PowerScale: Understanding NDMP Timestamp vs. Snapshot Backup Mode and Operations

PowerScale OneFS utilizes the Network Data Management Protocol (NDMP) variable BACKUP_MODE to determine how snapshots are handled during backup operations. The BACKUP_MODE variable has two possible values: TIEMSATMP (token) or SNAPSHOT. 

TIMESTAMP (Default)

1. Snapshot Creation

When a backup job is initiated by a Data Management Application (DMA), OneFS creates a snapshot of the dataset to ensure a consistent view of the file system during the backup.

2. Token-Based Change Detection

After the snapshot is created, OneFS performs a treewalk of the snapshot to identify files that have changed.

• It compares each file’s CTIME (change time) and MTIME (modification time) against a BASE_DATE token.
• The BASE_DATE represents the timestamp of the last successful backup.
• A second token is generated at the time of the current backup to mark the new reference point.

Only files with CTIME or MTIME newer than the BASE_DATE are included in the backup.

3. Backup Execution

• The system backs up only the files that have changed since the last backup.
• After the backup completes, the current time token becomes the new BASE_DATE for the next incremental backup.
• The system then deletes the NDMP snapshot that was originally taken at the beginning of the job. 

Performance Considerations

• Timestamp mode is efficient for most environments and is the recommended default.

Example Timestamp Full 

Fri May 29 07:10:21 2020 (1590754221): Received from  xxx.xx.xx.85; Session:24681
Message   : 0x401 (NDMP_DATA_START_BACKUP)
Timestamp : 1590754425
XSequence : 11
RSequence : 11
Error     : 0 (NDMP_NO_ERR)
        Bkup type : dump
        Num Env. Var : 10
                Name (value) : FILESYSTEM (/ifs/data/test)
                Name (value) : FILES ()
                Name (value) : EXCLUDE ()
                Name (value) : HIST (d)
                Name (value) : DIRECT (N)
                Name (value) : UPDATE (Y)
                Name (value) : RECURSIVE (Y)
                Name (value) : PATHNAME_SEPARATOR (/)
                Name (value) : BASE_DATE (0)
                Name (value) : TYPE (tar)

Example Timestamp Incremental 

Fri May 29 07:41:29 2020 (1590756089): Received from  xxx.xx.xx.85; Session:31618
Message   : 0x401 (NDMP_DATA_START_BACKUP)
Timestamp : 1590755961
XSequence : 11
RSequence : 11
Error     : 0 (NDMP_NO_ERR)
        Bkup type : dump
        Num Env. Var : 10
                Name (value) : FILESYSTEM (/ifs/data/test)
                Name (value) : FILES ()
                Name (value) : EXCLUDE ()
                Name (value) : HIST (d)
                Name (value) : DIRECT (N)
                Name (value) : UPDATE (Y)
                Name (value) : RECURSIVE (Y)
                Name (value) : PATHNAME_SEPARATOR (/)
                Name (value) : BASE_DATE (1590418268)
                Name (value) : TYPE (tar)

SNAPSHOT (FI)

During a Snapshot Faster-Incremental (FI) backup, OneFS checks the snapshot that is taken for the previous NDMP backup operation and compares it to a new snapshot. OneFS then backs up all files that were modified since the last snapshot was made.

1. Snapshot Creation

When an NDMP backup job is triggered by a Data Management Application (DMA), OneFS creates a snapshot of the dataset. This snapshot provides a consistent, point-in-time view of the file system for the backup.

2. Change Database (CDB) Comparison

If the backup is configured to use Snapshot mode with a BASE_DATE, OneFS uses the Change Database to identify differences between two snapshots:

• OneFS compares the new snapshot (created at the start of the job) with a previous snapshot associated with the BASE_DATE.
• It builds a CDB, which is a list of files that have changed (added, modified, or deleted) between the two snapshots.

This comparison allows OneFS to:

• • Avoid scanning the entire file system.
  • Treewalk only the changed subset of files, significantly reducing backup time and resource usage.

Performance Considerations

• This mode is ideal when less than 2% of the data has changed between snapshots.
• If the change rate is higher, the backup will still complete successfully, but may take significantly longer due to the time required to build the CDB.
• You can view the time spent on this operation in the trace log summary at the end of the backup job.

Example Snapshot Full 

Mon Jun  8 00:56:42 2020 (1591595802): Received from  xxx.xx.xx.85; Session:87792
Message   : 0x401 (NDMP_DATA_START_BACKUP)
Timestamp : 1591596299
XSequence : 11
RSequence : 11
Error     : 0 (NDMP_NO_ERR)
        Bkup type : dump
        Num Env. Var : 12
                Name (value) : FILESYSTEM (/ifs/data/test)
                Name (value) : FILES ()
                Name (value) : EXCLUDE ()
                Name (value) : HIST (n)
                Name (value) : DIRECT (N)
                Name (value) : UPDATE (Y)
                Name (value) : RECURSIVE (Y)
                Name (value) : PATHNAME_SEPARATOR (/)
                Name (value) : BASE_DATE (0)
                Name (value) : BACKUP_OPTIONS (7)
                Name (value) : TYPE (tar)
                Name (value) : BACKUP_MODE (snapshot)

Example Snapshot Incremental 

Sun Jun  7 23:47:07 2020 (1591591627): Received from  xxx.xx.xx.85; Session:4354
Message   : 0x401 (NDMP_DATA_START_BACKUP)
Timestamp : 1591591767
XSequence : 11
RSequence : 11
Error     : 0 (NDMP_NO_ERR)
        Bkup type : dump
        Num Env. Var : 12
                Name (value) : FILESYSTEM (/ifs/data/test)
                Name (value) : FILES ()
                Name (value) : EXCLUDE ()
                Name (value) : HIST (d)
                Name (value) : DIRECT (N)
                Name (value) : UPDATE (Y)
                Name (value) : RECURSIVE (Y)
                Name (value) : PATHNAME_SEPARATOR (/)
                Name (value) : BASE_DATE (1591505215)
                Name (value) : BACKUP_OPTIONS (3)
                Name (value) : TYPE (tar)
                Name (value) : BACKUP_MODE (snapshot)