Data Domain: How to monitor data movement using a Data Domain Restorer

Certain models of Data Domain Restorer (DDR) can be configured with functionality known as Extended Retention (ER) or Long Term Retention (LTR). This functionality allows a second tier of storage to be added to the system to which files needing long term retention can be migrated. The type of storage used for this second tier depends on specific functionality being used, i.e.:

• Extended Retention: The second tier of storage is known as the archive tier and is formed from disk enclosures physically attached to the DDR
• Long Term Retention: The second tier of storage is known as the cloud tier and is formed from object storage provisioned from a supported cloud provider (i.e. it is external to the DDR)

Mechanics of data movement:

In both cases, to physically migrate file from the active to archive/cloud tier, a process known as data movement (FMIG) is used. When started FMIG runs through 5 phases of operation as described below:

Phase 1 - Bulk Seeding:

• This is skipped on all LTR enabled systems
• This is skipped on ER systems running DDOS 5.5.1.x (and later)
• On ER systems running DDOS 5.5.0.x (and earlier) this can perform a block level copy of selected data from the active tier to archive tier. The mechanics of this copy are beyond the scope of this document hence not discussed here.

Phase 2 - Scanning:

• The Data Domain File System (DDFS) looks for files in the active tier which are candidates for migration to the archive/cloud tier (files which have not been modified in longer than the parent mtrees data movement policy/age threshold)
• A number of copy threads are started (specific count depends on system model/configuration) with each of the threads reading a candidate file on the active tier and copying unique data to the archive/cloud tier (i.e. data which does not already exist on the target tier)
• A number of verification threads are started (again specific thread count depends on system model/configuration) which, once a candidate file has completed copy, read back the copy of the file from the target tier to make sure that it is valid/all data is reachable
• Once a candidate file has been copied/verified it is added to a list of files pending 'installation' in the target tier (a metadata update against the file to indicate that it now exists on the target tier). Periodically an installation thread wakes up to 'install' all pending files

Phase 3 - Verification:

• During this phase DDFS waits for verification of all copied files on the target tier to finish

Phase 4 - Packing:

• This is skipped on all LTR enabled systems
• On ER systems FMIG attempts to remove any duplicate segments of data which were copied to the archive tier during the scanning phase. Note that duplicate segments of data may be copied to speed up the copy process however given the design of the archive tier (maximum overall compression ratio with degraded read performance) such duplicate data needs to be removed

Phase 5 - Installation:

• During this phase DDFS waits for installation of all copied/verified files to complete

Monitoring progress of FMIG:

Primarily the Data Domain command line shell (DDSH) is used to monitor the progress of FMIG on ER and LTR systems. Note that:

• On ER systems the 'archive data-movement [status|watch]' commands should be used whereas on LTR systems the 'data-movement [status|watch]' commands should be used, i.e.:

• On ER systems progress of all 5 phases is reported (i.e. output will show 'phase x of 5') whereas on LTR systems, given that phases 1 and 4 are always skipped, progress of only phases 2/3/5 is shown (i.e. output will show 'phase x of 3')
• The 'status' command shows a one time snapshot of FMIG progress:

• The 'watch' command shows rolling output as FMIG progresses through phases:

A set of registry keys are also used to track FMIG progress - these are captured in autosupports generated by the system and can also be displayed via DDSH:

ER: # reg show dynamic.art.fmig
LTR: # reg show state.fmig2

In both cases registry keys displayed are very similar and as a result only those from an LTR enabled system are shown here:

# reg show state.fmig2
state.fmig2_checked_inodes = 13 <=== TOTAL NUMBER OF FILES IN DDFS NAME SPACE CHECKED TO SEE IF THEY MATCH CRITERIA TO BE CANDIDATE
state.fmig2_end_time = 1470059610 <=== END TIME OF LAST RUN OF FMIG
state.fmig2_fcopy_byte_count = 11274289151 <=== PRE-COMP (LOGICAL) BYTES COPIED TO TARGET TIER THUS FAR
state.fmig2_fcopy_post_lc_size = 10944662855 <=== POST-COMP (PHYSICAL) BYTES COPIED TO TARGET TIER THUS FAR
...
state.fmig2_files_copied = 8 <=== FILES WHICH HAVE COMPLETED COPY
state.fmig2_files_failed_in_copy = 0 <=== FILES WHICH FAILED TO COPY
state.fmig2_files_failed_in_install = 0 <=== FILES WHICH COPIED/VERIFIED BUT FAILED TO INSTALL
state.fmig2_files_failed_in_verify = 0 <=== FILES WHICH COPIED BUT FAILED TO VERIFY
state.fmig2_files_installed = 4 <=== FILES WHICH HAVE COMPLETED COPY/VERIFY/INSTALL
state.fmig2_files_verified = 4 <=== FILES WHICH HAVE COMPLETED COPY/VERIFY
...
state.fmig2_is_running = 1 <=== FMIG IS RUNNING
state.fmig2_is_waiting = 0 <=== FMIG IS WAITING (FOR EXAMPLE FOR CLEAN TO COMPLETE)
state.fmig2_progress_current_phase = 2 <=== CURRENT PHASE (IN THIS CASE COPYING/SCANNING)
state.fmig2_progress_percent_complete = 92 <=== % COMPLETION OF CURRENT PHASE BASED ON FILES CHECKED VS TOTAL FILES IN NAME SPACE
state.fmig2_progress_phase_time = 1245 <=== SECONDS SPENT IN CURRENT PHASE THUS FAR
state.fmig2_progress_total_time = 1255 <=== SECONDS FMIG HAS BEEN RUNNING THUS FAR
state.fmig2_start_time = 1470061560 <=== START TIME OF CURRENT RUN OF FMIG
...
state.fmig2_total_inodes = 14 <=== TOTAL FILES TO CHECK IN NAMESPACE
...

Finally various messages are logged to DDFS log files (/ddr/var/log/debug/ddfs.info) to indicate progress of FMIG. Again output is very similar for ER and LTR functionality so only those from an LTR enabled system are shown here:

08/01 15:26:00.945 (tid 0x7ff4c683e090): NOTICE: MSG-FMIG-00004: Data-movement started <=== FMIG STARTS
...
08/01 15:26:00.977 (tid 0x7ff4cbeabcf0): INFO: Fmig: phase 1 (seeding) started <=== BULK SEEDING STARTS (DOES NOT RUN ON LTR)
08/01 15:26:10.978 (tid 0x7ff4cbeabcf0): INFO: Fmig: phase 1 (seeding) ended
08/01 15:26:10.978 (tid 0x7ff4cbeabcf0): INFO: Fmig: phase 2 (scanning) started <=== SCANNING PHASE STARTS
 
08/01 15:26:22.162 (tid 0x4ca2150): Fmig: candidate file /data/col1/jftest/file2, src file snap_fh 10:0:12:0:e8b645a6:57989b6e:0 <=== CANDIDATE FILE PICKED UP BY COPY THREAD
08/01 15:26:22.164 (tid 0x7ff4c5214670): Fmig: candidate file /data/col1/jftest/file3, src file snap_fh 10:0:13:0:e9b64736:57989b6e:0 <=== CANDIDATE FILE STARTS COPY
08/01 15:26:22.273 (tid 0x7ff7d252a9e0): Fmig: candidate file /data/col1/jftest/file4, src file snap_fh 10:0:14:0:e6b64286:57989b6e:0 <=== CANDIDATE FILE STARTS COPY
08/01 15:26:22.274 (tid 0x7ff7d090a630): Fmig: candidate file /data/col1/jftest/file5, src file snap_fh 10:0:15:0:e7b64416:57989b6e:0 <=== CANDIDATE FILE STARTS COPY
 
08/01 15:27:11.072 (tid 0x7ff45ad95d30): Fmig: fmig_dump_stats ( cloud): phase 2, phase_time 61, total_time 71, total_inodes 14, checked_inodes 5, copied_files 0, verified_files 0, installed_files 0, failed_files_in_copy 0, failed_files_in_verify 0, failed_files_in_install 0, copied_vbytes 0, in-progress vbytes 1936986672 <=== STATS DUMPED PERIODICALLY
 
08/01 15:45:59.040 (tid 0x7ff7d25298c0): Fmig: fmig verified file5, src file snap_fh 10:0:15:0:e7b64416:57989b6e:0 <=== COPIED FILE COMPLETES VERIFICATION
08/01 15:45:59.140 (tid 0x7ff4c5218310): Fmig: fmig verified file3, src file snap_fh 10:0:13:0:e9b64736:57989b6e:0 <=== COPIED FILE COMPLETES VERIFICATION
08/01 15:45:59.225 (tid 0x7ff4c540b5f0): Fmig: fmig verified file2, src file snap_fh 10:0:12:0:e8b645a6:57989b6e:0 <=== COPIED FILE COMPLETES VERIFICATION
08/01 15:45:59.228 (tid 0x7ff4e591bcd0): Fmig: fmig verified file4, src file snap_fh 10:0:14:0:e6b64286:57989b6e:0 <=== COPIED FILE COMPLETES VERIFICATION
 
08/01 15:47:15.113 (tid 0x7ff45ad95d30): Fmig: fmig_dump_stats ( cloud): phase 2, phase_time 1265, total_time 1275, total_inodes 14, checked_inodes 13, copied_files 8, verified_files 4, installed_files 4, failed_files_in_copy 0, failed_files_in_verify 0, failed_files_in_install 0, copied_vbytes 10737418240, in-progress vbytes 536870911 <=== MORE PERIODIC STATS
 
08/01 15:47:41.858 (tid 0x7ff4cbeabcf0): INFO: Fmig: phase 2 (scanning) ended <=== SCANNING PHASE COMPLETES (ALL CANDIDATE FILES COPIED)
08/01 15:47:41.858 (tid 0x7ff4cbeabcf0): INFO: Fmig: phase 3 (verify) started <=== VERIFYING PHASE STARTS
 
08/01 15:47:57.855 (tid 0x7ff50351a620): Fmig: fmig verified file10, src file snap_fh 10:0:1a:0:64f703c6:57989b6e:0 <=== REMAINING FILES VERIFIED
08/01 15:47:57.875 (tid 0x7ff4cbeabcf0): INFO: Fmig: phase 4 (packing) ended <=== PACKING PHASE (DOES NOT RUN ON LTR)
08/01 15:47:57.875 (tid 0x7ff4cbeabcf0): INFO: Fmig: phase 5 (install) started <=== INSTALL PHASE STARTS
08/01 15:48:08.890 (tid 0x7ff4cbeabcf0): INFO: Fmig: phase 5 [install] ended <=== INSTALL PHASE ENDS
 
08/01 15:48:35.337 (tid 0x7ff4cbeabcf0): NOTICE: MSG-FMIG-00001: Data-movement completed (cloud) on cloud cps: 0:22:35 elapsed, 9 of files migrated, 11274289152 of total bytes migrated <=== FMIG COMPLETES
08/01 15:48:35.337 (tid 0x7ff4cbeabcf0): INFO: Fmig:  Phase[1]: 10 secs Phase[2]: 1291 secs Phase[3]: 16 secs Phase[4]: 0 secs Phase[5]: 11 secs <=== PHASE TIMINGS SHOWN

Note that:

• Figures indicating the amount of post-comp (physical) data copied to the target tier are only updated each time a file finishes copying. As a result if a single large file were being migrated post-comp will show 0 bytes until the copy (and scanning phase) is complete
• Figures indicating the amount of pre-comp (logical) data copied to the target tier are updated periodically and not only on completion of file copy
• The % complete shown in the scanning phase does not indicate the total % of candidate data copied thus far. Instead it indicates the % of total files in the DDFS name space have been checked thus far by copy threads to determine whether they are candidates for migration. As a result when FMIG is in the scanning phase it is common to see: