NetWorker: How to force expired save set removal
Summary: While expired save sets removal from Disk devices is an automated procedure, some situations require to force the removal of expired save sets. This article shows how to enforce this procedure to run, and the show stoppers that may prohibit the removal of expired save sets. ...
Instructions
E) is preferred for normal situations where space reclaim is required but the database integrity is not in question. Incorrect usage of this information can lead to data loss. Dell support cannot advise on which data should be deleted or delete any data from an environment. This NetWorker administrator must identify, validate, and perform the deletion with the approval of the sites storage and infrastructure management.
Prerequisites:
The following steps must be performed before proceeding with save set removal from disk devices. Do not skip any steps and do not proceed unless you are certain the data is no longer required.
- The volume containing the expired save sets must be mounted on a device in Read/write mode. You can cross-check that requirement by running the command:
nsrmm
nsrmm shows a list of all devices, and volumes mounted on those devices. Ensure that the volume that you are looking to clean up is mounted, and in read/write mode, check below sample output, the output lists:
<media type> <volume name> mounted on <Device name>, <mount mode>
[root@srvr ~] nsrmm Data Domain disk DDBoost.002 mounted on rd=SN1:boostdevARC, write enabled Data Domain disk DDBoostDaily.001 mounted on rd=SN1:boostdev01, write enabled Data Domain disk IndexPool.005 mounted on Index, write enabled Data Domain disk DDBoost.006 mounted on rd=SN1:boostARC, write enabled Data Domain disk DDBoost.024 mounted on rd=SN1:boostdev11, write enabled Data Domain disk DDBoost.022 mounted on rd=SN1:boostdev09, write enabled Data Domain disk DDBoost.006 mounted on rd=SN1:boostdev05, write enabled Data Domain disk DDBoost.037 mounted on rd=SN1:boostdev2, write enabled Data Domain disk DDBoost.020 mounted on rd=SN1:boostdev06, write enabled Data Domain disk CDDBoost.002 mounted on Cboost02, write enabled Data Domain disk CDDBoost.001 mounted on Cboost01, write enabled Data Domain disk DDBoost.028 mounted on rd=SN1:boostdev12, write enabled Data Domain disk VDDBoost.006 mounted on rd=SN1:vboostdev2, write enabled Data Domain disk DDBoost.037 mounted on boostdev2, write enabled
- The volumes containing expired save sets must not be set to
read-onlystate orscan-neededstate. The way to verify this point is to list the volume attributes using themminfoquery:mmifo -mvV -q "volume=<volume name>"
Or:mminfo -mvV -q "type=Data Domain,type=adv_file"
[root@srvr ~]# mminfo -mvV -q "type=Data Domain,type=adv_file" state volume written (%) expires read mounts capacity volid next type flags M CBoost.001 120 TB full manual 0 KB 10 0 KB 2932002210 0 Data Domain CBoost.002 114 TB full 01/19/2038 0 KB 11 0 KB 4254375504 0 Data Domain CBoostARC.001 0 KB 0% undef 0 KB 6 0 KB 3767837147 0 Data Domain CBoostw.001 12 TB full 01/19/2038 0 KB 9 0 KB 4237598388 0 Data Domain CBoostw.002 15 TB full 01/19/2038 0 KB 9 0 KB 4204044079 0 Data Domain CvBoostw.001 14 TB 100% 09/17/2016 0 KB 10 0 KB 3818168663 0 Data Domain CvBoostw.002 14 TB 100% 09/17/2016 0 KB 11 0 KB 3801391451 0 Data Domain Cindex.001 3726 GB 100% 01/19/2038 0 KB 6 0 KB 3784614359 0 Data Domain Boost.001 13 TB 100% 12/07/2016 15 TB 22 0 KB 2294641726 0 Data Domain Boost.004 13 TB 100% 12/07/2016 7880 GB 91 0 KB 1707440518 0 Data Domain Boost.006 13 TB 100% 12/07/2016 15 TB 61 0 KB 2059761217 0 Data Domain Boost.020 13 TB 100% 12/07/2016 13 TB 23 0 KB 1623554894 0 Data Domain Boost.021 14 TB 100% 12/06/2016 15 TB 21 0 KB 1640332083 0 Data Domain Boost.022 13 TB full 12/07/2016 11 TB 21 0 KB 1590000634 0 Data Domain Boost.023 14 TB 100% 12/06/2016 14 TB 22 0 KB 1606777826 0 Data Domain Boost.024 13 TB 100% 12/07/2016 11 TB 22 0 KB 1808103225 0 Data Domain Boost.028 15 TB 100% 12/07/2016 14 TB 26 0 KB 2244310451 0 Data Domain Boost.029 79 TB 100% 11/30/2016 221 TB 28 0 KB 4229919 0 Data Domain Boost.030 86 TB 100% 11/30/2016 212 TB 40 0 KB 4282420003 0 Data Domain Boost.031 74 TB 100% 11/30/2016 203 TB 33 0 KB 4265642791 0 Data Domain Boost.032 83 TB 100% 11/30/2016 224 TB 32 0 KB 4248865579 0 Data Domain Boost.033 87 TB 100% 11/30/2016 222 TB 38 0 KB 4232088367 0 Data Domain Boost.034 81 TB 100% 11/28/2016 214 TB 31 0 KB 4215311155 0 Data Domain Boost.035 83 TB 100% 11/30/2016 229 TB 31 0 KB 4198533944 0 Data Domain Boost.036 100 TB 100% 11/30/2016 232 TB 30 0 KB 4181756734 0 Data Domain Boost.037 13 TB 100% 12/07/2016 10 TB 2 0 KB 2277864636 0 Data Domain BoostARC.002 586 GB 100% 09/21/2016 516 GB 18 0 KB 1673886395 0 Data Domain BoostARC.005 801 GB 100% 09/13/2016 139 GB 35 0 KB 2806030958 0 Data Domain BoostARC.006 605 GB 100% 09/21/2016 183 GB 0 0 KB 3032861639 0 Data Domain BoostCLN.001 13 TB 100% 01/19/2038 52 GB 52 0 KB 2529052554 0 Data Domain BoostCLN.002 12 TB 100% 02/10/2039 16 GB 52 0 KB 2512275346 0 Data Domain BoostW.005 107 KB 100% undef 5100 GB 31 0 KB 3594048161 0 Data Domain r BoostW.006 111 KB 100% undef 5175 GB 36 0 KB 2537117179 0 Data Domain r BoostW.011 1261 GB 100% 10/02/2016 1261 GB 21 0 KB 1774548991 0 Data Domain BoostW.012 1206 GB 100% 10/03/2016 1021 GB 21 0 KB 1757771848 0 Data Domain BoostW.015 14 TB 100% 11/30/2016 35 TB 32 0 KB 2789253746 0 Data Domain BoostW.016 17 TB 100% 11/30/2016 28 TB 31 0 KB 2755699318 0 Data Domain BoostWCLN.001 314 GB 100% 02/03/2039 0 KB 53 0 KB 2461943727 0 Data Domain BoostWCLN.002 993 GB 100% 02/03/2039 0 KB 55 0 KB 2445166522 0 Data Domain HVBoost.005 26 TB 100% 09/20/2016 282 GB 30 0 KB 2705367674 0 Data Domain HVBoost.006 25 TB 100% 09/20/2016 408 GB 26 0 KB 2671813247 0 Data Domain IndexC.001 83 GB 100% 01/19/2038 4361 KB 72 0 KB 2428389314 0 Data Domain S IndexPool.004 2358 GB 100% 07/13/2017 5163 GB 30 0 KB 4234285 0 Data Domain IndexPool.005 2500 GB 100% 12/07/2016 5196 GB 52 0 KB 4282424369 0 Data Domain IndexPool.006 2540 GB 100% 12/06/2016 5519 GB 32 0 KB 2621543382 0 Data Domain srvr.007 1024 MB 100% 06/20/2016 4510 GB 9 0 KB 3846074226 0 adv_file r [root@srvr ~]#
- Volumes with flag "
r" are read-only volumes, and thus the expired save sets of those volumes are not removed unless the volume is released from the Read-only state by running:nsrmm -o notreadonly <volume name> - Volumes with flag "
S" are Scan-needed volumes (this is often the result of a Disaster Recovery (nsrdr) operation) are not subject to Space recovery procedure because they must be resynchronized by:scanner -m <mounting device name>, then once thescanneroperation is done, you may remove the Flag by:nsrmm -o notscan <volume name>
- The volume should not be set to
Manualexpiration state, as marked in the below example:
[root@srvr ~]# mminfo -mvV -q "type=Data Domain,type=adv_file,manual"
state volume written (%) expires read mounts capacity volid next type flags
M CCR.008 80 TB full manual 0 KB 10 0 KB 2932458710 0 Data Domain
[root@srvr ~]#
Manual expiration state volumes maintain all expired save sets until those save set are deleted manually, and in order to reset that state to Automatic state: nsrmm -o notmanual <volume name>
- The volumes containing expired save sets must be "idle," this means no Read or Write operations running on that Volume.
- Perform a Server Protection backup and collect the bootstrap save set details:
mminfo -B - Export the existing media database into XDR format: How To Export the NetWorker Media Database Using nsrmmdbasm
Procedure:
- Collect
mminfooutput. Keep the output in a text file for verification:mminfo -mvV -q "type=Data Domain,type=adv_file" - Run the expiration process:
nsrim -X
nsrim -X performance may be impacted. You can limit the data recycler to specific volumes through: nsrim -X -V <volume name>
- Make a list of volumes having expired save sets
- Ensure that the server is idle; no backups, clones, or recoveries are running:
nsrwatch(must show zero sessions, ensure that no jobs are scheduled to run imminently) - For each disk volume containing expired save sets, run:
nsrstage -vvvv -C -V <volume name>
Verification:
The NetWorker server's daemon.raw shows NSRD logging the save sets deletion (for each save set) off each volume:
- Linux:
/nsr/logs/daemon.raw - Windows (Default):
C:\Program Files\EMC NetWorker\nsr\logs\daemon.raw
NetWorker: How to use nsr_render_log to render .raw log filesnsrd NSR info Media Info: Deleted 18 MB from save set 882684863 on volume CBoostARC.001
At the end of the space recovery procedure, the storage node process (nsrsnmd) logs the amount of space recovered for each device and volume. The NetWorker server process (nsrd) logs similar messaging:
nsrsnmd NSR info Recovered 624 MB from device DD4500_Index for volume (Index.005)nsrd NSR info Media Info: Recovered 624 MB by deleting 50 savesets from device DD4500_Index.
nsrsnmd messages are found in the remote Storage Node's daemon.raw.
Check the disk volume sizes and compare with the sizes reported before performing this procedure: mminfo -mvV -q "type=Data Domain,type=adv_file"