Highlighted
yourssubash
1 Nickel

Space usage in Data Collection has exceeded 90% or 100% threshold

After checking few Kb's, I found these scenario's as common steps to check on DD putty:

1) Steps and scenarios: If found huge snapshots:

# df in sysadmin

File Distribution. Also check here if there are snapshots on the system ( # snapshot list mtree * (this will show the list of snapshots in a particular Mtree). If snapshots are available: in CLI, then run and check for all Mtrees the following (/data/col1/backup)

# mtree list

# Snapshot Summary for mtree <mtree name full path> e.g., # Snapshot Summary for mtree /data/col1/xxxxxxxxxxxyyyy

# replication show config

# fi st for Cleaning started at as well phase.

# fi clean watch

Inform customer to expire snapshots from GUI OR from command line which are no longer needed as well retain snapshots that are in use. If needed take help from specialist team(s).

Expired snapshots will be cleared in next cleaning cycle. Once done, check the output of # df and #fi st

Note: If Customer is on DDOS is at 5.4.0.8 there are couple of known issues.

1) Cleaning bug

2) Replication snapshots do not get expire.

Please plan to upgrade it to DDOS 5.4.2.2.

2) Steps and scenarios: Cleaning is in progress. Alert with 90% threshold

Investigate the common factors that might affect claiming space:-

•          Replication lag

•          Stale snapshots

•          Any abnormal small files count

•          Compression rate changes

Pre-Comp = Data written before compression

Post-Comp = Storage used after compression

Global-Comp Factor = Pre-Comp / (Size after de-dupe)

Local-Comp Factor = (Size after de-dupe) / Post-Comp

Total-Comp Factor = Pre-Comp / Post-Comp

Reduction % = ((Pre-Comp - Post-Comp) / Pre-Comp) * 100

/data/col1/backup

# df in sysadmin

# filesys clean show schedule

# filesys clean status

#filesys clean watch

# nfs show active

If filesys clean status is already running and in last phases, then keep the case in observation for the cleaning to complete and check for available space (# filesys show space)

How To Determine Compression Rates: https://support.emc.com/kb/306103

3) Steps and scenarios: Expired snapshots not getting removed even after FS cleaning: Alert with 90% threshold

This would be a possible cause of snapshots with Soft lock. Check these steps to isolate the same:

# df in sysadmin

File Distribution log. Also check here if there are snapshots on the system ( # snapshot list mtree * (this will show the list of snapshots in a particular Mtree). If snapshots are available: in CLI, then run and check for all Mtrees the following (/data/col1/backup)

# mtree list

# replication show config

Run the below command to list all the Snapshots of the MTree on both Source and Destination: 

# replication status <mtree>

#snapshot list mtree /data/col1/<abc>

# Snapshot Summary for mtree <mtree name full path>  e.g., #  snapshot list mtree /data/col1/avamar-48645863863

If found the "dm_rmsnapshot" in the ddfs.info* files, then we need to perform the below steps which should release the softlocks and the snapshot should be removed during the next cleaning:

1) Break replication on both source and destination (# replication break)

2) Resync replication on the source. (# replication sync)

Run the Global Cleaning

# filesys clean start

• Once the GC finishes please make sure the Snapshots are deleted.

#snapshot list mtree </data/col1/mtree-name>

How to collect sfs_dump output for specialist team log analysis:

Login to the DDR as "sysadmin" user and execute the following commands:

1. #system show serial

2. # priv set se

3. Collect sfs_dump: # se sfs_dump -h

4. This will take time and dump a lot of data to the screen, but it will be captured to the log file.

5. Once it finishes disable putty log.

6. Run the following command to get back to admin mode: # priv set admin

7. Close putty session

8. Compress sfsdump1_<hostname>.out files

9. Upload them using support portal to this SR or a temp FTP if needed.

These are my initial observations. Kindly share other scenarios and steps to include in this document and for every ones reference.

Thank you for reading and welcome your comments ...

0 Kudos