How to list and delete orphaned backup data present on Datadomain.

I have a case open on this version of Data Domain, and using Networker instead of Avamar. with the exact same symptoms.

Thus far they have determined that there is a failure in communication between the DDBoost devices/cleaning process and the backup server. Essentially the backup server isn't telling DDBoost which savesets can be cleaned.

They are having me remove them from the Backup server side. Open a case with EMC and if you like reference my case PM me for the case number.

can we list the orphaned backups?

I have not found anyway to do so on the data domain. If it works the way I think it does, Networker maintains the index of savesets and associated files. DDBoost maintains the files, a list of those files, and the pointers.

I don't know that Data Domain even knows about the saveset numbers, or the host names.

You need to go to the Bash shell (Shell escape ) mode then need to mount the Mtree and find out the files which are older then retention period ( 30 days in this case ) and delete those files .

As you mentioned you use DDboost to do the backup for Oracle DB.

This is likely to be an Oracle/RMAN issue, not a plugin issue.The plugin never deletes a file on its own;  it deletes a file only in response to a specific request from Oracle/RMAN, so if files are not getting deleted it would be because the plugin is not being asked to delete them.

I would recommend to run the purge RMAN script with TRACE 5 .

you don't need to drop to shell to delete data backed via RMAN DDBoost, simply export that direcotry to a Linux box via NFS and work from there.

Hi

I want ask you about more details

What does it mean?  "purge RMAN script with TRACE 5"

could you put more details for it.

Hi everyone,

I wanted to provide some quick points around the issue being discussed here. Firstly how file removal e.t.c. works when you store data on a Data Domain system:

- Essentially the DDR is a dumb piece of storage which happens to offer functionality such as de-duplication and so on

- The DDR itself provides no 'management' of the files/data which it receives

- As a result it is the responsibility of the backup application to track which files are sat on the back end DDR and then delete those files once they are no longer required (i.e. once the backups have expired)

- If the backup application does not do this (and therefore leaves orphaned files on the DDR) these files will continue to exist indefinitely using space on the DDR

- As a result if you have a DDR holding orphaned data this will be because the backup application is not working as expected (either due to misconfiguration or a defect)

If you need to look at files to work out what data is orphaned there are many ways of doing this. As mentioned above you can NFS/CIFS share the particular mtree/directory you are interested in then browse this from some other client on the network. In addition to this you can generate a report of all files in a specific location from the DD CLI using the 'sfs_dump' command. For example:

- Log into the DD CLI

- Drop to 'se' mode:

# system show serialno

[system serial number displayed]

# priv set se

[password prompt - enter system serial number from above]

Note that systems configured with encryption and/or retention lock may prompt for credentials of a user with 'securty' role at this point

- Run 'sfs_dump -h [path you are interested in]:

# se sfs_dump -h /data/col1/[mtree name]/[directory]

Running this against a single file, as an example, produces the following output:

SE@ddve-3-ltr## se sfs_dump -h /data/col1/jftest/testfile1

/data/col1/jftest/testfile1: mtime: 1367259786000000000 fileid: 37 size: 1073741824 type: 9 seg_bytes: 1077336236 seg_count: 128226 redun_seg_count: 0 (0%) pre_lc_size: 1077336236 post_lc_size: 1094633088 (102%) mode: 02000100644

export:/data/col1/jftest/testfile1-segs:128226

export:/data/col1/jftest/testfile1-redun-segs:0

We can ignore the lines starting 'export' and just concentrate on the first line of output. Useful things on this line are:

File name (first entry)

mtime: Date/time of last modification of the file

size: Pre-compressed (i.e. original/rehydrated) size of the file in bytes

pre_lc_size: Size of the file after de-duplication in bytes

post_lc_size: Size of the file after de-duplication and compression in bytes

Note that:

- The above file actually increases in size after de-duplication and compression as it contains purely random data (i.e. its not representative of real backup data)

- The post de-dupe/de-dupe and compression sizes are generated at the time the file is originally ingested and are not updated after this point. As a result they should not be relied upon to be 100% accurate (however they are generally good enough for trend analysis)

- The first 10 digits of the mtime can be taken and converted into a human readable date/time format - for example using the above on a linux system (note that I set my time zone to the same as being used on the DDR to make sure the date/time is correct):

fordj6@ubuntu-jamesf:~$ export TZ=Europe/London

fordj6@ubuntu-jamesf:~$ perl -e "print scalar localtime (1367259786)"

Mon Apr 29 19:23:06 2013

Note that if the sfs_dump command is given a directory name it will dump details of all files in all subdirectories. Therefore to dump the details of all files in a specific mtree you would use:

# se sfs_dump -h /data/col1/[mtree name]

To dump the details of all files on a DDR you would use:

# se sfs_dump -h /data/col1

Obviously most systems will hold many thousands of files so you may wish to enable logging in your terminal session, run the sfs_dump command, then use the contents of the log for offline analysis. Knowing what each of the fields means (as described above) means you can put together scripts to analyse the sfs_dump output or import somewhere else such as Excel.

Note also that Data Domain support have a variety of scripts which can take this output and use it to generate various reports/convert it as necessary.

Once you have identified the orphaned files Data Domain recommends removing these files somehow via the backup application. This is by far the best way to do this as whilst the files can be deleted directly on the DDR (or via NFS/CIFS) this can lead to inconsistencies between the backup applications catalogue/media database which causes issues further down the line.

If you have no option but to remove the files directly from the DDR Data Domain support can help with this however you will be required to:

- Open an SR

- Provide a list of all files which need to be removed (note that it is the customers responsibility to ensure this list is correct)

- Sign a waiver form before the deletions can physically be performed (this is a legal requirement which must be met before support can proceed)

Some of the information discussed here is also available in the following article: https://support.emc.com/kb/487055

Thanks, James

Hi James, your explanation is always vivid and informative every time. It is serving as a reference while troubleshooting customer issues. Kudos to your subject matter expertise ..