deduplication questions: single instance only, complete scan force and fs_dedup figures update

Bound to my questions above, the document Achieving Storage Efficiency through EMC Celerra Data Deduplication (h6065-achieve-storage-effficiency-celerra-dedup-wp.pdf) mention the following on page 7:

"Note that Celerra detects non-compressible files and stores them in their original form.
However, these files can still benefit from file-level deduplication."

This would be great, but I don't feel it's actually working like this...

Can someone confirm those features are implemented ? I'm running DART 5.6.47.11.

Thanks

Eric

Hi Eric,

should work.

Just to be clear - it means that even if a file isnt compressible it is still a candidate for single-instancing - i.e. if multiple files with the same content exist in this file system only one instance gets stored - the others are represented by 8k stubs

Rainer

Hi,

I have the same understanding of the statements I quoted from this doc.

It would be a good point if it worked like this, but I'm not as confident as you.

From file size information taken from right-click properties in windows explorer on a jpg file.

Size: 91659 bytes

Size on disk: 98304 bytes

91659 / 8192 = 11.18

So this files needs 12 blocks

12 * 8k = 96ko (98304 bytes)

After compression

Size on disk: 106496 bytes

It means it now occupies 13 blocks

It guess this single extra block has been accounted because of the stub.

Can you confirm ?

This file is unique so there is no reason for me to have this stub mecanism in place for this file.

Is it mandatory for any file evaluated for deduplication to have a stub file ? even though the file is finally not compressible and the file is unique.

Sound weird to me...

Is it the way it behaves ?

Let's take another example: another jpg file, but a big one this time.

The size on disk raised from 103 blocks to 107 blocks. Even if one block is due to the stub, it means it needs 3 extra blocks to store the file in it's compressed form.

This is not surprising that the file is bigger in it's compressed form. A compression algorithme aimed for speed rather than killing compression ratio (I guess) will make a jpg file bigger when compressed because of it's compression metadata.

If my hypothesis is right in that example, why is DART not considering that this file is not a good candidate for compression ?

Thanks,

Eric

The doc says there is an additional stub (therefore 2 stubs for one file)

You can also specify a list of extensions not to be deduped

Claude

I haven't seen dedup doing any real selection based on how well file gets compressed. But it doesn't really matter. For normal mixed shit fs one gains 20-30% of space. If you are not using single instancing then I think you should exclude stuff like jpeg.

Also fs_dedupe -info seems to lie. For example I have deduped filesystem (2,5tb) of which fs_dedupe says 3,7tb is original size but actually it is 3,2tb. My guess is that it counts all the files that have been deduped and never minuses size if file(s) gets deleted.

There's no option to use just single instancing.

We've been using dedupe about as long it has existed. Backups are a bit slower. Otherwise it's been working nicely and transparently for users. Now we're archiving and getting rid of dedupe in tier 1. Tier 2 gets deduped. Main reason for reduplication is to move to vbb backups (with ndmp tape drives). In our environment it is only way to backup in reasonable time window. And one can't do file level restores from deduped fs backed up with vbb.

--

Jussi 

Jussi,

Thanks for your experience feedback, including backup xp.

As I already said, the problem is not only jpeg files or pictures files in general.

I'm dealing with files of very different types. If EMC provided a tool to collect and report dedup efficiency per file type (based on extension) for each FS, maybe I could rely on this to update the blacklist... but once again, it's a pity because it would also exclude those files from single instancing...

I disagree with your interpretation of fs_dedup -i (or did I misunderstood your remark ?).

fs_dedup report correctly savings from compression and single instancing:

On the same dataset:

With compression only (Duplicate Detection Method      = off)

As of the last file system scan (Thu Jan  6 17:55:18 CET 2011):
   Files scanned                   = 13680
   Files deduped                   = 13665 (100% of total files)
   File system capacity            = 54160 MB
   Original data size              = 27462 MB (51% of current file system capacity)
   Space saved                     = 14410 MB (52% of original data size)

With compression + single instance (Duplicate Detection Method      = sha1)

As of the last file system scan (Fri Jan  7 09:44:37 CET 2011):
   Files scanned                   = 13680
   Files deduped                   = 13665 (100% of total files)
   File system capacity            = 54160 MB
   Original data size              = 27462 MB (51% of current file system capacity)
   Space saved                     = 19386 MB (71% of original data size)

It this example, files removing has been accounted by fs_dedupe.

To me, fs_dedup is lying for another reason: it reports 0 savings when actually running dedup have had a negative effect because the dataset was not a good candidate (inodes overhead, compression overhead).

Eric

Just try it.

If you know filesystem contains mostly music, videos, compressed files and that sort of stuff forget the dedupe. If it's "normal" mixed stuff you'll gain something like 20-30%. First dedupe round for millions of files takes few days. After that you'll see what's the outcome. If it's not good you can always redupe the fs (you'll have impact on checkpoint storage and backups on both operations).

We've gained terabytes with dedupe so it's been good to us.

--

Jussi