SmartDedupe and SmartQuotas questions

If your case, would you not just want to have a policy that has the path of /ifs/clustername?

As far as quotas, the point is that deduplication is not taken into account with respect to quotas.  This is a good thing because trying to do so would be weird.  Let's look at a simple case:

5 users have the same PPT file in their respective home directories and the file is 10M.  So that takes up 50M in the filesystem by default.  The SmartDedup job comes in and reduces the real consumed space on disk to 10M.  But if each user has a quota, each user will be charged 50M to their respective quotas.

Trying to do otherwise could get really complicated.  If there is only one real copy, who gets charged for it and how do you decide?  This way, the user sees the space consumed that the filesystem tells him he has consumed.  But you, as the admin, get the space savings.  Exposing deduplication to the end user is just wrought with trouble and confusion. You can run a report and show the customer how much space you are saving, but we don't want the user to see that.  The numbers just wouldn't add up and there's always the question of where to charge the unique block.

I hope this makes sense.

AdamFox wrote: If your case, would you not just want to have a policy that has the path of /ifs/clustername?

i will have customers with data that will have horrible for dedupe (media files, DNA sequencer runs ..etc).

Dedupe and Quotas - here is where the problem is. Back when my customers were on Celerra, i could enable dedupe per file system and customers would immediately see the savings because file system utilization when down. I would like to be able to show/provide dedupe savings to my customers.  I understand it's very hard to determine who is paying for that first block, we are struggling with charge/show back on other dedupe enabled platforms like Avamar/Data Domain. My customer do not care how much I save, they care how much they save. While i could take overall cluster savings and maybe lower the price per gigabyte, i can't display benefits for customers with good dedupe data versus others with bad dedupe data.

And here lies a difference between having multiple filesystems and having a single filesystem.  When you dedup on the Celerra, you are deduping over an individual filesystem.  The good news about that is you can easily report on savings at the filesystem level.  The bad news is that your deduplication domain is very limited, in the case of the Celerra today to a 16TB filesystem.  Contrast that with what OneFS can do.  It's one filesystem so you can have a very large deduplication domain, 10s of PB if you want to.  But you lose some of the granularity of having isolated filesystems. 

So if you had a tree quota and you are deduplicating across mutliple tree quotas, where do you charge the space?  Same problems as the user.

It's possible that some of these recommended limits will be raised.  If you want that, you need to file RFEs with support.  The more they get, the more attention feature requests get. 

But there will be some features that won't translate well between a multi-filesystem array and an single file system array and vice versa.  This one (lots of directories in the policy or perhaps multiple policies) may be on that can be improved.  For others, it may be more difficult or impossible.  I can't necessarily say which is which as I'm not in OneFS Engineering.  Again, it's not that one is better than the other, but that when you make architectural choices, you make compromises.

In your particular case, notice is says "Avoid".  That is most likely a recommendation.  In some cases, it may not be bad to have more than 10, but the point is that in some cases it will so they have the guideline.  I'm sure the more over 10  you are, the better chance of issues.

You also have the option of reorganizing your directory structure so that the deduplable data lives in one place and the non-dedupable stuff lives in a separate tree.  That works in the cases where you know the data up-front.  If you don't, then that may be tougher to implement.  I'm not saying this is a fix, but for some environments, it may be a reasonable work-around for some.

1) Although the SmartDedupe best practice recommends specifying a maximum of ten directory paths for dedupe configuration, this is not a hard limit and can be exceeded if necessary. Wouldn't suggest going much above the thirty directory paths you mention though.

2) OneFS will ignore deduplication for quota accounting purposes.

AdamFox wrote: You also have the option of reorganizing your directory structure so that the deduplable data lives in one place and the non-dedupable stuff lives in a separate tree.  That works in the cases where you know the data up-front.  If you don't, then that may be tougher to implement.  I'm not saying this is a fix, but for some environments, it may be a reasonable work-around for some.

Thanks Adam. Unfortunatelly reorg of the structure is not going to be easy. Cluster is already devided into "logical" containers by zone name, i have some zones names that are being audited by Varonis and some that are not. I also like to separate my NFS clients from my CIFS clients so there was already some thought put into the structure that will be very hard to manupulate at this point.

Rob and Adam,

Three follow up questions:

1)  Please help me better understand how the dedupe job handles that list of directories that need to be scanned. Let's say i have 5 directories specified in my list to be deduped:

/ifs/clustername/customerA

/ifs/clustername/customerB

/ifs/clustername/customerC

/ifs/clustername/customerD

/ifs/clustername/customerE

When the job runs ( i have mine scheduled to run once a month), will it try to scan 5 directories at the same time or will it start and run to completion (multiple phases) with customerA, then customerB and so on ?

2) Rob i think i see where you are going with this, you want me to cycle through the list of directories that get scanned So let's say i have 50 directories that i would like to scan. I would use the cli to configure 10 directories and kick of the job. Once that completes i would remove those 10 directories and add another set of 10 and re-run the job. Is that were you were going ?

3) Let's say i have a specific directory and the customer says " after dedupe my appliation response time has increased, please disable dedupe and re-hydrate my directory". How would i do that ?

Thank you

Although the dedupe job is split into multiple phases (sampling, detection, sharing, and index update), multiple configured directory paths are not segregated into distinct phases within the job.

Regarding un-dedupe/redupe/re-hydration, there is no customer facing way of achieving this. However, if necessary, Isilon support can assist with re-hydrating a previously deduplicated dataset.

so you are saying the job will scanning all the configured directories in parallel not serially ?

Why no customer facing way to disable dedupe/enable re-hydration and when is it going to be available via the WebUI/CLI ?  What is involved in re-hydration and are there any caviats in terms of performance, space utilization. Gimme the whole scoop please

The 'whole scoop' is best seen by reading pages 470-406 of the OneFS CLI Admin Guide (citing the 7.1.1 version).  It gives a great overview of OneFS deduplication, how it works, what system resources are involved, etc.

As Nick said, rehydration within a given cluster (aka undedupe) is possible but requires a service engagement.  Why no customer-facing way, you ask?  Good question - and the answer is simple - because there is no benefit to ever perform rehydration under normal circumstances.  None.  Zero, zip, nada.

If you really must use files in a rehydrated directory, you can SIQ it to another cluster or copy it to another (undeduped) directory within the same cluster.  Simple.  But useless.  No benefit, and in fact a detriment (you consume more space with no performance gain).  Your sales guy might like that - consume more space, buy more nodes - but it entirely defeats the purpose of the feature.

An 8K block is an 8K block is an 8K block - whether it's unique or shared - that block is persisted, pointed to by inode(s) and thus can be used.  'Rehydration' is really a useless exercise.

It is only under the rarest of abnormal circumstances that a given directory, once deduped, requires rehydration, and for those circumstances you want to engage service.  After all, this is part of why customers pay for service and maintenance...use it.

HTH

Rob

Rob,

everything is always wonderful and beautiful, rainbows and unicorns when it comes to Isilon. Right?  Well it's not when i can't perform online upgrades without sufferring outages, it's not when i go patch after patch and lwiod keeps dying on me.  So when you say i will never have to do something or i will never encounter this issue, i take it with a grain of salt. Yes, you visit many customers, you see a lot of environments. I am going off what i see in my environment, running on supported hardware, supported OneFS versions.

Copying data to another "un-deduped" directory is a waste, i need to re-hydrate in place. Every change that is made in my environment has to go through change control, every change control instance requires a backout plan. Copying data to another location and scheduled downtime for my application is not an acceptable backout plan for me.

Dynamox

there is an Undedupe job type, at least in 7.1.1.2.

It's not advertised in isi job types list but mentioned in man isi-job

where the --path parameter for isi job start is explained.

Not sure wether Rob was suggesting to engage Professional Service

or simple contact support for the procedure (like for removing audit logs).

hth

-- Peter

Thank you Peter, i am still on 7.1.0.6 at the moment ..will need to investigate it a little further. I did talk to support about "undedupe" functionality. I hope this undedupe job is better than the procedure i got from support (very cumbersome)

So i have this dedupe job running for almost 3 weeks now (still running). How much was it able to reduce my cluster utilization, by 51TB or by 109TB ?

Thank you

Thank you Jeffey. What does "51 TB from 160 TB configured for deduplication" mean ?