Version 7.1 of OneFS includes numerous enhancements that will improve the performance of Isilon cluster, the SmartDedupe is one of them. The SmartDedupe software module enables deduplication to save storage space on a cluster by reducing redundant data. This article will introduce SmartDedupe software module and demonstrate how to configure deduplication on the Isilon.

Deduplication is applied at the subdirectory level and targets all files and directories underneath one or more root directories. As you write file to the cluster, some of those files or blocks of data in the files might be duplicates. You can run a deduplication job that scans the file system to see if the data already exits. After duplicate blocks are discovered, SmartDedupe moves a single copy of those blocks to a special set of files known as shadow stores. During this process, duplicate blocks are removed from the actual files and replaced with pointers to the shadow stores.

With post-process deduplication, new data is first stored on the storage device and then a subsequent process analyzes the data looking for commonality. This means that initial file write or modify performance is not impacted because no additional computation is required in the write path.

The deduplication job is set to run at low priority by default, so impact to your workflow should be minimal. However, it’s a good idea to wait until users have finished modifying their files on the cluster before you run the job.

You can perform the following deduplication tasks from the CLI to configure SmartDedupe assessment and start Deduplication.

Step 1: Log into Isilon cluster.

Step 2: Configure SmartDedupe Assessment Directory.

The assessment directory is /ifs/Demo_Data/Data/”Legal Discovery” in this example, so I type isi dedupe settings modify –assess-parths/ifs/Demo_Data/Data/”Legal Discovery” and press Enter. This command is used to configure which directory SmartDedupe will perform an assessment on.

Step 3: Start Deduplication assessment job.

Type isi job jobs start dedupeassessmentand then press Enter. This command will start the DedupeAssessment job.

Step 4: View active jobs.

Type isi job jobs list and then press Enter. This command displays information about active jobs. Wait for the DedupeAssessment job to finish. To refresh re-run the above command. When it is finished, it will no longer display when you issue the isi job jobs list command.

Step 5: view Deduplication assessment report.

Type isi dedupe reports view (job ID from step 4) and then press Enter. This command will display the deduplication report. What is the Dedupe percent? Dedupepercent is the percentage of scanned blocks that would be deduplicated.

Step 6: Specify directory to performance Deduplication on.

Type isi dedupe settings modify –paths /ifs/Demo_Data/Data/”Legal Discovery”and then press Enter. This command is used to configure which directory SmartDedupe will deduplicate.

Step 7: Start Deduplication job

Type isi job jobs start dedupeand then press Enter. This command will start the dedupe job. This will help to deduplicate the directory.

Step 8: view active jobs.

Type isi job jobs list and then press Enter.

Step 9: view Deduplication report.

Type isi dedupe reports view (Job ID from setp 9) and then press Enter. Dedupepercent is the percentage of scanned blocks that were deduplicated.

Step 10: view disk space savings.

Type isi dedupe statsand then press Enter. This command will allow you to view the amount of disk space that you are currently saving with deduplication. Observe the estimated physical saving. This is the total amount of physical disk space saved by deduplication, including protection overhead and metadata.

                                                                                                                                              

Author: Jeffey Liu