PowerScale OneFS 9.2.1.0 CLI Administration Guide

Usage

The isi_sfse_assess command scans a set of files and simulates the work that Small Files Storage Efficiency would do. This command generates an estimate of disk space savings without moving any data. It does not require a license and does not require that Small Files Storage Efficiency be enabled.

Use this tool before enabling Small Files Storage Efficiency to see possible storage savings. Use the tool after some file packing has occurred to identify additional possible savings given the current state of the file system.

The assessment is based on calculating the blocks saved when small files are packed into containers at the same protection level. A file is categorized as small if its size is less than the value of the max_size option. The default is about 1MB.

Many of the options in the isi_sfse_assess command mirror options available during actual packing. These are system level control options (sysctl options) with preset default values. For packing to achieve the results predicted during assessment, you must use the same settings for packing and assessment.

• You can change the default settings for sysctl options used during packing with the isi_packing command.
• You can change the default settings for sysctl options used during assessment with this isi_sfse_assess command.

The command reports progress as it runs by displaying the following information:

• % complete.
• Estimated possible space savings on the files scanned so far. This number should continually increases as the program progresses.
• Estimated time remaining.

You can temporarily interrupt processing at any time using CTRL-C. The command saves its progress, allowing you to restart processing at a later time. Use the --resume (or -r) option to restart the processing. For details, see Example: Stop and restart processing below.

Syntax

Usage:
    isi_sfse_assess <assess-mode> [process options] [sysctl options]

Assess Modes:
    -a | --all                            : assess all files on OneFS
    -p <path> | --path=<path>             : assess <path> and sub-dirs
    -r | --resume                         : resume previous assessment

Process Options:
    -q | --quick                           : slow mode (better accuracy)
    -f <fails> | --max-fails=<fails>      : max failures before aborting (default: 1000)
    -v | --verbose                        : verbose mode

Sysctl Options:
    --max-size=<bytes>                    : max file size to pack
    --avoid-bsin[=on|off]                 : avoid cloned/deduped files   
    --mirror-translation-enabled[=on|off] : convert mirrored to FEC
    --mirror-containers-enabled[=on|off]  : process mirrored files
    --snaps-enabled[=on|off]              : process snapshots
    --ads-enabled[=on|off]                : process ADS files

Options

-a | --all

Scans all files across the cluster for possible storage savings. The scan includes snapshots if both of the following are true:

• The --snaps-enabled option is set to on.
• The default (slow) process option is selected. If slow is not the process option, the scan is adjusted for faster processing, and snapshots are not included.

-p <path> | --path=<path>

Scans files in the named path for possible storage savings across the named directory path. This option performs a tree walk across all files and subdirectories within the named path. Because snapshots are invisible to the directory tree structure, the tree walk does not process any snapshots. Both absolute and relative path names are acceptable.

-r | --resume

Users can interrupt a running assessment using theCTRL-C keys simultaneously. This option resumes the assessment processing at the point where it was interrupted. The resumed process uses all of the same options that were specified on the original command.

-q | --quick

Slow mode is the default. Use this option to override the default and run in quick mode. The differences are:

• Quick mode makes some assumptions during processing based on file and block size, as opposed to gathering actual data block information. If your OneFS system stores only regular files (no snapshots, cloned or deduped files, etc.), the results of quick mode can be very close to the accuracy achieved in slow mode.
• Slow mode is more accurate but is very time-consuming. This mode collects actual data block information, including overhead blocks, and the results are precise.

-f <fails> | --max-fails=<fails>

The maximum number of failures allowed before aborting the assessment process. The default is 1000.

The command first collects a list of files to process and then proceeds with actual processing. A failure occurs when it attempts to process a file that was modified or deleted after being added to the list.These failures are more likely to occur on a busy cluster with a very large number of files.

-v | --verbose

Turns on verbose output.

--max-size=<bytes>

Sets the maximum size of files to select for processing. The default is 1040384 bytes, which is 8192 bytes less than 1MB, or 127 fs blocks. This value makes files less than 1MB available for packing.

--avoid-bsin[=on | off]

Controls whether to avoid cloned and deduped files.

The default is on, or true, meaning that deduped files are not processed. We recommend not to pack deduped files. Packing them has the effect of undoing the benefits of dedupe. Also, packing deduped files may affect performance when reading the packed file.

NOTE The dedupe functionality does not dedupe packed files.

--mirror-translation-enabled[=on | off]

Controls whether to pack mirrored files into FEC containers with equivalent protection. The default is off, or false.

• The off setting ensures that a mirrored file remains a true mirror. This is an important quality for some users.
• The on setting allows packing of files with mirror protection polices into containers with equivalent FEC protection policies. The on setting can increase space savings.

--mirror-containers-enabled[=on | off]

Controls whether to process mirrored files. The default is off, or false.

• The off setting does not process mirrored files.
• The on setting allows creation of containers with mirrored protection policies. Mirrored files remain mirrored, so there is no space saving. However, this setting can reduce the total protection group count and potentially reduce rebuild times.

--snaps-enabled[=on | off]

Controls whether to process snapshots. The default is off, or false.

• The off setting does not process snapshot files. Use this setting if processing time is an issue.
• The on setting processes snapshot files. This processing can significantly increase the time it takes to pack a data set if there are many snapshots with data. The advantage to using the on setting is the storage savings that may be gained. Snapshot files are often sporadically allocated, which typically results in poor storage efficiency. Packing can improve the storage efficiency.

--ads-enabled[=on | off]

Controls whether to process ADS files. The default is off, or false.

• The off setting does not process ADS files. Typically, these stream files are too large to be considered for packing. In addition, it is more efficient to process directories of streams files, but not efficient to process them singly from various locations.
• The on setting processes ADS files. Use this setting if you have small ADS files located together in a directory.

Example: Start assessment in slow mode on all files

The following command scans all files in slow mode.

# isi_sfse_assess -a

Example: Start assessment in quick mode on a directory

The following command uses quick mode to generate precise space saving estimates on the /ifs/my-data directory.

# isi_sfse_assess -q -p /ifs/my-data

Example: Stop and restart processing

# isi_sfse_assess -a --snaps-enabled --mirror-containers-enabled
# <CTRL-C> 
# isi_sfse_assess -r 

The process resumes using all of the same options that were originally entered.

Example: Verbose output

root# isi_sfse_assess -a -s -v --mirror-translation-enabled --mirror-containers-enabled
------------------------------------------------
SFSE simulation options:
  Slow mode: on
  Max fails: 1000
  Verbose output: on
  Sysctls:
    efs.sfm.pack.max_size: 1040384
    efs.sfm.pack.avoid_bsin: 1
    efs.sfm.pack.mirror_translation_enabled: 1
    (5 nodes involved in mirror translation)
    efs.sfm.pack.mirror_containers_enabled: 1
    efs.sfm.pack.snaps_enabled: 0
    efs.sfm.pack.ads_enabled: 0
------------------------------------------------
>> Starting LIN scan assessment...
>> 1632 files (13.27%) scanned...[ETC: 1 minute, 5 seconds]
>> 3014 files (19.51%) scanned...[ETC: 1 minute, 22 seconds]
>> 4423 files (22.95%) scanned...[ETC: 1 minute, 40 seconds]
>> 5685 files (25.75%) scanned...[ETC: 1 minute, 55 seconds]
>> 6960 files (28.87%) scanned...[ETC: 2 minutes, 3 seconds]
>> 8367 files (35.34%) scanned...[ETC: 1 minute, 49 seconds]
>> 9708 files (38.96%) scanned...[ETC: 1 minute, 49 seconds]
>> 11019 files (46.11%) scanned...[ETC: 1 minute, 33 seconds]
>> 12307 files (49.25%) scanned...[ETC: 1 minute, 32 seconds]
>> 13565 files (52.68%) scanned...[ETC: 1 minute, 29 seconds]
>> 14892 files (56.11%) scanned...[ETC: 1 minute, 26 seconds]
>> 16289 files (63.05%) scanned...[ETC: 1 minute, 10 seconds]
>> 17738 files (66.40%) scanned...[ETC: 1 minute, 5 seconds]
>> 19135 files (73.03%) scanned...[ETC: 51 seconds]
>> 20455 files (76.86%) scanned...[ETC: 45 seconds]
>> 21779 files (80.93%) scanned...[ETC: 37 seconds]
>> 22989 files (82.21%) scanned...[ETC: 36 seconds]
>> 24309 files (88.52%) scanned...[ETC: 23 seconds]
>> 25635 files (100.00%) scanned...[ETC: 0 seconds]
>> 25938 files (100.00%) scanned...[ETC: 0 seconds]
------------------------------------------------
25938 files scanned:
  * Packable: 23978
  * Non-packable: 14
    - Oversized: 14
    - Cloned/deduled: 0
    - Snapshots: 0
    - ADS: 0
    - Stubbed: 0
    - Zero-sized: 0
  * Failed: 0
  * Skipped: 1946

SFSE estimation summary:
  * Raw space saving: 1.5 GB
  * PG reduction: 22298

SFSE estimation details:
  * prot level: 3x, files: 3995, size: 314857823, data blks: 41090
    - effective prot level: 3+2
    - prot overhead: 82180 -> 27396
    - prot groups: 5666 -> 857
  * prot level: 4x, files: 3996, size: 315210935, data blks: 41134
    - effective prot level: 4x
    - prot overhead: 123402 -> 123402
    - prot groups: 5669 -> 2571
  * prot level: 5x, files: 3995, size: 314857823, data blks: 41090
    - effective prot level: 5x
    - prot overhead: 164360 -> 164360
    - prot groups: 5666 -> 2569
  * prot level: 8+2/2, files: 4002, size: 314867566, data blks: 41097
    - prot overhead: 38374 -> 10290
    - prot groups: 4002 -> 322
  * prot level: 12+3/3, files: 3995, size: 314857823, data blks: 41090
    - prot overhead: 57540 -> 10278
    - prot groups: 3995 -> 215
  * prot level: 16+4/4, files: 3995, size: 314857823, data blks: 41090
    - prot overhead: 76720 -> 10304
    - prot groups: 3995 -> 161
------------------------------------------------