Data Domain：如何列出 Data Domain 文件系统上的文件，sfs-dump

• 与备份应用程序管理的文件列表进行比较，以检查该备份应用程序中是否有孤立的文件。
• 列出超过特定存储时长阈值的所有文件，以确定备份应用程序是否正常执行备份保留

• 对于低于 7.13.0.0、7.10.1.15、7.7.5.25 和 6.2.1.110 的 DDOS

se sfs_dump

• 对于等于或高于 7.13.0.0、7.10.1.15、7.7.5.25、6.2.1.110 的任何 DDOS

filesys sfs-dump

sfs_dump

要从 CLI 获取 Data Domain 中文件的详细列表，以及有关获取存储在 Data Domain 中的文件的情报的过程，取决于正在运行的 DDOS 版本。收集包含文件详细信息的文本输出文件后，将输出处理为更方便使用的格式的过程始终是相同的，因为所有 DDOS 版本的输出格式都是相同的（或者可以使用脚本转换为通用格式）。

所有 DDOS 版本的共同点是要求以管理员用户身份登录到 Data Domain，并使用支持将控制台输出记录到客户端上的文本文件的 SSH 客户端执行此操作（PuTTY 非常适合）。用户必须配置 SSH 客户端，以便它将输出记录到客户端计算机上的文本文件中（确保记录的行数或单个行长度没有限制）。SSH 会话日志文件（以及 DD 文件列表）不会写入 Data Domain，而是会写入发起 SSH 连接的（通常是）台式计算机。

在收集 SSH 会话输出时，确保为文件指定一个有意义的名称（例如添加 Data Domain 主机名和要转储的 MTree 名称），并确保有足够的空间用于存储日志文件（对于具有 100 万个文件的系统，此空间大约为 200 MB）。

从 DD CLI 运行以收集文件列表的确切命令取决于使用的 DDOS 版本，总结如下。
 

7.13.0.0、7.10.1.15、7.7.5.25 和 6.2.1.110 之前的 DDOS 版本：

#### To produce the output directly as CSV
# se sfs_dump -c <mtree-path>

#### To produce the output as default format
# se sfs_dump <mtree-path>

# se sfs_dump -c

name    mtime   fileid  size    seg_bytes       seg_count       redun_seg_count pre_lc_size     post_lc_size
/data/col1/backup/file       1656995350000000000     78      33554431        33668007        4016    0       33668007        34208564

DDOS 版本 7.13.0.0、7.10.1.15、7.7.5.25 和 6.2.1.110 或更高版本：

se sfs_dump

# filesys sfs-dump mtree <mtree-path>

• 新命令不支持使用 -c 选项将文件信息转储为 CSV（列）格式
• 新命令一次只能运行一个 MTree，不支持在一次调用中针对整个 FS 运行

# mtree list

filesys sfs-dump

# filesys sfs-dump mtree <mtree-path>

如果您要获取多个 MTree 的文件详细信息，请选择将所有内容转储到同一输出文件，或者在针对每个 MTree 运行命令之前，在 SSH 客户端配置中切换到不同的文件。

或者，您可以通过 SSH 运行命令，以无人值守的方式收集输出，如下所示：

#### Releases 7.13.0.0, 7.10.1.15, 7.7.5.25 and 6.2.1.110 onwards only
# ssh sysadmin@DD-HOSTNAME "filesys sfs-dump mtree /data/col1/backup" > sfs-dump-DD-backup.txt

对于任一情况，在输出信息中每个文件占一行，格式如下：

/data/col1/backup/file: mtime: 1580211658000000000 fileid: 175 size: 136794 type: 9 seg_bytes: 138282 seg_count: 18 redun_seg_count: 14 (78%) pre_lc_size: 15045 post_lc_size: 8760 (58%) mode: 02000100644 start_offset: 0 end_offset: 18446744073709551615

DDOS 版本 8.1.0.0、7.13.1.10、7.10.1.30 和 7.7.5.40 或更高版本：

• 可以添加 -c 选项，以类似 CSV 的格式版本打印输出（字段由列分隔）
• 可以针对 FS 中的所有文件同时运行文件转储信息

#### File listing for all the files in a given MTree path
# filesys sfs-dump mtree <mtree-path>

#### File listing for all the files in a given MTree path, with CSV-like output
# filesys sfs-dump -c mtree <mtree-path>

#### File listing for all the files in the FS
# filesys sfs-dump

#### File listing for all the files in the FS, with CSV-like output
# filesys sfs-dump -c

如何将 sfs_dump 或 filesys sfs-dump 数据转化为有用的洞见：

/data/col1/backup/file: mtime: 1580211658000000000 fileid: 175 size: 136794 type: 9 seg_bytes: 138282 seg_count: 18 redun_seg_count: 14 (78%) pre_lc_size: 15045 post_lc_size: 8760 (58%) mode: 02000100644 start_offset: 0 end_offset: 18446744073709551615

Field 01 : filename with full path (/data/col1/backup/file)
Field 03 : file last modification time as UNIX epoch in nanoseconds (nanoseconds since January 1st 1970 at midnight)
Field 05 : file (inode) ID, inode numbers are per MTree (175)
Field 07 : pre-comp size for the file in bytes (136794)
Field 09 : type of file (9)
Field 11 : segment store bytes (this is NOT the file size), please ignore (138282)
Field 13 : segment count, or the number of segments the file consists of (18)
Field 15 : number of file segments which are redundant, or not unique (14)
Field 16 : percentage of non-unique segments to total segments (78%)
Field 18 : combined size of the file's unique segments after deduplication (15045)
Field 20 : combined size of the file's unique segments after deduplication and local compression (8760)
Field 21 : ratio of post-deduplicated + post-compressed unique segments size to post-deduplicated unique segment size (58%), shows how well local compression worked
Field 23 : file mode, please ignore

• 文件重复数据消除（我们称之为“gcomp”或全局压缩）的系数为 x9.09（136,794 到 15,045 字节）
• 文件本地压缩（我们称之为“lcomp”，表示本地压缩）的系数为 x1.72（15,045 到 8760 字节）
• 估计的总文件大小缩减（称为“压缩比”）的系数为 x15.62（136,794 到 8760 字节）

/data/col1/backup/file       1656995350000000000     78      33554431        33668007        4016    0       33668007        34208564

Field 01 : filename with full path (/data/col1/backup/file)
Field 02 : file last modification time as UNIX epoch in nanoseconds (nanoseconds since January 1st 1970 at midnight)
Field 03 : file (inode) ID, inode numbers are per MTree
Field 04 : pre-comp size for the file in bytes (33554431)
Filed 05 : segment store bytes (this is NOT the file size), please ignore (33668007)
Field 06 : segment count, or the number of segments the file consists of (4016)
Field 07 : number of file segments which are redundant, or not unique (0)
Field 08 : combined size of the file's unique segments after deduplication (33668007)
Field 09 : combined size of the file's unique segments after deduplication and local compression (34208564)

• 文件重复数据消除（我们称之为“gcomp”或全局压缩）的系数为 x0.99（33,554,431 到 33,668,007 字节）
• 文件本地压缩（我们称之为“lcomp”，表示本地压缩）的系数为 x0.98（33,668,007 到 34,208,564 字节）
• 估计的总文件大小缩减（称为“压缩比”）的系数为 x0.98（33,554,431 到 34,208,564 字节）

mtree show compression

# cat sfs-dump-noCSV.txt | grep ^/ | awk '
BEGIN {print "name\tmtime\tfileid\tsize\tseg_bytes\tseg_count\tredun_seg_count\tpre_lc_size\tpost_lc_size"}
{
    colon_index = index($0, ":")
    filename = substr($0, 1, colon_index - 1)
    gsub(/^[[:space:]]+|[[:space:]]+$/, "", filename)
    n = split(substr($0, colon_index + 1), fields, " ")
    print filename "\t" fields[2] "\t" fields[4] "\t" fields[6] "\t" fields[10] "\t" fields[12] "\t" fields[14] "\t" fields[17] "\t" fields[19]
}' > sfs-dump-CSV.csv

将 CSV 格式的 sfs_dump 输出导入到电子表格（Excel 示例）：

1. 打开 Excel 并创建一个新的空白电子表格
2. 单击顶部的数据菜单，然后单击名为从文本/CSV 的图标
3. 使用对话框找到 CSV 格式的文件数据（sfs-dump-CSV.csv ，如非 CSV 格式的转换脚本示例中所示），选择该文件，然后单击导入
4. 如果输入文件格式正确，并且 Excel 可以从前 200 行生成格式（它应该这样做），则对话框应显示要生成的电子表格的预览。检查信息以确保一切看起来正常，包括将第一行检测为标头
5. 如果预览没问题，请单击对话框底部的加载按钮，您将获得格式正确的新电子表格，并且字段标题行启用搜索和筛选。

添加额外的计算字段

时间戳转换：

1. 右键单击单元格 J2，选择设置单元格格式
2. 在左侧的分类列表中，选择自定义。
3. 如果其中没有人类可读的日期时间格式，则创建该格式（示例在末尾包含 EST 字符串。您可以用您的文本时区定义替换它，或者如果不感兴趣，则完全删除“EST”）：yyyy-mm-dd hh:mm:ss "EST"
4. 完成后单击“确定”。现在，单元格 J2 具有自定义数据格式。
5. 将以下公式添加到单元格 J2。在公式中，将 (-6*3600) 替换为在获取“sfs-dump”数据时与 Data Domain 中配置的时区的 UTC 对应的正确时区差异。例如，在夏季，美国东部时间比 UTC 晚 6 小时，因此替换为“-6”。

(((B2/1000000000+(-6*3600))/86400)+25569)
(((B2/1000000000)/86400)+25569)

• 复制以下公式并在单元格 K2 中选择性粘贴为文本，以创建一个列，用于显示每个文件的重复数据消除或 gcomp 信息：

  =IF(H2=0,0,D2/H2)

• 复制以下公式并在单元格 L2 中选择性粘贴为文本，以创建一个列，用于显示每个文件的本地压缩或 lcomp 信息：

  =IF(I2=0,0,H2/I2)

• 复制以下公式并在单元格 M2 中选择性粘贴为文本，以创建一个列，用于显示每个文件的总文件压缩率：

  =K2*L2

• A 包含文件名
• B 包含上次写入文件的日期戳（UNIX 时间戳，以纳秒为单位）
• D 是文件的原始大小
• H 是全局压缩后的大小
• I 是全局和本地压缩后的大小
• J 包含上次写入文件的日期戳，采用人类可读格式
• K 包含文件的全局压缩（重复数据消除）
• L 包含文件的本地压缩
• M 包含文件的总压缩

• 例如，如需仅显示 MTree /data/col1/backup 中超过 30 天的文件：

1. 单击名称标题中的向下箭头，选择文本筛选，然后选择开头是并在文本框中键入 /data/col1/backup/ 。不要遗漏尾部斜杠。单击确定
2. 单击最后写入日期标题中的向下箭头，选择日期筛选，然后选择之前。使用对话框右侧的日期选择器选择 30 天前的日期。单击确定。

转储文件位置的命令（活动和云）

#### For the DD FS as a whole
# filesys report generate file-location

#### For an individual MTree or subdirectory
# filesys report generate file-location path /data/col1/test

这两种情况的输出为以下类型：

-------------------------      ----------------------      ----------------------      ---------------------------
File Name                      Location(Unit Name)         Size                        Placement Time
-------------------------      ----------------------      ----------------------      ---------------------------
/data/col1/test/file1.bin                  Cloud-Unit                  200.00 GiB      Sat Mar  5 13:24:10 2022
/data/col1/test/file2.bin                      Active                   10.00 TiB      Sat May 14 00:48:17 2022
-------------------------      ----------------------      ----------------------      ---------------------------

sfs_dump -c

# split  -b <size> <filename> <new filename prefix>  (splits by file size)
# split  -l <size> <filename> <new filename prefix>  (splits by number of lines)
# split  -n <size> <filename> <new filename prefix>  (splits by number of pieces)

例如，要将输入文本文件拆分为每个大小为 200 MiB 的数据块，并为每个块命名，运行：

"sfs_dump.out.split"

然后运行：

# split -b 200M sfs_dump.out sfs_dump.out.split

使用 "filesys sfs-dump" 进行整个 FS 文件数据转储

#!/bin/bash

#### WARNING
####     This script is provided to you by Dell Technologies with NO GUARANTEE, as best-effort sample code to get a full FS sfs-dump collected
####     for DDOS releases which do not support "se sfs-dump", or as an alternative to it, in releases which support "filesys fs-dump"
####     That this script does not work for you, or if you need help setting it up, extending it, or resolving any issues, is not entitled to support
####     This script is not part of Dell PowerProtect / Data Domain, and hence it is not supported

#### Replace values below to suit your needs
USERNAME="sysadmin"
DD_HOSTNAME="10.60.36.172"
#### NO CHANGES NEEDED BEYOND THIS POINT

clear
echo "Script collects a full FS sfs-dump from \"${DD_HOSTNAME}\", using the command \"filesys sfs-dump\", one MTree at a time"
echo "    * Script has to be configured by settting the \"USERNAME\" and \"DD_HOSTNAME\" variables within the top of the script"
echo "    * Script expects passwordless SSH connection as \"USERNAME\" to the \"DD_HOSTNAME\" configured"
echo "    * To configure passwordless SSH login to a DataDomain, check KB at https://www.dell.com/support/kbdoc/000004033 "
echo "    * Test passwordless login is configured and working prior to going ahead with this script"
echo "    * If passwordless login is not configured, script will ask for the \"${USERNAME}\" password "
echo "          - Once for getting the MTree list"
echo "          - And once for each one of the MTrees in \"${DD_HOSTNAME}\" "
echo
echo -n "Are you sure you want to continue? (y/n) : "
read -n 1 answer
echo
if [ "${answer}" = "y" ]; then
    echo "Going ahead with the script."
    echo
else
    echo "Stopping script now. Re-run when passwordless login to \"${DD_HOSTNAME}\" as \"${USERNAME}\" works. Bye."
    exit 1
fi

echo -n "1/6 : Collecting list of MTrees from DD..."
ssh ${USERNAME}@${DD_HOSTNAME} "mtree list" 2>/dev/null | grep ^/ | awk '{print $(NF-3)}' > mtree-list.txt
echo "Done."

n_mtrees=$( wc -l mtree-list.txt | cut -d" "  -f1 )
echo -n "2/6 : Collecting per-Mtree sfs-dump information for ${n_mtrees} MTrees ..."
for mtree in `cat mtree-list.txt`; do
    name=$(echo $mtree | cut -d/ -f4)
    ssh ${USERNAME}@${DD_HOSTNAME} "filesys sfs-dump mtree ${mtree}" 2>/dev/null | grep ^/ > sfs-dump-${name}.txt
    echo -n "."
done
echo "Done."

echo -n "3/6 : Putting all the files together..."
for file in `ls sfs-dump-*.txt`; do 
    if [ -s "${file}" ]; then cat ${file} >> sfs-dump-noCSV.txt; fi
done
echo "Done."

echo -n "4/6 : Converting sfs-dump output to CSV format..."
cat sfs-dump-noCSV.txt | grep ^/ | grep -v ^$ | awk '
BEGIN {print "name\tmtime\tfileid\tsize\tseg_bytes\tseg_count\tredun_seg_count\tpre_lc_size\tpost_lc_size"}
{
    colon_index = index($0, ":")
    filename = substr($0, 1, colon_index - 1)
    gsub(/^[[:space:]]+|[[:space:]]+$/, "", filename)
    n = split(substr($0, colon_index + 1), fields, " ")
    print filename "\t" fields[2] "\t" fields[4] "\t" fields[6] "\t" fields[10] "\t" fields[12] "\t" fields[14] "\t" fields[17] "\t" fields[19]
}' > sfs-dump-CSV.csv
echo "Done."

echo -n "5/6 : Cleaning up..."
for mtree in `cat mtree-list.txt`; do name=$(echo $mtree | cut -d/ -f4); rm -f sfs-dump-${name}.txt ; done
rm sfs-dump-noCSV.txt
rm mtree-list.txt
echo "Done."


echo -n "6/6 : Summary"
echo
n_files=$( wc -l sfs-dump-CSV.csv | cut -d" "  -f1 )
echo
echo "Collecting whole FS sfs-dump data from ${HOSTNAME} completed"
echo "File includes output for ${n_mtrees} MTrees, with a combined $(( ${n_files} - 1 )) files across Active and Cloud Tiers (if applicable)"
echo "Start of file shown below for your convenience :"
echo "===================="
head -5 sfs-dump-CSV.csv
echo "===================="
echo
echo "Done."

exit 0

用于将非 CSV“sfs-dump”和“filesys report generate file-location”的输出合并为 CSV 文件的脚本，该文件包含与上文脚本相同的所有信息以及每个文件层和放置时间信息

#!/bin/bash

#### WARNING
####     This script is provided to you by Dell Technologies with NO GUARANTEE, as best-effort sample code to match the output from commands :
####       * sfs-dump (in non CSV format)
####       * filesys report generate file-location
####     So that a new CSV with the file paths appearing on both is created for all the data in sfs-dump file with the added tier and placement time information from location report
####
####     This script is not part of Dell PowerProtect / Data Domain, and hence it is not supported
####
####     Usage : extend-sfs-dump-with-tier.sh sfs-dump-output.csv file-location-output.log
####     Output : static "sfs-dump-output-tiers.csv" file name (may be changed below)

#### Replace values below to suit your needs
OUTPUT_FILENAME="sfs-dump-output-tiers.csv"
#### NO CHANGES NEEDED BEYOND THIS POINT

clear

if [ ! $# == 2 ]; then
    echo "Combine output from sfs-dump and tier location report into a CSV file with tier and placement time information"
    echo
    echo "Usage : $0 SFS-DUMP-OUTPUT-FILE    REPORT-FILE-LOCATION-FILE"
    echo "NOTE : SFS-DUMP-OUTPUT-FILE has to be in non-CSV format"
    exit 1
fi

INPUT_SFSDUMP="$1"
INPUT_LOCATION="$2"


echo -n "1/6 : Sanity checking input files..."
if [ ! -s "${INPUT_SFSDUMP}" ]; then
    echo "Input file ${INPUT_SFSDUMP} does not exist"
    exit 1
fi
if [ ! -s "${INPUT_LOCATION}" ]; then
    echo "Input file ${INPUT_LOCATION} does not exist"
    exit 1
fi
n_files_sfsdump=`grep ^/ ${INPUT_SFSDUMP} | wc -l`
n_files_location=`grep ^/ ${INPUT_LOCATION} | wc -l`
if [ ${n_files_sfsdump} -eq ${n_files_location} ]; then
    echo -n "both have the same amount of files (${n_files_location}). "
else
    echo -n "sfs-dump has ${n_files_sfsdump} files whereas location report has ${n_files_location} files, this may be normal if the difference is small. "
fi
echo "Done."


echo -n "2/6 : Sanitize \"file-location\" input..."
cat ${INPUT_LOCATION} | awk 'BEGIN {rejected="temp-location-rejected.log"; accepted="temp-location-accepted.log"} { if ( $0 ~ "Missing -unit") { gsub(/Missing -unit/, "Missing-Cloud-Unit", $0); print $0 > rejected } else { if ($0 ~ "^/" ) print $0 > accepted } }'
if [ -s "temp-location-rejected.log" ]; then
    REJECTS_EXIST="yes"
    echo -n "Some files in location report sit in unavailable or deleted cloud units, you may need to re-run this script after fixing the issue and gathering a new location report. "
    cat temp-location-rejected.log temp-location-accepted.log | sed -e 's/\t/:\t/' | sort > temp-location-report-sorted.log
    rm temp-location-rejected.log
else
    cat temp-location-accepted.log | sed -e 's/\t/:\t/' | sort > temp-location-report-sorted.log
    REJECTS_EXIST="no"
fi
rm temp-location-accepted.log
echo "Done."


echo -n "3/6 : Sanitize \"sfs-dump\" input..."
cat ${INPUT_SFSDUMP} | grep ^/ | sort > temp-sfs-dump.log
echo "Done."


echo -n "4/6 : Merging information for sfs-dump and location report..."
join -1 1 -2 1 temp-sfs-dump.log temp-location-report-sorted.log > temp-merged-information.log
rm temp-sfs-dump.log
rm temp-location-report-sorted.log
n_files_combined=`grep ^/ temp-merged-information.log | wc -l`
if [ ${n_files_combined} -eq 0 ]; then
    echo "No files matched from input files. sfs-dump output must NOT be in CSV format. Exiting."
    rm temp-merged-information.log
    exit 1
fi
echo -n "Input files matched on ${n_files_combined} files. "
echo "Done."


echo -n "5/6 : Converting merged sfs-dump / location-report output to CSV format..."
cat temp-merged-information.log | grep ^/ | grep -v ^$ | awk '
BEGIN {print "name\tmtime\tfileid\tsize\tseg_bytes\tseg_count\tredun_seg_count\tpre_lc_size\tpost_lc_size\ttier\tplacement_time"}
{
    colon_index = index($0, ":")
    filename = substr($0, 1, colon_index - 1)
    gsub(/^[[:space:]]+|[[:space:]]+$/, "", filename)
    n = split(substr($0, colon_index + 1), fields, " ")
    print filename "\t" fields[2] "\t" fields[4] "\t" fields[6] "\t" fields[10] "\t" fields[12] "\t" fields[14] "\t" fields[17] "\t" fields[19] "\t" fields[27] \
        "\t" fields[length(fields)-4] " " fields[length(fields)-3] " " fields[length(fields)-2] " " fields[length(fields)-1] " " fields[length(fields)]
}' > ${OUTPUT_FILENAME}
rm temp-merged-information.log
echo "Done."


echo -n "6/6 : Summary"
echo
echo
echo "Merging information from sfs-dump (${INPUT_SFSDUMP}) and location-report ${INPUT_LOCATION} output completed."
echo "Output file (${OUTPUT_FILENAME}) includes information for a total ${n_files_combined} files, out of ${n_files_sfsdump} in input sfs-dump, and ${n_files_location} in input location report."
if [ "${REJECTS_EXIST}" == "yes" ]; then
    echo "Note there are some files in disconnected or deleted cloud units, for which the \"tier\" field has been replaced with \"Missing-Cloud-Unit\"."
fi
echo
echo "Start of file shown below for your convenience :"
echo "===================="
head -5 ${OUTPUT_FILENAME}
echo "===================="
echo
echo "You may follow the instructions in https://www.dell.com/support/kbdoc/000081345 to process this CSV file in an spreadhseet"
echo
echo "Done."

exit 0

对于 Veritas NetBackup 用户：

/data/col1/MTREE_NAME/POLICY-NAME_1400502741_C1_F1:1400502741:VM_PRD-02:4:1::
/data/col1/MTREE_NAME/POLICY-NAME_1400502741_C1_F2:1400502741:VM_PRD-02:4:1::
/data/col1/MTREE_NAME/POLICY-NAME_1400502741_C1_HDR:1400502741:VM_PRD-02:4:1::

--- iterate-dd-for-fs-sfs-dump.sh       2024-01-23 06:32:16.521409000 -0500
+++ iterate-dd-for-fs-sfs-dump-NBU.sh   2024-02-27 03:26:42.808246000 -0500
@@ -55,11 +55,11 @@
 cat sfs-dump-noCSV.txt | grep ^/ | grep -v ^$ | awk '
 BEGIN {print "name\tmtime\tfileid\tsize\tseg_bytes\tseg_count\tredun_seg_count\tpre_lc_size\tpost_lc_size"}
 {
-    colon_index = index($0, ":")
-    filename = substr($0, 1, colon_index - 1)
+    colon_index = index($0, ":::")
+    filename = substr($0, 1, colon_index + 1)
     gsub(/^[[:space:]]+|[[:space:]]+$/, "", filename)
     n = split(substr($0, colon_index + 1), fields, " ")
-    print filename "\t" fields[2] "\t" fields[4] "\t" fields[6] "\t" fields[10] "\t" fields[12] "\t" fields[14] "\t" fields[17] "\t" fields[19]
+    print filename "\t" fields[3] "\t" fields[5] "\t" fields[7] "\t" fields[11] "\t" fields[13] "\t" fields[15] "\t" fields[18] "\t" fields[20] "\t"
 }' > sfs-dump-CSV.csv
 echo "Done."