Data Domain：如何列出 Data Domain 檔案系統上的檔案, sfs-dump

• 比較由備份應用程式管理的檔案清單，來檢查是否有任何未由備份應用程式管理的檔案。
• 根據特定建立時間閾值來列出所有檔案，以判斷備份應用程式是否正確強制執行備份保留

• 適用於早於 7.13.0.0、7.10.1.15、7.7.5.25 和 6.2.1.110 的 DDOS

se sfs_dump

• 適用於等於或晚於 7.13.0.0、7.10.1.15、7.7.5.25、6.2.1.110 的任何 DDOS

filesys sfs-dump

sfs_dump

若要從 CLI 取得 Data Domain 中檔案的詳細清單，以及最終取得儲存在 Data Domain 中檔案情報的程序，取決於執行的 DDOS 版本。收集包含檔案詳細資料的文字輸出檔案後，將輸出處理成更實用的形式一律相同，因為所有 DDOS 版本的輸出格式都相同 (或者可以使用指令檔轉換為一般格式)。

所有 DDOS 版本的共同點是要求以系統管理員使用者身分登入 Data Domain，並使用支援將主控台輸出記錄至用戶端文字檔案的 SSH 用戶端執行此動作 (PuTTY 適用於此方式)。使用者必須配置 SSH 用戶端，以便它將輸出記錄到用戶端電腦上的文字檔案 (確保記錄的行數或個行長度沒有限制)。SSH 工作階段記錄檔 (以及 DD 檔案清單) 不會寫入 Data Domain 上，而是在啟動 SSH 連線的 (一般為) 桌上型電腦上。

收集 SSH 工作階段輸出時，請務必為檔案指定有意義的名稱 (例如新增 Data Domain 主機名稱和要傾印的 MTree 名稱)，並確保有足夠的空間容納記錄檔，對於包含 100 萬個檔案的系統而言，這大約為 200 MB。

從 DD CLI 執行以收集檔案清單的確切命令取決於所使用的 DDOS 版本，摘要如下。
 

7.13.0.0、7.10.1.15、7.7.5.25 和 6.2.1.110 之前的 DDOS 版本：

#### To produce the output directly as CSV
# se sfs_dump -c <mtree-path>

#### To produce the output as default format
# se sfs_dump <mtree-path>

# se sfs_dump -c

name    mtime   fileid  size    seg_bytes       seg_count       redun_seg_count pre_lc_size     post_lc_size
/data/col1/backup/file       1656995350000000000     78      33554431        33668007        4016    0       33668007        34208564

DDOS 版本 7.13.0.0、7.10.1.15、7.7.5.25 和 6.2.1.110 或更新版本：

se sfs_dump

# filesys sfs-dump mtree <mtree-path>

• 新命令不支援以 CSV (欄) 格式傾印檔案資訊的 -c 選項
• 新命令一次只能執行一個 MTree，不支援在單一調用中針對整個 FS 執行

# mtree list

filesys sfs-dump

# filesys sfs-dump mtree <mtree-path>

如果您要取得多個 MTree 的檔案詳細資料，請選擇將所有傾印複製到相同的輸出檔案，或在針對每個 MTree 執行命令之前，將 SSH 用戶端組態切換至不同的檔案。

或者，您可以透過 SSH 以自動方式收集輸出，如下所示：

#### Releases 7.13.0.0, 7.10.1.15, 7.7.5.25 and 6.2.1.110 onwards only
# ssh sysadmin@DD-HOSTNAME "filesys sfs-dump mtree /data/col1/backup" > sfs-dump-DD-backup.txt

在任一情況下，每個檔案的輸出資訊都有一行，格式如下：

/data/col1/backup/file: mtime: 1580211658000000000 fileid: 175 size: 136794 type: 9 seg_bytes: 138282 seg_count: 18 redun_seg_count: 14 (78%) pre_lc_size: 15045 post_lc_size: 8760 (58%) mode: 02000100644 start_offset: 0 end_offset: 18446744073709551615

DDOS 版本 8.1.0.0、7.13.1.10、7.10.1.30 和 7.7.5.40 或更新版本：

• 新增 -c 選項的可能性，因此輸出會以類似 CSV 格式的版本列印 (依欄分隔的欄位)
• 一次為 FS 中的所有檔案執行檔案傾印資訊的可能性

#### File listing for all the files in a given MTree path
# filesys sfs-dump mtree <mtree-path>

#### File listing for all the files in a given MTree path, with CSV-like output
# filesys sfs-dump -c mtree <mtree-path>

#### File listing for all the files in the FS
# filesys sfs-dump

#### File listing for all the files in the FS, with CSV-like output
# filesys sfs-dump -c

如何將 sfs_dump 或 filesys sfs-dump 資料處理成有用的東西：

/data/col1/backup/file: mtime: 1580211658000000000 fileid: 175 size: 136794 type: 9 seg_bytes: 138282 seg_count: 18 redun_seg_count: 14 (78%) pre_lc_size: 15045 post_lc_size: 8760 (58%) mode: 02000100644 start_offset: 0 end_offset: 18446744073709551615

Field 01 : filename with full path (/data/col1/backup/file)
Field 03 : file last modification time as UNIX epoch in nanoseconds (nanoseconds since January 1st 1970 at midnight)
Field 05 : file (inode) ID, inode numbers are per MTree (175)
Field 07 : pre-comp size for the file in bytes (136794)
Field 09 : type of file (9)
Field 11 : segment store bytes (this is NOT the file size), please ignore (138282)
Field 13 : segment count, or the number of segments the file consists of (18)
Field 15 : number of file segments which are redundant, or not unique (14)
Field 16 : percentage of non-unique segments to total segments (78%)
Field 18 : combined size of the file's unique segments after deduplication (15045)
Field 20 : combined size of the file's unique segments after deduplication and local compression (8760)
Field 21 : ratio of post-deduplicated + post-compressed unique segments size to post-deduplicated unique segment size (58%), shows how well local compression worked
Field 23 : file mode, please ignore

• 檔案重複資料刪除 (我們稱之為「gcomp」，或全域壓縮) 為因數 x9.09 (136,794 至 15,045 位元組)
• 檔案本機壓縮 (我們稱之為「lcomp」，用於本機壓縮) 為因數 x1.72 (15,045 至 8760 位元組)
• 縮減的總檔案大小預估 (稱為「壓縮率」) 為因數 x15.62 (136,794 至 8760 位元組)

/data/col1/backup/file       1656995350000000000     78      33554431        33668007        4016    0       33668007        34208564

Field 01 : filename with full path (/data/col1/backup/file)
Field 02 : file last modification time as UNIX epoch in nanoseconds (nanoseconds since January 1st 1970 at midnight)
Field 03 : file (inode) ID, inode numbers are per MTree
Field 04 : pre-comp size for the file in bytes (33554431)
Filed 05 : segment store bytes (this is NOT the file size), please ignore (33668007)
Field 06 : segment count, or the number of segments the file consists of (4016)
Field 07 : number of file segments which are redundant, or not unique (0)
Field 08 : combined size of the file's unique segments after deduplication (33668007)
Field 09 : combined size of the file's unique segments after deduplication and local compression (34208564)

• 檔案重複資料刪除 (我們稱之為「gcomp」，或全域壓縮) 為因數 x0.99 (33,554,431 至 33,668,007 位元組)
• 檔案本機壓縮 (我們稱之為「lcomp」，用於本機壓縮) 為因數 x0.98 (33,668,007 至 34208564 位元組)
• 縮減的總檔案大小預估 (稱為「壓縮率」) 為因數 x0.98 (33,554,431 至 34208564 位元組)

mtree show compression

# cat sfs-dump-noCSV.txt | grep ^/ | awk '
BEGIN {print "name\tmtime\tfileid\tsize\tseg_bytes\tseg_count\tredun_seg_count\tpre_lc_size\tpost_lc_size"}
{
    colon_index = index($0, ":")
    filename = substr($0, 1, colon_index - 1)
    gsub(/^[[:space:]]+|[[:space:]]+$/, "", filename)
    n = split(substr($0, colon_index + 1), fields, " ")
    print filename "\t" fields[2] "\t" fields[4] "\t" fields[6] "\t" fields[10] "\t" fields[12] "\t" fields[14] "\t" fields[17] "\t" fields[19]
}' > sfs-dump-CSV.csv

匯入 CSV 格式的 sfs_dump 輸出到試算表 (Excel 範例)：

1. 開啟 Excel 並建立新的空白試算表
2. 前往上方的資料功能表，然後按一下名為自文字/CSV 的圖示
3. 使用該對話方塊找出 CSV 格式的檔案資料 (sfs-dump-CSV.csv 如非 CSV 格式的範例轉換指令檔中所示)，選取後按一下匯入
4. 如果輸入檔案格式正確，且 Excel 可以從前 200 行 (應可) 擷取該格式，則對話方塊應會顯示要產生的試算表預覽。查看資訊以確保外觀正常，包括第一行偵測為標題
5. 如果預覽一切正常，按一下對話方塊底部的載入按鈕，您的新試算表即會以合適的格式呈現，欄位標題會變為支援搜尋和篩選。

新增額外的計算欄位

時間戳記轉換：

1. 在儲存格 J2 上按右鍵，然後選擇設定儲存格格式
2. 在左側的類別清單中，選擇自訂。
3. 如果尚未存在，請為人類可讀的日期和時間格式建立格式 (範例中的結尾包含 EST 字串。您可以將其取代為您的文字時區定義，或者如果不想使用的話，請完全移除「EST」)：yyyy-mm-dd hh:mm:ss "EST"
4. 完成後，按一下確定。現在儲存格 J2 即有自訂的資料格式。
5. 將下列公式新增到儲存格 J2。在公式中，針對在取得「sfs-dump」資料時，於 Data Domain 中設定的時區，將 (-6*3600) 取代為對應至 UTC 的正確時區差異。例如，美國東部時間在夏季比 UTC 晚 6 小時，因此此處為「-6」。

(((B2/1000000000+(-6*3600))/86400)+25569)
(((B2/1000000000)/86400)+25569)

• 複製以下公式並在儲存格 K2 中選擇性貼上成為文字，以為依檔案的重複資料刪除或 gcomp 建立一欄：

  =IF(H2=0,0,D2/H2)

• 複製以下公式並在儲存格 L2 中選擇性貼上成為文字，以為依檔案的本機壓縮或 lcomp 建立一欄：

  =IF(I2=0,0,H2/I2)

• 複製以下公式並在儲存格 M2 中選擇性貼上成為文字，以為依檔案的總檔案壓縮率建立一欄：

  =K2*L2

• A 包含檔案名稱
• B 包含上次寫入檔案時的日期戳記 (在 UNIX epoch 中以奈秒為單位)
• D 是檔案的原始大小
• H 是全域壓縮後的大小
• I 是全域與本機壓縮後的大小
• J 包含上次寫入檔案時的日期戳記，採用人類可讀的形式
• K 包含檔案的全域壓縮 (重複資料刪除)
• L 包含檔案的本機壓縮
• M 包含檔案的總壓縮

• 例如，若要僅顯示 MTree /data/col1/backup 中超過 30 天的檔案：

1. 按一下名稱標題中的向下箭頭，選取文字篩選，然後選取開頭為，並鍵入 /data/col1/backup/ 至方塊中。結尾的斜線很重要。按一下確定。
2. 按一下上次寫入日期標題中的向下箭頭，選取日期篩選，然後選取早於。使用對話方塊右側的日期選取器選擇 30 天前的日期。按一下確定。

傾印檔案位置的命令 (使用中和雲端)

#### For the DD FS as a whole
# filesys report generate file-location

#### For an individual MTree or subdirectory
# filesys report generate file-location path /data/col1/test

任一情況下的輸出都是以下類型：

-------------------------      ----------------------      ----------------------      ---------------------------
File Name                      Location(Unit Name)         Size                        Placement Time
-------------------------      ----------------------      ----------------------      ---------------------------
/data/col1/test/file1.bin                  Cloud-Unit                  200.00 GiB      Sat Mar  5 13:24:10 2022
/data/col1/test/file2.bin                      Active                   10.00 TiB      Sat May 14 00:48:17 2022
-------------------------      ----------------------      ----------------------      ---------------------------

sfs_dump -c

# split  -b <size> <filename> <new filename prefix>  (splits by file size)
# split  -l <size> <filename> <new filename prefix>  (splits by number of lines)
# split  -n <size> <filename> <new filename prefix>  (splits by number of pieces)

例如，要將輸入文字檔案分割為各 200 MiB 的區塊，並將這些部分的開頭命名為：

"sfs_dump.out.split"

然後執行：

# split -b 200M sfs_dump.out sfs_dump.out.split

要對整個檔案資料的 FS 傾印使用 "filesys sfs-dump" 的指令檔

#!/bin/bash

#### WARNING
####     This script is provided to you by Dell Technologies with NO GUARANTEE, as best-effort sample code to get a full FS sfs-dump collected
####     for DDOS releases which do not support "se sfs-dump", or as an alternative to it, in releases which support "filesys fs-dump"
####     That this script does not work for you, or if you need help setting it up, extending it, or resolving any issues, is not entitled to support
####     This script is not part of Dell PowerProtect / Data Domain, and hence it is not supported

#### Replace values below to suit your needs
USERNAME="sysadmin"
DD_HOSTNAME="10.60.36.172"
#### NO CHANGES NEEDED BEYOND THIS POINT

clear
echo "Script collects a full FS sfs-dump from \"${DD_HOSTNAME}\", using the command \"filesys sfs-dump\", one MTree at a time"
echo "    * Script has to be configured by settting the \"USERNAME\" and \"DD_HOSTNAME\" variables within the top of the script"
echo "    * Script expects passwordless SSH connection as \"USERNAME\" to the \"DD_HOSTNAME\" configured"
echo "    * To configure passwordless SSH login to a DataDomain, check KB at https://www.dell.com/support/kbdoc/000004033 "
echo "    * Test passwordless login is configured and working prior to going ahead with this script"
echo "    * If passwordless login is not configured, script will ask for the \"${USERNAME}\" password "
echo "          - Once for getting the MTree list"
echo "          - And once for each one of the MTrees in \"${DD_HOSTNAME}\" "
echo
echo -n "Are you sure you want to continue? (y/n) : "
read -n 1 answer
echo
if [ "${answer}" = "y" ]; then
    echo "Going ahead with the script."
    echo
else
    echo "Stopping script now. Re-run when passwordless login to \"${DD_HOSTNAME}\" as \"${USERNAME}\" works. Bye."
    exit 1
fi

echo -n "1/6 : Collecting list of MTrees from DD..."
ssh ${USERNAME}@${DD_HOSTNAME} "mtree list" 2>/dev/null | grep ^/ | awk '{print $(NF-3)}' > mtree-list.txt
echo "Done."

n_mtrees=$( wc -l mtree-list.txt | cut -d" "  -f1 )
echo -n "2/6 : Collecting per-Mtree sfs-dump information for ${n_mtrees} MTrees ..."
for mtree in `cat mtree-list.txt`; do
    name=$(echo $mtree | cut -d/ -f4)
    ssh ${USERNAME}@${DD_HOSTNAME} "filesys sfs-dump mtree ${mtree}" 2>/dev/null | grep ^/ > sfs-dump-${name}.txt
    echo -n "."
done
echo "Done."

echo -n "3/6 : Putting all the files together..."
for file in `ls sfs-dump-*.txt`; do 
    if [ -s "${file}" ]; then cat ${file} >> sfs-dump-noCSV.txt; fi
done
echo "Done."

echo -n "4/6 : Converting sfs-dump output to CSV format..."
cat sfs-dump-noCSV.txt | grep ^/ | grep -v ^$ | awk '
BEGIN {print "name\tmtime\tfileid\tsize\tseg_bytes\tseg_count\tredun_seg_count\tpre_lc_size\tpost_lc_size"}
{
    colon_index = index($0, ":")
    filename = substr($0, 1, colon_index - 1)
    gsub(/^[[:space:]]+|[[:space:]]+$/, "", filename)
    n = split(substr($0, colon_index + 1), fields, " ")
    print filename "\t" fields[2] "\t" fields[4] "\t" fields[6] "\t" fields[10] "\t" fields[12] "\t" fields[14] "\t" fields[17] "\t" fields[19]
}' > sfs-dump-CSV.csv
echo "Done."

echo -n "5/6 : Cleaning up..."
for mtree in `cat mtree-list.txt`; do name=$(echo $mtree | cut -d/ -f4); rm -f sfs-dump-${name}.txt ; done
rm sfs-dump-noCSV.txt
rm mtree-list.txt
echo "Done."


echo -n "6/6 : Summary"
echo
n_files=$( wc -l sfs-dump-CSV.csv | cut -d" "  -f1 )
echo
echo "Collecting whole FS sfs-dump data from ${HOSTNAME} completed"
echo "File includes output for ${n_mtrees} MTrees, with a combined $(( ${n_files} - 1 )) files across Active and Cloud Tiers (if applicable)"
echo "Start of file shown below for your convenience :"
echo "===================="
head -5 sfs-dump-CSV.csv
echo "===================="
echo
echo "Done."

exit 0

將非 CSV「sfs-dump」和「filesys report generate file-location」的輸出合併為 CSV 檔案，其中包含與上述指令檔相同的所有資訊，以及每個檔案層和放置時間的資訊

#!/bin/bash

#### WARNING
####     This script is provided to you by Dell Technologies with NO GUARANTEE, as best-effort sample code to match the output from commands :
####       * sfs-dump (in non CSV format)
####       * filesys report generate file-location
####     So that a new CSV with the file paths appearing on both is created for all the data in sfs-dump file with the added tier and placement time information from location report
####
####     This script is not part of Dell PowerProtect / Data Domain, and hence it is not supported
####
####     Usage : extend-sfs-dump-with-tier.sh sfs-dump-output.csv file-location-output.log
####     Output : static "sfs-dump-output-tiers.csv" file name (may be changed below)

#### Replace values below to suit your needs
OUTPUT_FILENAME="sfs-dump-output-tiers.csv"
#### NO CHANGES NEEDED BEYOND THIS POINT

clear

if [ ! $# == 2 ]; then
    echo "Combine output from sfs-dump and tier location report into a CSV file with tier and placement time information"
    echo
    echo "Usage : $0 SFS-DUMP-OUTPUT-FILE    REPORT-FILE-LOCATION-FILE"
    echo "NOTE : SFS-DUMP-OUTPUT-FILE has to be in non-CSV format"
    exit 1
fi

INPUT_SFSDUMP="$1"
INPUT_LOCATION="$2"


echo -n "1/6 : Sanity checking input files..."
if [ ! -s "${INPUT_SFSDUMP}" ]; then
    echo "Input file ${INPUT_SFSDUMP} does not exist"
    exit 1
fi
if [ ! -s "${INPUT_LOCATION}" ]; then
    echo "Input file ${INPUT_LOCATION} does not exist"
    exit 1
fi
n_files_sfsdump=`grep ^/ ${INPUT_SFSDUMP} | wc -l`
n_files_location=`grep ^/ ${INPUT_LOCATION} | wc -l`
if [ ${n_files_sfsdump} -eq ${n_files_location} ]; then
    echo -n "both have the same amount of files (${n_files_location}). "
else
    echo -n "sfs-dump has ${n_files_sfsdump} files whereas location report has ${n_files_location} files, this may be normal if the difference is small. "
fi
echo "Done."


echo -n "2/6 : Sanitize \"file-location\" input..."
cat ${INPUT_LOCATION} | awk 'BEGIN {rejected="temp-location-rejected.log"; accepted="temp-location-accepted.log"} { if ( $0 ~ "Missing -unit") { gsub(/Missing -unit/, "Missing-Cloud-Unit", $0); print $0 > rejected } else { if ($0 ~ "^/" ) print $0 > accepted } }'
if [ -s "temp-location-rejected.log" ]; then
    REJECTS_EXIST="yes"
    echo -n "Some files in location report sit in unavailable or deleted cloud units, you may need to re-run this script after fixing the issue and gathering a new location report. "
    cat temp-location-rejected.log temp-location-accepted.log | sed -e 's/\t/:\t/' | sort > temp-location-report-sorted.log
    rm temp-location-rejected.log
else
    cat temp-location-accepted.log | sed -e 's/\t/:\t/' | sort > temp-location-report-sorted.log
    REJECTS_EXIST="no"
fi
rm temp-location-accepted.log
echo "Done."


echo -n "3/6 : Sanitize \"sfs-dump\" input..."
cat ${INPUT_SFSDUMP} | grep ^/ | sort > temp-sfs-dump.log
echo "Done."


echo -n "4/6 : Merging information for sfs-dump and location report..."
join -1 1 -2 1 temp-sfs-dump.log temp-location-report-sorted.log > temp-merged-information.log
rm temp-sfs-dump.log
rm temp-location-report-sorted.log
n_files_combined=`grep ^/ temp-merged-information.log | wc -l`
if [ ${n_files_combined} -eq 0 ]; then
    echo "No files matched from input files. sfs-dump output must NOT be in CSV format. Exiting."
    rm temp-merged-information.log
    exit 1
fi
echo -n "Input files matched on ${n_files_combined} files. "
echo "Done."


echo -n "5/6 : Converting merged sfs-dump / location-report output to CSV format..."
cat temp-merged-information.log | grep ^/ | grep -v ^$ | awk '
BEGIN {print "name\tmtime\tfileid\tsize\tseg_bytes\tseg_count\tredun_seg_count\tpre_lc_size\tpost_lc_size\ttier\tplacement_time"}
{
    colon_index = index($0, ":")
    filename = substr($0, 1, colon_index - 1)
    gsub(/^[[:space:]]+|[[:space:]]+$/, "", filename)
    n = split(substr($0, colon_index + 1), fields, " ")
    print filename "\t" fields[2] "\t" fields[4] "\t" fields[6] "\t" fields[10] "\t" fields[12] "\t" fields[14] "\t" fields[17] "\t" fields[19] "\t" fields[27] \
        "\t" fields[length(fields)-4] " " fields[length(fields)-3] " " fields[length(fields)-2] " " fields[length(fields)-1] " " fields[length(fields)]
}' > ${OUTPUT_FILENAME}
rm temp-merged-information.log
echo "Done."


echo -n "6/6 : Summary"
echo
echo
echo "Merging information from sfs-dump (${INPUT_SFSDUMP}) and location-report ${INPUT_LOCATION} output completed."
echo "Output file (${OUTPUT_FILENAME}) includes information for a total ${n_files_combined} files, out of ${n_files_sfsdump} in input sfs-dump, and ${n_files_location} in input location report."
if [ "${REJECTS_EXIST}" == "yes" ]; then
    echo "Note there are some files in disconnected or deleted cloud units, for which the \"tier\" field has been replaced with \"Missing-Cloud-Unit\"."
fi
echo
echo "Start of file shown below for your convenience :"
echo "===================="
head -5 ${OUTPUT_FILENAME}
echo "===================="
echo
echo "You may follow the instructions in https://www.dell.com/support/kbdoc/000081345 to process this CSV file in an spreadhseet"
echo
echo "Done."

exit 0

對於 Veritas NetBackup 使用者：

/data/col1/MTREE_NAME/POLICY-NAME_1400502741_C1_F1:1400502741:VM_PRD-02:4:1::
/data/col1/MTREE_NAME/POLICY-NAME_1400502741_C1_F2:1400502741:VM_PRD-02:4:1::
/data/col1/MTREE_NAME/POLICY-NAME_1400502741_C1_HDR:1400502741:VM_PRD-02:4:1::

--- iterate-dd-for-fs-sfs-dump.sh       2024-01-23 06:32:16.521409000 -0500
+++ iterate-dd-for-fs-sfs-dump-NBU.sh   2024-02-27 03:26:42.808246000 -0500
@@ -55,11 +55,11 @@
 cat sfs-dump-noCSV.txt | grep ^/ | grep -v ^$ | awk '
 BEGIN {print "name\tmtime\tfileid\tsize\tseg_bytes\tseg_count\tredun_seg_count\tpre_lc_size\tpost_lc_size"}
 {
-    colon_index = index($0, ":")
-    filename = substr($0, 1, colon_index - 1)
+    colon_index = index($0, ":::")
+    filename = substr($0, 1, colon_index + 1)
     gsub(/^[[:space:]]+|[[:space:]]+$/, "", filename)
     n = split(substr($0, colon_index + 1), fields, " ")
-    print filename "\t" fields[2] "\t" fields[4] "\t" fields[6] "\t" fields[10] "\t" fields[12] "\t" fields[14] "\t" fields[17] "\t" fields[19]
+    print filename "\t" fields[3] "\t" fields[5] "\t" fields[7] "\t" fields[11] "\t" fields[13] "\t" fields[15] "\t" fields[18] "\t" fields[20] "\t"
 }' > sfs-dump-CSV.csv
 echo "Done."