PowerScale: One cluster in a SyncIQ relationship reports more used disk space than the other cluster.

One PowerScale cluster in a syncIQ relationship (source or target) has more used disk space than the other PowerScale cluster. Both clusters should have identically used space.

Example, as seen with the 'isi status' command:

Target cluster shows that 349T used hard drive (HDD) storage:

Cluster Name: TARGET-POWERSCALE
Cluster Health: [ ATTN]
Data Reduction:     1.00 : 1
Storage Efficiency: 0.73 : 1
Cluster Storage:  HDD                 SSD Storage
Size:             417.2T (432.5T Raw) 5.7T (5.7T Raw)
VHS Size:         15.4T
Used:             348.6T (84%)        669.1G (11%)    ==================>
Avail:            68.6T (16%)         5.1T (89%)

                   Health  Throughput (bps)  HDD Storage      SSD Storage
ID |IP Address     |DASR |  In   Out  Total| Used / Size     |Used / Size
---+---------------+-----+-----+-----+-----+-----------------+-----------------
  9|1099.2.6      | OK  | 1.0M| 488k| 1.5M|87.2T/ 104T( 84%)| 167G/ 1.4T( 11%)
 10|10.99.2.7      | OK  | 820M| 4.2M| 824M|87.2T/ 104T( 84%)| 167G/ 1.4T( 11%)
 11|10.99.2.8      | OK  | 678M| 2.9M| 681M|87.2T/ 104T( 84%)| 167G/ 1.4T( 11%)
 12|10.99.2.9      | OK  |    0| 123k| 123k|87.1T/ 104T( 84%)| 167G/ 1.4T( 11%)
---+---------------+-----+-----+-----+-----+-----------------+-----------------
Cluster Totals:          | 1.5G| 7.7M| 1.5G| 349T/ 417T( 84%)| 669G/ 5.7T( 11%)

     Health Fields: D = Down, A = Attention, S = Smartfailed, R = Read-Only

Source only has 286T used HDD storage:
 

Cluster Name: SOURCE-POWERSCALE
Cluster Health:     [ ATTN]
Data Reduction:     1.00 : 1
Storage Efficiency: 0.71 : 1
Cluster Storage:  HDD                 SSD Storage
Size:             417.2T (432.5T Raw) 5.7T (5.7T Raw)
VHS Size:         15.4T
Used:             285.5T (68%)        721.9G (12%)    ========>
Avail:            131.7T (32%)        5.0T (88%)

                   Health  Throughput (bps)  HDD Storage      SSD Storage
ID |IP Address     |DASR |  In   Out  Total| Used / Size     |Used / Size
---+---------------+-----+-----+-----+-----+-----------------+-----------------
  9|10.99.3.13     | OK  | 290M| 9.9M| 300M|71.4T/ 104T( 68%)| 180G/ 1.4T( 12%)
 10|10.99.3.11     | OK  | 324M| 832M| 1.2G|71.4T/ 104T( 68%)| 181G/ 1.4T( 12%)
 11|10.99.3.12     | OK  | 194M| 8.4M| 202M|71.4T/ 104T( 68%)| 181G/ 1.4T( 12%)
 12|10.99.3.10     | OK  | 1.9M| 2.0M| 3.8M|71.4T/ 104T( 68%)| 181G/ 1.4T( 12%)
---+---------------+-----+-----+-----+-----+-----------------+-----------------
Cluster Totals:          | 809M| 852M| 1.7G| 286T/ 417T( 68%)| 722G/ 5.7T( 12%)

Common causes for differences in space between clusters in a syncIQ relationship are as follows:

--large system files or audit logs which live within /ifs/.ifsvar (most common)
--differences in snapshot size
--differences in protection levels
--Collect job not running on one cluster, thus not freeing up orphaned disk blocks.
--large support-related files/folders living within /ifs/data/Isilon_Support

Assuming all syncIQ jobs are up to date with replication, check the following:

1) It is often the case that the cluster using more space has large system files or audit logs that live within /ifs/.ifsvar.

On each cluster, from a screen session (as the command may take a long time to return), run the following:
 

# du -sh /ifs/.ifsvar

Run that command on each cluster.

PowerScale Support has seen this to be the culprit in the past, especially due to with large audit logs.

Check the /ifs/.ifsvar/audit directory and the following subdirectories,

 Where <nodeXXX> is the node ID (for example node001):/ifs/.ifsvar/audit/logs/.
 /ifs/.ifsvar/audit/logs/<nodeXXX>
/ifs/.ifsvar/audit/logs/<nodeXXX>/protocol

If needed, delete the audit files by following article:

KB 000167091Powerscale: How to Remove Audit Log Files

2) confirm that there are no differences in the protection level between the two clusters.

Live cluster:
 

isi stat -p -q -v

Logs:
 

# cat <base log set>/local/isi_stat-p

Dell Technologies can also check the diskpools protection levels with an internal command with Support assistance.

3) Confirm that there is no big difference with snapshot usage between the two clusters by running the following command on each cluster:
 

# isi snapshot usage

4) On the cluster reporting more used space, check for any large support-related files/folders living within /ifs/data/Isilon_Support:

# du -so /ifs/data/Isilon_Support 

5) If still stuck, the next step is to check when Collect was last run on each cluster.

(Note: It is important to note that the 'MultiScan' does not always run the Collect job). 

This Collect job frees up any stale disk blocks on the cluster and can often be the cause as to why one cluster shows more used disk space than the other cluster.  One quick way to do this is to check the /var/log/messages on the nodes for the last time Collect successfully ran, for example:

# grep Collect /var/log/messages|grep Succeed
# zgrep Collect /var/log/messages* 

 6) If still stuck, run FSAnalyze job and use InsightIQ to see which folder is taking up more space on the cluster with more used space.