Data Rebalancing within the same tier across different node pools

Hey guys,

Can anyone let me know what would be the quickest way to drop the utilization of the nodes underlined below? AutoBalanceLin is very slow..

ID |IP Address |DASR | In Out Total| Used / Size |Used / Size
---+---------------+-----+-----+-----+-----+-----------------+-----------------
1|10.170.95.60 | OK |603.2| 0|603.2|(No Storage HDDs)|18.4T/34.6T( 53%)
2|10.170.95.61 | OK | 0| 124k| 124k|(No Storage HDDs)|18.4T/34.6T( 53%)
3|10.170.95.62 | OK | 896k| 177k| 1.1M|(No Storage HDDs)|18.4T/34.6T( 53%)
4|10.170.95.63 | OK | 0| 403k| 403k|(No Storage HDDs)|18.4T/34.6T( 53%)
5|10.170.95.64 |-A-- | 0|49.7k|49.7k|38.7T/43.2T( 90%)|54.8G/ 5.8T(< 1%)
7|10.170.95.66 | OK | 111M| 141k| 111M|67.5T/ 173T( 39%)|92.1G/ 5.8T( 2%)
8|10.170.95.67 | OK | 3.2M|64.0M|67.2M|67.5T/ 173T( 39%)|92.1G/ 5.8T( 2%)
9|10.170.95.68 | OK | 169M| 488M| 657M|38.6T/43.2T( 89%)|54.9G/ 5.8T(< 1%)
10|10.170.95.69 | OK | 0| 380M| 380M|38.8T/43.2T( 90%)|54.9G/ 5.8T(< 1%)
11|10.170.95.70 | OK | 100k|29.4M|29.5M|38.8T/43.2T( 90%)|54.8G/ 5.8T(< 1%)
12|10.170.95.71 | OK | 0| 1.3M| 1.3M|38.7T/43.2T( 89%)|54.8G/ 5.8T(< 1%)
13|10.170.95.72 | OK | 8.5G| 6.0G|14.5G|38.7T/43.2T( 90%)|54.8G/ 5.8T(< 1%)
14|10.170.95.73 | OK | 0| 282k| 282k|38.8T/43.2T( 90%)|54.8G/ 5.8T(< 1%)
15|10.170.95.74 | OK | 0| 149k| 149k|38.8T/43.2T( 90%)|54.8G/ 5.8T(< 1%)
16|10.170.95.75 | OK | 0| 9.5M| 9.5M|38.8T/43.2T( 90%)|54.8G/ 5.8T(< 1%)
17|10.170.95.76 | OK | 0|12.2M|12.2M|38.8T/43.2T( 90%)|54.9G/ 5.8T(< 1%)
18|10.170.95.77 | OK |92.9M| 176k|93.0M|38.7T/43.2T( 90%)|54.8G/ 5.8T(< 1%)
19|10.170.95.78 | OK | 156k|88.7M|88.9M|38.7T/43.2T( 89%)|54.8G/ 5.8T(< 1%)
20|10.170.95.79 | OK | 6.6G| 5.2G|11.8G|38.6T/43.2T( 89%)|54.8G/ 5.8T(< 1%)
21|10.170.95.80 | OK | 0| 169M| 169M|69.6T/86.5T( 80%)| 103G/ 5.8T( 2%)
22|10.170.95.81 | OK | 7.0M| 196M| 203M|69.6T/86.5T( 80%)| 102G/ 5.8T( 2%)
23|10.170.95.82 | OK |78.3k| 0|78.3k|69.9T/86.5T( 81%)| 103G/ 5.8T( 2%)
24|10.170.95.83 | OK |41.9M|70.8M| 113M|69.7T/86.5T( 81%)| 102G/ 5.8T( 2%)
25|10.170.95.84 | OK | 662M| 337M| 998M|69.8T/86.5T( 81%)| 103G/ 5.8T( 2%)
26|10.170.95.85 | OK |14.0M| 4.8G| 4.8G|69.6T/86.5T( 80%)| 102G/ 5.8T( 2%)
27|10.170.95.86 | OK |27.2k| 117M| 117M|69.6T/86.5T( 80%)| 103G/ 5.8T( 2%)
28|10.170.95.87 | OK |28.2M| 445M| 473M|69.5T/86.5T( 80%)| 102G/ 5.8T( 2%)
29|10.170.95.88 | OK |87.9M| 1.5G| 1.6G| 150T/ 173T( 87%)| 189G/ 5.8T( 3%)
30|10.170.95.89 | OK | 0| 188k| 188k| 150T/ 173T( 87%)| 190G/ 5.8T( 3%)
31|10.170.95.90 | OK | 0| 119k| 119k| 150T/ 173T( 87%)| 189G/ 5.8T( 3%)
32|10.170.95.91 | OK | 4.9M| 249k| 5.1M| 150T/ 173T( 87%)| 183G/ 5.8T( 3%)
---+---------------+-----+-----+-----+-----+-----------------+-----------------
Cluster Totals: |15.1G|21.1G|36.2G| 1.8P/ 2.2P( 78%)|76.0T/ 294T( 26%)

dubhpc312-22# isi storagepool nodepools list
ID Name Nodes Node Type IDs Protection Policy Manual
--------------------------------------------------------------------------------
1 f40_0b_48tb-ssd_256gb 1 +2d:1n No
1
2
3
4
9 h40_60tb_6.4tb-ssd_128gb 3 +2d:1n No
5
9
10
11
12
13
14
15
16
17
18
19
20
54 h40_120tb_6.4tb-ssd_128gb 2 +2d:1n No
21
22
23
24
25
26
27
28
91 h500_240tb_6.4tb-ssd_128gb 4 +2d:1n No
7
8
29
30
31
32
--------------------------------------------------------------------------------
Total: 4

dubhpc312-22# isi storagepool tiers list -v
ID: 31
Name: T1-SSD
Nodes: 1, 2, 3, 4
Children: f40_0b_48tb-ssd_256gb
Usage
Avail Bytes: 64.67T
Avail SSD Bytes: 64.67T
Balanced: Yes
Free Bytes: 99.18T
Free SSD Bytes: 99.18T
Total Bytes: 172.75T
Total SSD Bytes: 172.75T
Virtual Hot Spare Bytes: 34.52T
--------------------------------------------------------------------------------
ID: 42
Name: T2-SAS
Nodes: 5, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32
Children: h40_60tb_6.4tb-ssd_128gb, h40_120tb_6.4tb-ssd_128gb, h500_240tb_6.4tb-ssd_128gb
Usage
Avail Bytes: 649.50T
Avail SSD Bytes: 153.06T
Balanced: No
Free Bytes: 1.19P
Free SSD Bytes: 153.06T
Total Bytes: 2.95P
Total SSD Bytes: 155.47T
Virtual Hot Spare Bytes: 572.52T

dubhpc312-22# isi filepool policies list -v
Name: dubddt_data_to_T1
Description: Move subset of dubddt_data to T1 (F nodes)
CloudPools State: No access
CloudPools Details: Policy has no CloudPools actions
Apply Order: 1
File Matching Pattern: Path == ic/dubddt_data/SDMv2
OR
Path == ic/dubddt_data/DDT/WEC
OR
Path == ic/dubddt_data/Gazelle/BasePlatform3
Set Requested Protection: -
Data Access Pattern: concurrency
Enable Coalescer: Yes
Enable Packing: -
Data Storage Target: T1-SSD
Data SSD Strategy: data
Snapshot Storage Target: T2-SAS
Snapshot SSD Strategy: metadata
Cloud Pool: -
Cloud Compression Enabled: -
Cloud Encryption Enabled: -
Cloud Data Retention: -
Cloud Incremental Backup Retention: -
Cloud Full Backup Retention: -
Cloud Accessibility: -
Cloud Read Ahead: -
Cloud Cache Expiration: -
Cloud Writeback Frequency: -
ID: dubddt_data_to_T1
--------------------------------------------------------------------------------
Name: dubgaz_grid_to_T1
Description: Move dubgaz_grid to T1 (F nodes)
CloudPools State: No access
CloudPools Details: Policy has no CloudPools actions
Apply Order: 2
File Matching Pattern: Path == ic/dubgaz_grid
Set Requested Protection: -
Data Access Pattern: concurrency
Enable Coalescer: Yes
Enable Packing: -
Data Storage Target: T1-SSD
Data SSD Strategy: data
Snapshot Storage Target: T2-SAS
Snapshot SSD Strategy: metadata
Cloud Pool: -
Cloud Compression Enabled: -
Cloud Encryption Enabled: -
Cloud Data Retention: -
Cloud Incremental Backup Retention: -
Cloud Full Backup Retention: -
Cloud Accessibility: -
Cloud Read Ahead: -
Cloud Cache Expiration: -
Cloud Writeback Frequency: -
ID: dubgaz_grid_to_T1
--------------------------------------------------------------------------------
Name: dubddt_scratch_to_T1
Description: Move dubddt_scratch to T1 (F nodes)
CloudPools State: No access
CloudPools Details: Policy has no CloudPools actions
Apply Order: 3
File Matching Pattern: Path == ic/dubddt_scratch
Set Requested Protection: -
Data Access Pattern: concurrency
Enable Coalescer: Yes
Enable Packing: -
Data Storage Target: T1-SSD
Data SSD Strategy: data
Snapshot Storage Target: T2-SAS
Snapshot SSD Strategy: metadata
Cloud Pool: -
Cloud Compression Enabled: -
Cloud Encryption Enabled: -
Cloud Data Retention: -
Cloud Incremental Backup Retention: -
Cloud Full Backup Retention: -
Cloud Accessibility: -
Cloud Read Ahead: -
Cloud Cache Expiration: -
Cloud Writeback Frequency: -
ID: dubddt_scratch_to_T1
--------------------------------------------------------------------------------
Name: dubquant_auc_cache_to_T1
Description: Purpose: a fast filesystem access for caching hdf5 files. Avoid being constrained by other people's usage on scratch_data_01. Ref DISSD-984
CloudPools State: No access
CloudPools Details: Policy has no CloudPools actions
Apply Order: 4
File Matching Pattern: Path == ic/dubquant_auc_cache
Set Requested Protection: -
Data Access Pattern: concurrency
Enable Coalescer: Yes
Enable Packing: -
Data Storage Target: T1-SSD
Data SSD Strategy: data
Snapshot Storage Target: -
Snapshot SSD Strategy: -
Cloud Pool: -
Cloud Compression Enabled: -
Cloud Encryption Enabled: -
Cloud Data Retention: -
Cloud Incremental Backup Retention: -
Cloud Full Backup Retention: -
Cloud Accessibility: -
Cloud Read Ahead: -
Cloud Cache Expiration: -
Cloud Writeback Frequency: -
ID: dubquant_auc_cache_to_T1
--------------------------------------------------------------------------------
Name: dubquant_fe_to_T1
Description: Move dubquant_fe to T1 (F nodes)
CloudPools State: No access
CloudPools Details: Policy has no CloudPools actions
Apply Order: 5
File Matching Pattern: Path == ic/dubquant_fe
Set Requested Protection: -
Data Access Pattern: concurrency
Enable Coalescer: Yes
Enable Packing: -
Data Storage Target: T1-SSD
Data SSD Strategy: data
Snapshot Storage Target: T2-SAS
Snapshot SSD Strategy: metadata
Cloud Pool: -
Cloud Compression Enabled: -
Cloud Encryption Enabled: -
Cloud Data Retention: -
Cloud Incremental Backup Retention: -
Cloud Full Backup Retention: -
Cloud Accessibility: -
Cloud Read Ahead: -
Cloud Cache Expiration: -
Cloud Writeback Frequency: -
ID: dubquant_fe_to_T1
--------------------------------------------------------------------------------
Name: dubquant_conda_to_T1
Description: Move dubquant_conda to T1 (F nodes)
CloudPools State: No access
CloudPools Details: Policy has no CloudPools actions
Apply Order: 6
File Matching Pattern: Path == ic/dubquant_conda
Set Requested Protection: -
Data Access Pattern: concurrency
Enable Coalescer: Yes
Enable Packing: -
Data Storage Target: T1-SSD
Data SSD Strategy: data
Snapshot Storage Target: T2-SAS
Snapshot SSD Strategy: metadata
Cloud Pool: -
Cloud Compression Enabled: -
Cloud Encryption Enabled: -
Cloud Data Retention: -
Cloud Incremental Backup Retention: -
Cloud Full Backup Retention: -
Cloud Accessibility: -
Cloud Read Ahead: -
Cloud Cache Expiration: -
Cloud Writeback Frequency: -
ID: dubquant_conda_to_T1
--------------------------------------------------------------------------------
Name: dummy_share_to_T1
Description: move dummy_share to T1 ( F Nodes)
CloudPools State: No access
CloudPools Details: Policy has no CloudPools actions
Apply Order: 7
File Matching Pattern: Path == ic/dummy_share
Set Requested Protection: -
Data Access Pattern: -
Enable Coalescer: -
Enable Packing: -
Data Storage Target: T1-SSD
Data SSD Strategy: metadata-write
Snapshot Storage Target: T1-SSD
Snapshot SSD Strategy: metadata
Cloud Pool: -
Cloud Compression Enabled: -
Cloud Encryption Enabled: -
Cloud Data Retention: -
Cloud Incremental Backup Retention: -
Cloud Full Backup Retention: -
Cloud Accessibility: -
Cloud Read Ahead: -
Cloud Cache Expiration: -
Cloud Writeback Frequency: -
ID: dummy_share_to_T1