Isilon A200 1 node getting full

When you have an Isilon cluster with unevenly distributed data across the nodes, and auto-balance jobs are not effectively balancing the data on one specific node, there are a few steps you can take to investigate and potentially address the issue:

1. Check Auto-Balance Settings:

   • Ensure that the auto-balance feature is enabled and configured correctly on your Isilon cluster. You can do this through the OneFS web interface.

2. Review Data Distribution Policies:

   • Check the data distribution policies for your file pools. Verify that the default policy is distributing data evenly across all nodes in the cluster.
   • Consider creating custom file pool policies if necessary to ensure balanced data distribution.

3. Check Data Placement Rules:

   • Review the data placement rules to ensure that they are not preventing data from moving to the affected node.

4. Review Data Usage:

   • Investigate the data usage patterns on the cluster, especially on the node that is 87% full. Identify which directories or shares are consuming the most space. It's possible that certain data types or applications are causing this imbalance.

5. Data Mobility Policies:

   • Make sure that data mobility policies are not restricting data movement on the affected node. Check if there are specific policies preventing data from being moved.

6. Run Manual Balance Jobs:

   • Consider running manual balance jobs targeting the specific node that is almost full. You can use the isi job start balance command to initiate a manual balance job for a specific node.

7. Check for Errors or Alerts:

   • Review the Isilon cluster logs and event notifications for any errors or alerts related to the affected node. This may provide insights into why data isn't balancing as expected.

8. Monitor Data Growth:

   • Keep an eye on data growth patterns over time. It's possible that the data on the affected node is growing faster than on the other nodes, which can lead to imbalance.

9. Contact Isilon Support:

   • If you are unable to resolve the issue through the above steps, consider reaching out to Dell EMC Isilon support for assistance. They can provide specialized guidance and may be able to diagnose and address the issue more effectively.

Remember that data balancing in an Isilon cluster can take time, and it's important to monitor the situation periodically to ensure that the data distribution remains balanced across all nodes.

@ssmith2075 , what does default File Pool Policy say? is H500 supposed to be for hot data and after x amount of time, it goes to A200?

@Phil.Lam​    this is the only policy in place

@Gatto Sama , point the default storage  target to H500 (hot data)  and then use a file policy to tier it to A2000 (cold data)

@ssmith2075 did you ever find the root cause of this?  Running  OneFS 9.4.0.17, we have a pool with 8  A3000 nodes and while all are at 66-69%, one of them is at 84%.. and some disks on that node at 99%...(5 disks out 22 at 99). I am running autobalancelin to hopefully get back to an even pool.. but now seeing your comment that even after autobalance jobs that node is not balanced is worrying me... anyway keep you posted on my results, but any feedback is appreciated. 

@ssmith2075  Turns out we had node45 in a read-only state due to BBU issue... ad dell explained this can prevent pending deletes on the node hence uneven capacity.. also to reclaim the space on the node we are now running a collect job, which will be followed by  snapdelete...after which we can run autobalance.