It’s important to maintain enough free space on your EMC® Isilon® cluster to ensure that data is protected and workflows are not disrupted. At a minimum, you should have at least one node’s worth of free space available in case you need to protect data on a failing drive.
When your Isilon cluster fills up to more than 90% capacity, cluster performance is affected. Several issues can occur when your cluster fills up to 98% capacity, such as substantially slower performance, failed file operations, the inability to write or delete data, and the potential for data loss. It might take several days to resolve these issues. If you have a full cluster, nearly full cluster, or need assistance with maintaining enough free space, contact EMC Isilon Technical Support.
Fortunately, there are several best practices you can follow to help prevent your Isilon cluster from becoming too full. These are detailed in the “Best Practices Guide for Maintaining Enough Free Space on Isilon Clusters and Pools” (requires login to the EMC Online Support site). Some of these best practices are summarized in this blog post.
Monitoring cluster capacity
To prevent your cluster from becoming too full, monitor your cluster capacity. There are several ways to do this. For example, you can configure email event notification rules in the EMC Isilon OneFS® operating system to notify you when your cluster is reaching capacity. Watch the video “How to Set Up Email Notifications in OneFS When a Cluster Reaches Capacity” for a demonstration of this procedure.
[youtube_sc url=”http://www.youtube.com/watch?v=k6DCFJTtrjQ&index=18&list=PLbssOJyyvHuXZ_3JKT5ugbuHPQqZm3e5f” playlist=”How to Set Up Email Event Notifications in OneFS When a Cluster Reaches Capacity”]
Another way to monitor cluster capacity is to use EMC Isilon InsightIQ™ software. If you have InsightIQ licensed on your cluster, you can run FSAnalyze jobs in OneFS to create data for InsightIQ’s file system analytics tools. You can then use InsightIQ’s Dashboard and Performance Reporting to monitor cluster capacity. For example, Performance Reports enable you to view information about the activity of the nodes, networks, clients, disks, and more. The Storage Capacity section of a performance report displays the used and total storage capacity for the monitored cluster over time (Figure 1).
For more information about InsightIQ Performance Reports, see the InsightIQ User Guides, which can be found on the EMC Online Support site.
To learn about additional ways to monitor cluster capacity, such as using SmartQuotas, read “Best Practices Guide for Maintaining Enough Free Space on Isilon Clusters and Pools.”
More best practices
Follow these additional tips to maintain enough free space on your cluster:
- Manage your data
Regularly delete data that is rarely accessed or used. - Manage Snapshots
Snapshots, which are used for data protection in OneFS, can take up space if they are no longer needed. Read the best practices guide for several best practices about managing snapshots, or read the blog post “EMC Isilon SnapshotIQ: An overview and best practices.” - Make sure all nodes in a node pool or disk pool are compatible
If you have a node pool that contains a mix of different node capacities, you can receive “cluster full” errors even if only the smallest node in your node pool reaches capacity. To avoid this scenario, ensure that nodes in each node pool or disk pool are of compatible types. Read the best practices guide for information about node compatibility and for a procedure to verify that all nodes in each node pool are compatible. - Enable Virtual Hot Spare
Virtual Hot Spare (VHS) keeps space in reserve in case you need to move data off of a failing drive (smartfail). VHS is enabled by default. For more information about VHS, read the knowledgebase article, “OneFS: How to enable and configure Virtual Hot Spare (VHS) (88964)” (requires login to the EMC Online Support site). - Enable Spillover
Spillover allows data that is being sent to a full pool to be diverted to an alternate pool. If you have licensed EMC Isilon SmartPools™ software, you can designate a spillover location. For more information about SmartPools, read the OneFS Web Administration Guide. - Add nodes
If you want to scale-out your storage to add more free space, contact your sales representative.
If you have questions or feedback about this blog or video described in it, send an email to isi.knowledge@emc.com. To provide documentation feedback or request new content, send an email to isicontent@emc.com.