Isilon: Error: Disk Quota Exceeded when writing files to the cluster

When writing files to the cluster, the error "Disk Quota Exceeded" may appear. No quotas limits have been set or, if they have, they have not been reached.

There are multiple possible causes for this error. Three of the possible causes are covered in this article:

1. The error message is misleading because it is triggered when the SnapshotIQ reserve percentage has been reached.
2. The quota had linked users, including the root user, and the Network File System (NFS) export is being mounted as a root client.
3. The quota for the owner of the directory being written to has been exceeded, so the directory cannot be grown.

Cause #1:

If the SnapshotIQ reserve is set to a non-zero value (such as 1 percent, 5 percent, or 10 percent), it diminishes general storage availability. Use caution when setting reserve values, because this setting directly affects total cluster storage capacity.

If this error occurs and disk quotas have not been exceeded, most likely a high reserve has been specified for SnapshotIQ that limit writes.

Note: This issue might occur regardless of whether or not SmartQuotas is licensed on the cluster. Also, it may occur regardless of whether or not a quota limitation has been reached.

Cause #2:

If creating an All-Users directory quota, and the directory is exported for Root Client access, the root directory for that export is accessible by root clients. They may get a "Disk quota exceeded" message regardless of the quota usage for the specific user. One way to see this is by trying to reset the notification state for the directory:

isilon-9# isi_classic quota modify --path=/ifs/storage/scratch01 --reset-notify-state
Pretending to modify:                                                                                                               
    user:root@snaps=no@/ifs/storage/scratch01                                FAIL
    !! Cannot modify/save linked domains without unlinking: Invalid argument

You can see that the quota has been exceeded for the root user by querying the list of quotas:

isilon-9# isi quota quotas list --exceeded
Type  AppliesTo  Path                                    Snap  Hard    Soft  Adv  Used   
---------------------------------------------------------------------------------------------
user  user1      /ifs/storage/scratch_overflow/scratch04 No    275.00G -     -    275.000G
user  root       /ifs/storage/scratch01                  No    500.00G -     -    500.032G
user  user2      /ifs/storage/scratch01                  No    500.00G -     -    500.005G
user  user3      /ifs/storage/scratch01                  No    655.00G -     -    655.001G

If you review the exports for that directory, you can see if it is configured for 'root clients':

--------------------------------------------------------------------------------
                     ID: 2
                  Paths: /ifs/storage/scratch01
            Description: scratch
                Clients: -
           Root Clients: user1,user2,user3
      Read Only Clients: -
     Read Write Clients: -
     Unresolved Clients: -
               All Dirs: Yes
             Block Size: 8.0K
           Can Set Time: Yes
    Commit Asynchronous: No
Directory Transfer Size: 128.0K
               Encoding: DEFAULT
         Map Lookup UID: No
              Map Retry: Yes
                Map All
                       User : -
                     Groups : -
               Map Root
                       User : root
                     Groups : nogroup
               Map Full: Yes

Cause #3:

When creating an All-Users quota and the owner of the directory has exceeded their quota, no users can write new files to the directory. This is due to the requirement to grow the directory when new files are added or created. Quotas are charged against the owner of an object (file or directory). In this case, the owner of the directory does not have enough quota to enlarge the directory.

You can find the owner of a directory by using the ls command, or, to include the complete Access Control List (ACL), ls -led:

isilon-1# pwd
/ifs/data/smbsharebak
isilon-1# ls -led
drwxrwxrwx    2 admin  wheel  532 Oct  6 15:26 .
 OWNER: user:admin
 GROUP: group:wheel
 SYNTHETIC ACL
 0: user:admin allow dir_gen_read,dir_gen_write,dir_gen_execute,std_write_dac,delete_child
 1: group:wheel allow dir_gen_read,dir_gen_write,dir_gen_execute,delete_child
 2: everyone allow dir_gen_read,dir_gen_write,dir_gen_execute,delete_child

Note: This owner is dependent on who created the directory, which is often dependent on how data was migrated to the cluster or path. This could be any user including root, or even a group like DOMAIN\Domain Admins. A group owner can be inherited, or applied due to freeBSD Semantics being in place when the directory is created. If this owner is over quota, the directory cannot be grown, so new files cannot be added or created.

Use the --exceeded option to identify whether this owner is over quota:

isilon-1# isi quota quotas list --exceeded
Type      AppliesTo  Path                  Snap  Hard   Soft  Adv  Used
---------------------------------------------------------------------------
directory DEFAULT    /ifs/data/nfsexport   No    30.00M -     -    114.03M
user      admin      /ifs/data/smbsharebak No    80.00M -     -    1000.00M
---------------------------------------------------------------------------
Total: 2

For Cause #1:

To resolve this issue, set the snapshot reserve percentage to 0 (the default value).

Note: If you must set a reserve for SnapshotIQ, reduce the amount from its current setting until the "Disk Quota Exceeded" error no longer appears.

Use the command-line interface.

1. Run the following command to check the value of the snapshot reserve percentage:
   

   isi snapshot settings view

2. If the snapshot reserve percentage is anything other than zero, change the reserve by running the following command:
   

   isi snapshot settings modify --reserve 0

Use the Isilon web administration interface.

1. Open the Isilon web administration interface.
2. Click File System > SnapshotIQ > Settings.
3. If the value in the Reserve box is anything other than zero, delete that value and type zero(0).
4. Click Submit.

For Cause #2:

Unlink the root user from the quota and then remove the quota from that user to remedy the issue.

To do this in the Isilon web administration interface:

1. Click File System > SmartQuotas > Quotas & Usage.
2. The Quotas & Usage listing include the Linked User under the All Users listing for the quota.
3. Click the Unlink link to the right of the root user to unlink the quota from root.
4. View details for the root user and choose Edit usage limits.

You should now be able to save files in the root of the export.

For Cause #3:

Do any one of the following things to address the issue:

• Remove or raise the quota restrictions for that owner.
• If that owner is part of a linked quota, unlink the owner and change the quota for that unlinked user
• Change directory ownership
• Clean up resources used by that owner for that path.