PowerScale: Unable to remove files when hard quota is exceeded over NFS with error "No space left on device"

Summary: NFS clients are unable to remove data if the hard quota is exceeded with the error "No space left on device."

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.
Check out other resources

Symptoms

# isi quota quotas list
Type         AppliesTo  Path     Snap  Hard    Soft  Adv  Used    Efficiency
-----------------------------------------------------------------------------
default-user DEFAULT    /ifs/nfs No    100.00M -     -    0.00    -
user         root       /ifs/nfs No    100.00M -     -    195.00M 0.50 : 1  <<<<<<< Exceeded user quota
-----------------------------------------------------------------------------
Total: 2


[root@centos tmp]# rm -f testfile2
rm: cannot remove ‘testfile2’: No space left on device


In packet captures, NFS3ERR_NOSPC can be seen:
 

NFS 266 V3 RENAME Call (Reply In 121), From DH: 0x73953be7/testfile123 To DH: 0x73953be7/.nfs000000010001329500000009 

NFS 162 V3 RENAME Reply (Call In 120) Error: NFS3ERR_NOSPC

 

Cause

The file is in use which is causing a .nfs file to be written until the file is no longer in use. Because the quota is exceeded the .nfs file fails to be written. This is considered as a silly rename and is something the client does, not the server. 

NFS 266 V3 RENAME Call (Reply In 121), From DH: 0x73953be7/testfile123 To DH: 0x73953be7/.nfs000000010001329500000009 <<<<< can see Client tries to make a .nfs file

And fails because the quota is exceeded 

NFS 162 V3 RENAME Reply (Call In 120) Error: NFS3ERR_NOSPC

Resolution

Usually running lsof on the client identifies which process has the file open. However, in at least one instance, CrowdStrike on the client created the .nfs file but prevented lsof from showing it as open. Temporarily disabling CrowdStrike allowed the file to be removed. If the .nfs file appears in packet captures, that confirms a client-side process is using the file, which prevents its removal when a hard quota is exceeded.

[root@centos tmp]# lsof testfile123
COMMAND   PID USER   FD   TYPE DEVICE  SIZE/OFF       NODE NAME
tail    13992 root    3r   REG   0,43 104857600 4337893434 testfile123

Once the process that has the file in use is stopped the file can then be removed

[root@centos tmp]# rm -f testfile123
[root@centos tmp]#

Additional Information

What is a "silly rename"? Why do these .nfsXXXXX files keep showing up?

Affected Products

Isilon, PowerScale OneFS, Isilon Scale-out NAS
Article Properties
Article Number: 000202922
Article Type: Solution
Last Modified: 27 Nov 2025
Version:  3
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.