VNX: server_df command fails
Summary: VNX: server_df command fails
Symptoms
The command 'server_df server_2' fails with "Error 2: server_2 : No such file or directory failed to complete command":
[nasadmin@ssgnas03cs0 ~]$ server_df server_2 server_2 : Error 2: server_2 : No such file or directory failed to complete command
Pre-upgrade health checks fail with the following errors:
Blades : Check if root filesystem is too large Error HC_DM_14505148436: Output from command "/nas/bin/server_df server_2" has no information on file system root_fs_2 Blades : Check if root filesystem has enough free space Error HC_DM_14505148440: Output from command "/nas/bin/server_df server_2" has no information on file system root_fs_2 Blades : Check if file system usage is under limit Error HC_DM_14505148563: Could not determine available disk space for the mounted file systems on the following Blades: * server_2
All other server_df commands work successfully against other data movers:
[nasadmin@ssgnas03cs0 ~]$ server_df server_3 server_3 : Filesystem kbytes used avail capacity Mounted on root_fs_common 15368 5280 10088 34% /.etc_common root_fs_3 258128 6184 251944 2% /
Cause
The root cause in this instance was that the root file system for server_2 was unmounted.
[nasadmin@ssgnas03cs0 ~]$ server_mount ALL server_2 : <------ nothing mounted server_3 : root_fs_3 on / uxfs,perm,rw root_fs_common on /.etc_common uxfs,perm,ro
When you list all file systems, it was clear that the root file system 'root_fs_2' for server_2 was listed as not in use because it was unmounted.
[nasadmin@ssgnas03cs0 ~]$ nas_fs -l id inuse type acl volume name server 1 n 1 0 10 root_fs_1 2 y 1 0 40 root_fs_common 3,2 3 n 5 0 73 root_fs_ufslog 4 n 5 0 76 root_panic_reserve 5 n 5 0 93 root_fs_d3 6 n 5 0 94 root_fs_d4 7 n 5 0 95 root_fs_d5 8 n 5 0 96 root_fs_d6 9 n 1 0 12 root_fs_2 <------------ inuse = n 10 y 1 0 14 root_fs_3 2
Resolution
Prior to the customer requesting assistance from Support, a data mover reboot was completed and no improvement was seen. To resolve this issue, login as 'root' user to the Primary Control Station and re-mount the file system 'root_fs_2' to the data mover server_2:
[root@ssgnas03cs0 nasadmin]# server_mount server_2 root_fs_2 / server_2 : done
Ensure that the root file system is re-mounted:
[root@ssgnas03cs0 nasadmin]# server_mount ALL server_2 : root_fs_2 on / uxfs,perm,rw root_fs_common on /.etc_common uxfs,perm,ro server_3 : root_fs_3 on / uxfs,perm,rw root_fs_common on /.etc_common uxfs,perm,ro
After the root file system is mounted, issue the 'server_df server_2' command to ensure that it completes successfully:
[root@ssgnas03cs0 nasadmin]# server_df server_2 server_2 : Filesystem kbytes used avail capacity Mounted on root_fs_common 15368 5280 10088 34% /.etc_common root_fs_2 258128 6264 251864 2% /
Additional Information
In this particular instance, the customer was performing a cleanup of unnecessary file systems to reclaim space for the NAS pool. As part of the cleanup, they permanently unmounted and attempted to delete the root file system 'root_fs_2' for the data mover server_2. The root file system for a data mover should never be deleted.