Isilon: NetWorker NSM for NAS - Isilon or PowerScale snapshot browsable recovery fails with Permission Denied error on a Linux NetWorker Server.

An error message will be presented after the mount operation fails. In case of using the NMC for recovery it is something like - "Unable to open directory '/nsr/tmp/1234.1234567' on host 'networkerserver.dell.com'. Permission denied". In case of using the command line 'nsrsnapadmin' to do the recovery it is something like "access denied by server when mounting".

The 'root' user on the NetWorker Server needs sufficient privileges to be able to traverse the mounted snapshot. The 'root' user on the NetWorker Server needs 'read' and 'execute' permissions on the directories to be able to traverse the mounted snapshot and see the contents inside.

In this particular case, the Isilon/PowerScale '/ifs/data' originally had these permissions, which worked fine.

powerscale.dell.com# ls -led /ifs/data
drwxr-xr-x     6 root  wheel  113 Mar 28 13:10 /ifs/data
OWNER: user:root
GROUP: group:wheel
SYNTHETIC ACL
0: user:root allow dir_gen_read,dir_gen_write,dir_gen_execute,std_write_dac,delete_child
1: group:wheel allow dir_gen_read,dir_gen_execute
2: everyone allow dir_gen_read,dir_gen_execute

This filesystem subsequently had its permissions changed which triggered this behaviour -- note 'read' and 'execute' permissions for 'everyone' is removed on the filesystem

powerscale.dell.com# chmod 700 /ifs/data
powerscale.dell.com# ls -led /ifs/data
drwx------     6 root  wheel  113 Mar 28 13:10 /ifs/data
OWNER: user:root
GROUP: group:wheel
SYNTHETIC ACL
0: user:root allow dir_gen_read,dir_gen_write,dir_gen_execute,std_write_dac,delete_child
1: group:wheel allow std_read_dac,std_synchronize,dir_read_attr

Logically NetWorker is making an NFS mount of the snapshot and putting a path to it under ‘/nsr/tmp/123.1234567’ - this can be done manually (outside of NetWorker) and the same issue will be observed when trying to traverse the mounted filesystem. This 'Permission Denied' error is a characteristic of NFS itself. Here is an example of mounting the same snapshot and trying to traverse it (without NetWorker).
 
Mounting and browsing with correct permissions on NFS config (the 'root' user on NetWorker Server has read/execute access on the /ifs/data directory).

[root@networkerserver.dell.com tmp]# mount powerscale.dell.com:/ifs/data/.snapshot/NSMNAS-20220329075723-1648537042-0 /tmp/mount
[root@networkerserver.dell.com tmp]# ls -l /tmp/mount
total 8
drwxr-xr-x 5 root root  77 Jul  4  2021 Isilon_Support
drwxr-xr-x 2 root root  30 Dec 22  2020 recoverfolder
drwxr-xr-x 2 root root 176 Mar 23 14:27 testbackup

 
Mounting and browsing with incorrect permissions on NFS config (root user on NetWorker Server is denied read/execute access on the /ifs/data directory).

[root@networkerserver.dell.com tmp]# mount powerscale.dell.com:/ifs/data/.snapshot/NSMNAS-20220329075723-1648537042-0 /tmp/mount
[root@networkerserver.dell.com tmp]# ls -l /tmp/mount
ls: cannot open directory /tmp/mount: Permission denied

1. Have 'read' and 'execute' privileges on the filesystem for 'everyone' on the filesystem at snapshot creation time.

2. Allocate the 'root' user of the NetWorker Server the necessary permissions by creating an NFS share on the Isilon/PowerScale itself.

When trying to mount a snapshot to browse it using command line ('/ifs/data' has 700 permissions when snapshot was taken and there is no NFS export created).

[root@networkerserver.dell.com ~]# nsrsnapadmin -s networkerserver.dell.com -M networkerserver.dell.com -c powerscale.dell.com -S 3863397257 -r
180437:nsrsnapadmin:Trying NAS credentials from 'NSR client' 'powerscale.dell.com', group 'SNAP/NASSNAPWF/BACKUP/NAS'
38008:(pid 12645):Snapshot not found
173276:nsrsnapadmin: Will retry to mount path '/net/powerscale.dell.com/ifs/data/.snapshot/NSMNAS-20220401101904-1648804743-0' 4 more times: Unable to mount '/net/powerscale.dell.com/ifs/data/.snapshot/NSMNAS-20220401101904-1648804743-0': Command 'mount -t nfs "powerscale.dell.com:/ifs/data/.snapshot/NSMNAS-20220401101904-1648804743-0" "/nsr/tmp/12645.1648806854" 2>&1' exited with code 32: mount.nfs: access denied by server while mounting powerscale.dell.com:/ifs/data/.snapshot/NSMNAS-20220401101904-1648804743-0
173276:nsrsnapadmin: Will retry to mount path '/net/powerscale.dell.com/ifs/data/.snapshot/NSMNAS-20220401101904-1648804743-0' 3 more times: Unable to mount '/net/powerscale.dell.com/ifs/data/.snapshot/NSMNAS-20220401101904-1648804743-0': Command 'mount -t nfs "powerscale.dell.com:/ifs/data/.snapshot/NSMNAS-20220401101904-1648804743-0" "/nsr/tmp/12645.1648806859" 2>&1' exited with code 32: mount.nfs: access denied by server while mounting powerscale.dell.com:/ifs/data/.snapshot/NSMNAS-20220401101904-1648804743-0
173276:nsrsnapadmin: Will retry to mount path '/net/powerscale.dell.com/ifs/data/.snapshot/NSMNAS-20220401101904-1648804743-0' 2 more times: Unable to mount '/net/powerscale.dell.com/ifs/data/.snapshot/NSMNAS-20220401101904-1648804743-0': Command 'mount -t nfs "powerscale.dell.com:/ifs/data/.snapshot/NSMNAS-20220401101904-1648804743-0" "/nsr/tmp/12645.1648806865" 2>&1' exited with code 32: mount.nfs: access denied by server while mounting powerscale.dell.com:/ifs/data/.snapshot/NSMNAS-20220401101904-1648804743-0

When trying to mount a snapshot to browse it using command line - '/ifs/data' has 700 permissions when snapshot was taken and an NFS export is created correctly on the Isilon itself to allow just the 'root' user on the NetWorker Server to traverse the snapshot by NFS.

[root@networkerserver.dell.com ~]# nsrsnapadmin -s networkerserver.dell.com -M networkerserver.dell.com -c powerscale.dell.com -S 3863397257 -r
180437:nsrsnapadmin:Trying NAS credentials from 'NSR client' 'powerscale.dell.com', group 'SNAP/NASSNAPWF/BACKUP/NAS'
38008:(pid 10873):Snapshot not found
Current working directory is /ifs/data/
snaprecover> ls -l
total 8
drwxr-xr-x root               77 Jul 04 2021 Isilon_Support
drwxr-xr-x root               30 Dec 22 2020 recoverfolder
drwxr-xr-x root              176 Mar 23 14:27 testbackup