Isilon OneFS: SyncIQ "Error while transferring dir <PATH>/.snapshot/"

Modifying the snapshot setting  Global Visible Accessible or "sysctl efs.snapshot.default_sub_accessible // sysctl efs.snapshot.default_sub_visible".

By default ".snapshot" folder is a system folder used for minisnaps, where OneFS stores different snapshots version under this folder.

1- The folder is created by default under any directory in /ifs.
2- The folder is visible only under /ifs.
3- The folder is hidden under all rest subdirs in the filesystem.

This means that if you tried to create a ".snapshot" folder under any directory in /ifs , you will get the following error:

# cd /ifs/source
# mkdir .snapshot
mkdir: .snapshot: File exists

Modifying the snapshot setting  Global Visible Accessible or "sysctl efs.snapshot.default_sub_accessible // sysctl efs.snapshot.default_sub_visible" changes  the behavior of  ".snapshot"  folder as below.

1- The folder becomes not visible under /ifs.
2- You would be able to create a ".snapshot" folder under any subdirectory.

# isi snapshot settings modify --global-visible-accessible=no # cd /ifs/source # mkdir .snapshot # ls -a . .. .snapshot

In other words , changing the "global settings or sysctl" will remove the functionality of minisnap and will treat the ".snapshot" folder as a normal folder in the filesystem.

When the above happens , SyncIQ jobs will consider the ".snapshot" folder as a new folder created on source cluster and will attempt to replicate it to the target cluster.

However , if the target cluster has the default settings of "global settings or sysctl" , this means that the target cluster consider ".snapshot" folder a system folder used for minisnap. And by result , target cluster will deny any modification in its minisnap folder and will kick off an error to the running SyncIQ job.

Repro:

Source-1# isi snapshot settings modify --global-visible-accessible=no
Source-1# mkdir /ifs/source/.snapshot

Source-1# isi sync jobs start Policy_1
Source-1# isi sync reports list
Policy Name  Job ID  Start Time          End Time            Action  State
-----------------------------------------------------------------------------
Policy_1      2       2018-09-17T10:04:03 2018-09-17T10:04:05 run     failed
Policy_1      1       2018-09-17T10:00:32 2018-09-17T10:00:35 run     finished
-----------------------------------------------------------------------------
Total: 2

Source-1# isi sync reports list --reports-per-policy=1 -v | grep -i errors
                      Errors: (policy name: Policy_1 target: 192.xxx.xxx.xx) SyncIQ encountered a filesystem error on target cluster. Error at target cluster on node [Target-1]:  Unable to link /ifs/source/.snapshot in directory /ifs/source, Local error : Failed to move .snapshot from /ifs/target to tmp working directory: Read-only filesystem
 

Creating ".snapshot" folders on Isilon is a bad idea that should not be done on any cluster.
Changing the snapshot settings without understanding of the result of changes can result in the failure.

- Customer should not create .snapshot folders on Isilon Cluster and should track with application team to stop creating those folders on cluster.

- Follow Restricted Note in order to have a workaround of the problem.

NB: workaround of problem should be done with the guidance of SyncIQ "L2/Coach/SME".