Isilon: What are the CLI Steps for SyncIQ Failover and Failback

What are the CLI steps for SyncIQ failover and failback?
The UI process has a step-by-step guide, is there was a similar guide for the CLI?

Detailed steps on doing FOFB

CLI guide for failover and failback:

• PowerScale OneFS 9.5.0.0 CLI Administration Guide
  Page 271 for 9.5
• PowerScale OneFS 9.7.0.0 CLI Administration Guide
  Page 288 for 9.7
• PowerScale OneFS 9.8.0.0 CLI Administration Guide
  Page 295 for 9.8
• PowerScale OneFS 9.9.0.0 CLI Administration Guide
  Page 296 for 9.9
• PowerScale OneFS 9.10.0.0 CLI Administration Guide
  Page 299 for 9.10

While the above documentation offers some good information, the steps below are more detailed when performing a failover and failback using CLI.

The steps below use SyncIQ terminology for these two terms:

• SOURCE Cluster = PRIMARY
• TARGET Cluster = SECONDARY

FAILOVER:

1. At the PRIMARY cluster, consider running domainmark job days or weeks in advance if this is the first failover attempt for the cluster. If the dataset is large, it helps save time in speeding up the domainmark job phase.

   Note: A new "Accelerated Failback" option removes this step. This step must ONLY be done once. Once marked, future domainmark jobs are (see step 7 below) a no-op.

   # isi job jobs start domainmark --root=<path> --dm-type=synciq

   This marks every LIN with the appropriate protective domain ID ahead of time rather than having the failover job perform it all (see step 7). The domainmark job can take a long time depending on the size of the dataset.

2. Stop all writing to the PRIMARY policy's path.

   Note: Writes on the PRIMARY policies path that occurs from this step forward are not be retained resulting in possible DL. Confirm that all writes to that path ON THE PRIMARY have stopped.

3. At the PRIMARY cluster, backup the policy schedules and then disable all schedules by setting policies to manual.

   To save a backup copy of the schedules:

   # cat /ifs/.ifsvar/modules/tsm/config/siq-policies.gc|egrep 'common.name|schedule ' >> /ifs/.ifsvar/modules/tsm/config/policy-schedules.txt

   Then disable all schedules by setting policies to manual.

   Note: A sync job and failover job cannot run simultaneously by design and causes the failover attempt to fail. To avoid this condition, set all policies to manual.

   # isi sync policies modify  --policy=[POLICY] --schedule=""

4. At the PRIMARY cluster, run one last sync job and confirm it has been successfully completed.

   Note: This step is only recommended if testing FOFB functionality. DO NOT perform this step if the PRIMARY cluster has already encountered a failure event and the SECONDARY cluster has already been set to allow writes.

   # isi sync jobs start [POLICY]

   Run this command to confirm successful completion:

   # isi sync reports list --reports-per-policy=1
   *Confirm the End time and State=finished

   At the PRIMARY cluster, run one last sync job.

   # isi sync jobs start [POLICY]

5. At the SECONDARY cluster, perform 'Allow Writes' action and verify that the local job finishes that action.

   # isi sync recovery allow-write --policy-name=[POLICY]
   # isi sync target list
   Name             Source    Target Path              Last Job State  FOFB State
   -----------------------------------------------------------------------------------
   qtestsync  primary_clust      /ifs/data/siq_quota_test finished        writes_enabled
   -----------------------------------------------------------------------------------
   Total: 1

   NOTE: Reverse SmartLock directory settings as needed on both clusters.
   https://infohub.delltechnologies.com/en-us/l/dell-powerscale-smartlock-best-practices/synciq/

6. Redirect clients (SMB, NFS, HTTP, FTP, and so forth) to the SECONDARY cluster

   Note: The specifics of this step are outside the scope of this article and require creating SMB shares, Active Directory Domain join, Machine accounts, SPN, NFS exports, redirecting SmartConnect DNS and adding Authentication Providers.

7. Create a recovery snapshot on both clusters before proceeding with the resync-prep

   ON SOURCE

   # isi snapshot snapshots create --path=[SOURCE_PATH] --name=SIQ-recovery-policy-[POLICY_NAME] --expires=2W

   ON TARGET

   # isi snapshot snapshots create --path=[TARGET_PATH] --name=SIQ-recovery-policy-[POLICY_NAME] --expires=2W

8. At the PRIMARY cluster, perform the failover job with prepare resync and confirm that the resync_prep_finalize phase is completed.

   # isi sync recovery resync-prep --policy-name=[POLICY]
   
   # isi sync reports list --policy-name=qtestsync --sort job_id
   Policy Name  Job ID  Start Time          End Time            Action                  State
   ---------------------------------------------------------------------------------------------
   qtestsync    1       2015-02-11T08:31:27 2015-02-11T08:31:34 run                     finished
   qtestsync    2       2015-02-11T08:41:19 2015-02-11T08:41:31 resync_prep             finished
   qtestsync    3       2015-02-11T08:41:31 2015-02-11T08:41:34 resync_prep_domain_mark finished
   qtestsync    4       2015-02-11T08:41:34 2015-02-11T08:41:42 resync_prep_restore     finished
   qtestsync    5       2015-02-11T08:41:42 2015-02-11T08:41:45 resync_prep_finalize    finished

FAILBACK

ls -l /ifs/.ifsvar/modules/tsm/config/source_records/7da67596f099b75ad687a05f6b11781d*

1. The new [POLICY]_mirror policy on the SECONDARY cluster can be run to start syncing back to the PRIMARY.

   # isi sync jobs start --policy-name=[POLICY]_mirror

2. Stop all writing to the SECONDARY policy's path.

   Note: Writes on the SECONDARY policies path that occurs from this step forward are not retained resulting in possible DL. Confirm that all writes to that path ON THE SECONDARY have stopped.

3. Disable all schedules by setting the policies to manual.

   # isi sync policies modify  --policy=[POLICY]_mirror --schedule=""

4. At the SECONDARY cluster, run one last sync job

   # isi sync jobs start --policy-name=[POLICY]_mirror

5. At the PRIMARY cluster, perform 'Allow Writes' action and verify that the local job finishes that action.

   # isi sync recovery allow-write --policy-name=[POLICY]_mirror
   # isi sync target list
   Name             Source    Target Path              Last Job State  FOFB State
   -----------------------------------------------------------------------------------
   qtestsync_mirror secondary_clust      /ifs/data/siq_quota_test finished        writes_enabled
   -----------------------------------------------------------------------------------
   Total: 1

   NOTE: Reverse SmartLock directory settings as needed on both clusters.
   https://infohub.delltechnologies.com/en-us/l/dell-powerscale-smartlock-best-practices/synciq/

6. Redirect clients (SMB, NFS, HTTP, FTP, and so forth) to the PRIMARY cluster

   Note: The specifics of this step are outside the scope of this article and require creating SMB shares, NFS exports, and redirecting SmartConnect DNS.

7. Create a recovery snapshot on both clusters before proceeding with the resync-prep

   ON SOURCE

   # isi snapshot snapshots create --path=[SOURCE_PATH] --name=SIQ-recovery-policy-[POLICY_NAME] --expires=2W

   ON TARGET

   # isi snapshot snapshots create --path=[TARGET_PATH] --name=SIQ-recovery-policy-[POLICY_NAME] --expires=2W

8. At the SECONDARY cluster, perform the failback job with prepare resync and confirm that the resync_prep_finalize succeeded

   # isi sync recovery resync-prep --policy-name=[POLICY]_mirror
   
   # isi sync reports list --policy-name=[POLICY]_mirror --sort job_id --reports-per-policy=5
   Policy Name  Job ID  Start Time          End Time            Action                  State
   ---------------------------------------------------------------------------------------------
   qtestsync_mirror    1       2015-02-12T08:31:27 2015-02-12T08:31:34 run                     finished
   qtestsync_mirror    2       2015-02-12T08:41:19 2015-02-12T08:41:31 resync_prep             finished
   qtestsync_mirror    3       2015-02-12T08:41:31 2015-02-12T08:41:34 resync_prep_domain_mark finished
   qtestsync_mirror    4       2015-02-12T08:41:34 2015-02-12T08:41:42 resync_prep_restore     finished
   qtestsync_mirror    5       2015-02-12T08:41:42 2015-02-12T08:41:45 resync_prep_finalize    finished

   The SECONDARY is now READ-ONLY and the SECONDARY [POLICY]_mirror policy is disabled.

   Note: Do not delete any mirror policies.

9. The original policies on the PRIMARY are now enabled. Use the backup file from FAILOVER step 3 to restore your policy schedules.

   On the PRIMARY:
   View the saved copy of the policy schedules:

   # cat /ifs/.ifsvar/modules/tsm/config/policy-schedules.txt

   Restore the policy schedules:

   # isi sync policies modify  --policy=[POLICY] --schedule=[schedule]

10. On the original Secondary, the SIQ-<mirrorpolID>-latest snapshot will be left over after a successful failback. Manually clean up SIQ-<mirrorpolID>-latest snapshot to avoid COW writes to any existing snapshots on Secondary.

    # isi snapshot snapshots list
    ID   Name                                        Path
    -----------------------------------------------------------------------
    16   SIQ-recovery-policy-Test                    /ifs/data/failovertest
    18   SIQ-005056ac0655f7f5e267a71dae70c997-latest /ifs/data/failovertest  <-- pol_mirror-latest
    24   SIQ-ps9715x1-Test-2025-03-25_19-20-52       /ifs/data/failovertest
    -----------------------------------------------------------------------
    Total: 3
    
    # isi snapshot snapshots delete --id=<id>