PowerScale OneFS: Cluster attempts to access CloudPools Objects after Recall or Retirement

When a cloud storage provider is being retired, there are several things that must be accounted for in order to avoid issues. This includes:

• When an account or CloudPools are removed, the data that these referenced are no longer be accessible for the cluster
• When files stored on the cloud tier are recalled, the recall is not complete until the CloudPools job completes.
• Even if the files are fully recalled from the cloud provider, the objects in the cloud storage are not removed (cloud garbage collection) immediately. If the CloudPools or account is removed before this cleanup, CloudPools continues to clean up these objects at the designated time, resulting in extra CPU use and network traffic, along with log spam. Things to consider include:
  • When snapshots of the stubs exist, there are still references to the cloud objects, so the retention times do not start until snapshots are expired. This includes snapshots for FSAnalyze and SyncIQ jobs.
  • Retention times for files that are Not backed up default to 1 week. Cloud object cleanup does not begin until this time expires.
  • Retention times for files that are backed up using Network Data Management Protocol (NDMP) or SyncIQ default to 5 years. This means attempt to access and clean up the associated cloud objects can happen long after the account is removed
  • These retention times are set in the File Pool Policy used to archive the files to the cloud tier.

These retention times are necessary in order to allow backed up stub files to be valid should the files be restored from backup.

If a CloudPools and account are being retired, the retention time can be reset during recall or removal of the files. This can be avoided if the file is marked as Backed Up. This is done by changing the file-pool-policy values to shorter times before performing these operations. CloudPools checks the file pool policies on these actions and set the retention times appropriately before recall or delete.

In the file pool policy, these are the values that must be adjusted:

Cloud Data Retention: 604800

Cloud Incremental Backup Retention: 157680000

       Cloud Full Backup Retention: 157680000

Backed up stub files will no longer be valid after cloud cleanup has been performed.