Data Domain: Cloud Profile Remains Disconnected after Resolving Cloud Provider Capacity Issues

DDOS marks a cloud profile as "Unavailable" or "Disconnected" if protocol to or from the configured cloud provider returns errors. For the particular case of capacity issues in the cloud provider or account, HTTP 507 type errors would be returned.

For DDOS releases prior to 7.7.2 this error code would be treated such as any other one, and the cloud profile and provider may end up unavailable if the errors repeat frequently. However,

HTTP/507 being a special error, one which the cloud provider uses to report being at capacity, DDOS 7.7.2 and later handles this in a special way, namely, setting the cloud profile to the status shown below:

Name        Profile             Status         Reason
---------   -----------------   ------------   -------------------------------------------
cloud_unit  cloud_profile       Disconnected   Insufficient storage at the cloud provider.
---------   -----------------   ------------   -------------------------------------------

So that the DD administrator knows why the cloud unit is not working anymore a persistent registry key is set, indicating the cloud provider is unavailable. The DD FS process does not contact the cloud provider anymore.

After the capacity issues in the cloud provider are resolved, the cloud profile and unit still stay in the same status. No indication of cloud activity can be seen in the usual "ddfs.info" and "csm.log" files. Restarting the FS does not solve the issue.

 

When the cloud provider reports to be at capacity (through HTTP/507 errors) the cloud profile and provider are marked as disconnected, and a registry key is set. This registry key prevents the cloud code from reaching out to the cloud provider again, so even if the space issue is resolved, the cloud unit cannot be accessed without further intervention of the administrator.

Although this issue has been seen only for ECS as a cloud provider, there is nothing preventing it from occurring for other cloud providers, if they can return HTTP 507 error codes to the DD when capacity issues exist (that is, when the account that is used for storage has reached some given quota).

There is a new command in DDOS 7.7.2 and later releases which must be used to clear the cloud unit state to be able to access it again after the cloud turned disconnected due to capacity:

#  cloud unit clear-state <unit-name> {nospace}
                                       Clear the registry entry indicating
                                       cloud unit is not available for writing

For the example above the command below would need to be run after resolving the cloud provider capacity issues to allow for the cloud provider to be used again:

# cloud unit clear-state cloud-unit nospace