ECS: Dial Home: ObjectGCObjectCleanup - Symptom Code: 1313 and 1314

An ECS has dialed home with the following alert.
Description: CAS Processing object cleanup speed is slow.
 

Symptom Code	Severity
1313	WARNING
1314	ERROR

Example alert:

<?xml version="1.0" encoding="UTF-8"?>
<ConnectHome SchemaVersion="1.0.8"
    xsi:schemaLocation="/opt/connectemc/schema/ema_ConnectHome108.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
    <TransType>10</TransType>
    <TransTypeDesc>EventXML</TransTypeDesc>
    <TransID>0</TransID>
    <Node Status="Unknown" State="Unknown" ID="ECSQE00079">
        <Identifier>
            <DeviceState>1</DeviceState>
            <DeviceStatus>1</DeviceStatus>
            <ClarifyID>ECSQE00079</ClarifyID>
            <SiteName>UNKNOWN</SiteName>
            <Vendor>EMC</Vendor>
            <DeviceType>ElasticCloudStorageApp</DeviceType>
            <Model>ElasticCloudStorage</Model>
            <SerialNumber>ECSQE00079</SerialNumber>
            <WWN>ECSQE00079</WWN>
            <Platform>platform</Platform>
            <OS>SLES</OS>
            <OS_VER>12.3</OS_VER>
            <UcodeVer>3.3.0.0-4349.d0de882</UcodeVer>
            <EmbedLevel>2</EmbedLevel>
            <InternalMaxSize>512800</InternalMaxSize>
            <Comment>Object Alert</Comment>
        </Identifier>
        <Connection>
            <ConnectType>ESRS</ConnectType>
            <IPName>austin-burlywood.ecs.lab.emc.com</IPName>
            <ConnectNum>169.254.109.16</ConnectNum>
            <Port>22</Port>
        </Connection>
        <HeartBeat>
            <HBStatus></HBStatus>
            <HBState></HBState>
        </HeartBeat>
        <InternalData>
            <FileList/>
            <EventList>
                <Event>
                    <SymptomCode>1314</SymptomCode>
                    <Category>Configuration</Category>
                    <Severity>Error</Severity>
                    <Status>Unknown</Status>
                    <Component>Object</Component>
                    <ComponentID></ComponentID>
                    <SubComponent>CAS Processing Object Cleanup</SubComponent>
                    <SubComponentID></SubComponentID>
                    <CallHome>true</CallHome>
                    <FirstTime>2019-01-28T22:15:30.182Z</FirstTime>
                    <LastTime>2019-01-28T22:15:30.182Z</LastTime>
                    <Count>0</Count>
                    <EventData><![CDATA[]]></EventData>
                    <Description><![CDATA[CAS Processing object cleanup speed is slow.  [Ref_ID : ObjectGCObjectCleanup]]]></Description>
                </Event>
            </EventList>
        </InternalData>
        <ExternalFiles/>
    </Node>
</ConnectHome>

ECS has old unprocessed CAS GC Object Cleanup tasks, and this could trigger the slow CAS GC alert.

Note: There is an issue in ECS versions less than 3.5 which causes an alert for this sensor if there are no deletes occurring. Even if there is no space to reclaim. 

Connect to one of the nodes in the ECS over SSH.

1. Verify the current sensor status:

# svc_rest_cmd /vdc/sensor/status/ObjectGCObjectCleanup

#  svc_rest_cmd /vdc/sensor/status/ObjectGCObjectCleanup
svc_rest_cmd 1.0.10

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<SensorStatusList>
  <SensorId>ObjectGCObjectCleanup</SensorId>
  <ListSensorStatus>
    <SensorStatusRestRep>
      <ContextId>ObjectGCObjectCleanup</ContextId>
      <CurrentState>OK</CurrentState>
      <NodeId>1f4368c3-37cd-467f-9efc-289264806ea0</NodeId>
      <PreviousState>OK</PreviousState>
      <Time>1551118064264</Time>
    </SensorStatusRestRep>
  </ListSensorStatus>
<SensorStatusList>

2. Check if CAS GC is enabled, Object is enabled and dry_run set to false

# svc_param list com.emc.ecs.objectgc | grep -e cas.enabled -e dry_run -e object.enabled

com.emc.ecs.objectgc.cas.enabled                                   Enable or disable CAS GC                                       true
com.emc.ecs.objectgc.cas.process_object.dry_run                    If enabled then delete marker insertion will be skipped a...   *false
com.emc.ecs.objectgc.cas.process_object.enabled                    Enable or disable GC process object                            true

REFLECTIONS:
CAS GC reclaims the blobs associated with the delete clips (also called as Reflections). In ECS, both clips and blobs are considered objects. When a clip is deleted, it becomes a reflection which holds audit information like when the delete call was made and from which IP.
Some Applications keep this information forever. The typical clip or reflection size would be in KB, so not a lot of space is consumed by the reflections. 

If you prefer to reclaim the reflections as well, then they must set Reflection Expiration for the CAS bucket.
This can be done from the ECS UI (Manage >  Buckets > Select namespace and click Edit Bucket. Go to Required tab and enable Reflection Expiration and set Reflection Age.

To delete a CAS bucket, the reflections must be expired, as they are still considered objects in the bucket.

ECS Appliance Hardware Gen3 EX300, ECS Appliance Hardware Gen3 EX3000, ECS Appliance Hardware Gen2 D-Series, ECS Appliance Hardware Gen2 U-Series, ECS Appliance Hardware Gen3 EX500, ECS Appliance Hardware Gen3 EXF900, ECS Appliance Hardware Series , ECS Appliance Software with Encryption, ECS Appliance Software without Encryption, Elastic Cloud Storage ... View More View Less