ECS: Dial Home - Ref_ID: RepoChunkLevelGC - Symptom Code: 1333

An ECS has dialed home with the following alert.

<?xml version="1.0" encoding="UTF-8"?>
<ConnectHome SchemaVersion="1.0.8"
    xsi:schemaLocation="/opt/connectemc/schema/ema_ConnectHome108.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
                <Event>
                    <SymptomCode>1333</SymptomCode>
                    <Category>Configuration</Category>
                    <Severity>Warning</Severity>
                    <Status>Unknown</Status>
                    <Component>Object</Component>
                    <ComponentID></ComponentID>
                    <SubComponent>REPO Chunk GC</SubComponent>
                    <SubComponentID></SubComponentID>
                    <CallHome>true</CallHome>
                    <FirstTime>2024-01-29T17:07:36.302Z</FirstTime>
                    <LastTime>2024-01-29T17:07:36.302Z</LastTime>
                    <Count>0</Count>
                    <EventData><![CDATA[]]></EventData>
                    <Description><![CDATA[User garbage collection throughput is too slow to catch up with garbage detection. Please involve ECS engineering.  [Ref_ID : RepoChunkLevelGC]]]></Description>
                </Event>
            </EventList>
        </InternalData>
        <ExternalFiles/>
    </Node>
</ConnectHome>

The Space Reclamation on user data is too slow. The bottleneck lies on chunk level reclamation. This alert is generated when:

1. More than 10 TB of full garbage is present.
2. The difference between the garbage detection rate and the garbage chunk reclaim rate is greater than 100 GB.

This can trigger as a false alert if a recent large deletion has been made. Follow the steps below to determine if so.

1. Connect to one of the nodes in the cluster over a putty session.
2. Verify Current Sensor Status. 

Command:

# svc_rest_cmd  /vdc/sensor/status/RepoChunkLevelGC

Example:

# svc_rest_cmd  /vdc/sensor/status/RepoChunkLevelGC

svc_rest_cmd 1.0.10
<?xml version="1.0" encoding="UTF-8" standalone="yes"?> svc_rest_cmd  /vdc/sensor/status/RepoChunkLevelGC <SensorStatusList>
  <SensorId>RepoChunkLevelGC</SensorId>
  <ListSensorStatus>
    <SensorStatusRestRep>
      <ContextId>Repo Chunk Level GC</ContextId>
      <CurrentState>OK</CurrentState>
      <NodeId>xxxxxxxxxxxxxxxxxxxxxxxxxxxxx</NodeId>
      <PreviousState>OK</PreviousState>
      <Time>1551118064264</Time>
    </SensorStatusRestRep>
  </ListSensorStatus>

3. Look for <CurrentState>, if it is set to OK, then error is not active and this is likely a false alert. Ignore the alert, but if it triggers again within the next week then open a service request with Technical Support.