ECS: Dial Home - Ref_ID: BtreeChunkLevelGC - Symptom Code: 1321

An ECS system has dialed home with the following alert. 

<?xml version="1.0" encoding="UTF-8"?>
<ConnectHome SchemaVersion="1.0.8"
    xsi:schemaLocation="/opt/connectemc/schema/ema_ConnectHome108.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
    <TransType>10</TransType>
    <TransTypeDesc>EventXML</TransTypeDesc>
    <TransID>0</TransID>
    <Node Status="Unknown" State="Unknown" ID="psnt1">
        <Identifier>
            <DeviceState>1</DeviceState>
            <DeviceStatus>1</DeviceStatus>
            <ClarifyID>psnt1</ClarifyID>
            <SiteName>UNKNOWN</SiteName>
            <Vendor>EMC</Vendor>
            <DeviceType>ElasticCloudStorageApp</DeviceType>
            <Model>ElasticCloudStorage</Model>
            <SerialNumber>psnt1</SerialNumber>
            <WWN>ECSQE00077</WWN>
            <Platform>platform</Platform>
            <OS>SLES</OS>
            <OS_VER>12.3</OS_VER>
            <UcodeVer>3.3.0.0-4352.834410f</UcodeVer>
            <EmbedLevel>2</EmbedLevel>
            <InternalMaxSize>512800</InternalMaxSize>
            <Comment>Object Alert</Comment>
        </Identifier>
        <Connection>
            <ConnectType>ESRS</ConnectType>
            <IPName>orem-maize.ecs.lab.emc.com</IPName>
            <ConnectNum>169.254.197.3</ConnectNum>
            <Port>22</Port>
        </Connection>
        <HeartBeat>
            <HBStatus></HBStatus>
            <HBState></HBState>
        </HeartBeat>
        <InternalData>
            <FileList/>
            <EventList>
                <Event>
                    <SymptomCode>1321</SymptomCode>
                    <Category>Configuration</Category>
                    <Severity>Warning</Severity>
                    <Status>Unknown</Status>
                    <Component>Object</Component>
                    <ComponentID></ComponentID>
                    <SubComponent>BTREE Chunk GC</SubComponent>
                    <SubComponentID></SubComponentID>
                    <CallHome>true</CallHome>
                    <FirstTime>2019-01-29T20:09:07.861Z</FirstTime>
                    <LastTime>2019-01-29T20:09:07.861Z</LastTime>
                    <Count>0</Count>
                    <EventData><![CDATA[]]></EventData>
                    <Description><![CDATA[System metadata garbage collection throughput is too slow to catch up with garbage detection. Please involve ECS engineering.  [Ref_ID : BtreeChunkLevelGC]]]></Description>
                </Event>
            </EventList>
        </InternalData>
        <ExternalFiles/>
    </Node>
</ConnectHome>

BTREE GC works slow at the chunk reclamation stage. This alert is generated where there is more than 1 TB of full garbage and the difference between the garbage detection rate and the garbage chunk reclaim rate is greater than 100 GB. The alert sends when the system has been in this state for 7 days.

1. Connect to one of the nodes in the cluster over a putty session.
2. Verify Current Sensor Status.

# svc_rest_cmd -i `hostname -i` /vdc/sensor/status/BtreeChunkLevelGC
svc_rest_cmd 1.0.10

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<SensorStatusList>
  <SensorId>BtreeChunkLevelGC</SensorId>
  <ListSensorStatus>
    <SensorStatusRestRep>
      <ContextId>Btree Chunk Level GC</ContextId>
      <CurrentState>OK</CurrentState>
      <NodeId>1f4368c3-37cd-467f-9efc-289264806ea0</NodeId>
      <PreviousState>OK</PreviousState>
      <Time>1551118064264</Time>
    </SensorStatusRestRep>
  </ListSensorStatus>

3. Look for <CurrentState>, if it is set to OK, then error is not active and this is likely a false alert. Ignore the alert, but if it triggers again within the next week then open a service request with Technical Support