ECS: Dial Home - Ref_ID: BtreeDiskLevelGC - Symptom Code: 1325

An ECS system has dialed home with the following alert. 

<?xml version="1.0" encoding="UTF-8"?>
<ConnectHome SchemaVersion="1.0.8"
    xsi:schemaLocation="/opt/connectemc/schema/ema_ConnectHome108.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
    <TransType>10</TransType>
    <TransTypeDesc>EventXML</TransTypeDesc>
    <TransID>0</TransID>
    <Node Status="Unknown" State="Unknown" ID="psnt1">
        <Identifier>
            <DeviceState>1</DeviceState>
            <DeviceStatus>1</DeviceStatus>
            <ClarifyID>psnt1</ClarifyID>
            <SiteName>UNKNOWN</SiteName>
            <Vendor>EMC</Vendor>
            <DeviceType>ElasticCloudStorageApp</DeviceType>
            <Model>ElasticCloudStorage</Model>
            <SerialNumber>psnt1</SerialNumber>
            <WWN>ECSQE00077</WWN>
            <Platform>platform</Platform>
            <OS>SLES</OS>
            <OS_VER>12.3</OS_VER>
            <UcodeVer>3.3.0.0-4352.834410f</UcodeVer>
            <EmbedLevel>2</EmbedLevel>
            <InternalMaxSize>512800</InternalMaxSize>
            <Comment>Object Alert</Comment>
        </Identifier>
        <Connection>
            <ConnectType>ESRS</ConnectType>
            <IPName>sandy-maize.ecs.lab.emc.com</IPName>
            <ConnectNum>169.254.197.2</ConnectNum>
            <Port>22</Port>
        </Connection>
        <HeartBeat>
            <HBStatus></HBStatus>
            <HBState></HBState>
        </HeartBeat>
        <InternalData>
            <FileList/>
            <EventList>
                <Event>
                    <SymptomCode>1325</SymptomCode>
                    <Category>Configuration</Category>
                    <Severity>Warning</Severity>
                    <Status>Unknown</Status>
                    <Component>Object</Component>
                    <ComponentID></ComponentID>
                    <SubComponent>BTREE Disk GC</SubComponent>
                    <SubComponentID></SubComponentID>
                    <CallHome>true</CallHome>
                    <FirstTime>2019-01-29T20:05:09.755Z</FirstTime>
                    <LastTime>2019-01-29T20:05:09.755Z</LastTime>
                    <Count>0</Count>
                    <EventData><![CDATA[]]></EventData>
                    <Description><![CDATA[Capacity free-up throughput is too slow to catch up with system metadata garbage collection. Please involve ECS engineering.  [Ref_ID : BtreeDiskLevelGC]]]></Description>
                </Event>
            </EventList>
        </InternalData>
        <ExternalFiles/>
    </Node>
</ConnectHome>

 

Btree GC works slow at the disk capacity free stage. This alert is generated where there is more than 1 TB of Garbage pending delete and the difference between Garbage Chunk Reclaim Rate and Garbage Capacity Reclaim Rate is greater than 100 GB. The alert sends when the system has been in this state for 7 days.

1. Connect to one of the nodes in the cluster over a putty session.
2. Verify Current Sensor Status.

# svc_rest_cmd /vdc/sensor/status/BtreeDiskLevelGC
svc_rest_cmd 1.0.10

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<SensorStatusList>
  <SensorId>BtreeDiskLevelGCC</SensorId>
  <ListSensorStatus>
    <SensorStatusRestRep>
      <ContextId>BtreeDiskLevelGC</ContextId>
      <CurrentState>OK</CurrentState>
      <NodeId>47b32995-35fc-4593-8f3c-39c878fb10ad</NodeId>
      <PreviousState>OK</PreviousState>
      <Time>1551104230509</Time>
    </SensorStatusRestRep>
  </ListSensorStatus>

3. Look for <CurrentState>, if it is set to OK, then error is not active and this is likely a false alert. Ignore the alert, but if it triggers again within the next week then open a service request with Technical Support