ECS: Nodes out of memory and or swap and metering service over consuming memory.

Summary: Nodes out of memory and or swap and metering service over consuming memory.

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.
Check out other resources

Symptoms

Nodes are out of memory and swap resources:

# free -m
             total       used       free     shared    buffers     cached
Mem:         60104      57332       2771       2460          6       3621
-/+ buffers/cache:      53704       6400
Swap:         8191       8191          0

The metering service is over consuming memory:

dtquery-error.log
2016-02-02 23:15:01,190 [Thread-6] ERROR  DtQueryServiceImpl.java (line 112) WSCritical. Memory usage threshold exceeded. usedMemory=499017736, percentageUsed=96.15363520735336

metering-error.log
2015-12-02 17:14:09,682 [Thread-6] ERROR  MeteringManager.java (line 102) WSCritical. Memory usage threshold exceeded. usedMemory=2045441408, percentageUsed=98.44486179780154
2015-12-02 17:28:01,737 [Thread-6] ERROR  MeteringManager.java (line 102) WSCritical. Memory usage threshold exceeded. usedMemory=2068862680, percentageUsed=99.57210204831705

Also, because the metering service is being restarted as an attempt to manage memory usage, Metering DTs may show as unready:

# curl -ks http://$(hostname -f):9101/stats/dt/DTInitStat/ | xmllint --format -
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<result>
  <entry>
    <total_dt_num>1920</total_dt_num>
    <unready_dt_num>7</unready_dt_num>
    <unknown_dt_num>0</unknown_dt_num>
  </entry>
  <entry>
    <type>MR</type>
    <level>0</level>
    <total_dt_num>128</total_dt_num>
    <unready_dt_num>7</unready_dt_num>
    <ObjectControllerException.ERROR_DIRECTORYTABLE_NOT_INITIALIZED>[urn:storageos:OwnershipInfo:17bd10f9-7141-42c5-b2b2-02db154ab84d__MR_18_128_0:, urn:storageos:OwnershipInfo:17bd10f9-7141-42c5-b2b2-02db154ab84d__MR_107_128_0:, urn:storageos:OwnershipInfo:17bd10f9-7141-42c5-b2b2-02db154ab84d__MR_7_128_0:, urn:storageos:OwnershipInfo:17bd10f9-7141-42c5-b2b2-02db154ab84d__MR_74_128_0:, urn:storageos:OwnershipInfo:17bd10f9-7141-42c5-b2b2-02db154ab84d__MR_1_128_0:, urn:storageos:OwnershipInfo:17bd10f9-7141-42c5-b2b2-02db154ab84d__MR_76_128_0:, urn:storageos:OwnershipInfo:17bd10f9-7141-42c5-b2b2-02db154ab84d__MR_99_128_0:]</ObjectControllerException.ERROR_DIRECTORYTABLE_NOT_INITIALIZED>
  </entry>
</result>

Cause

Issue in metering service which can lead to over usage of memory and swap resources.

Resolution

There is no workaround for this issue. Work to upgrade the ECS Appliance to ECS 2.1 HF3 or ECS 2.2 HF2 or any higher release. The issue with the metering service is resolved.

Affected Products

ECS Appliance

Products

ECS Appliance, ECS Appliance Hardware Gen1 U-Series, ECS Appliance Software without Encryption
Article Properties
Article Number: 000052897
Article Type: Solution
Last Modified: 22 Feb 2023
Version:  3
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.