ECS: Negative values for flux query of system_gc_full_garbage and system_gc_pending

Summary: This knowledge base identifies and resolves negative GC values returned from queries of Space Reclamation (SR) values from stat service and flux on ECS nodes.

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.
Check out other resources

Symptoms

Negative numbers are seen in the result of flux query:

admin@ecs_node1:~> curl https://localhost:4443/flux/api/external/v2/query -XPOST -k -sS -H "${TOKEN}" -H 'accept:application/json' -H 'content-type:application/json' -d '{"query": "from(bucket:\"monitoring_vdc\")|> range(start:-30m)|> filter(fn: (r) => contains(value: r[\"_measurement\"], set:[\"cq_gc_data\",\"cq_gc_remaining_elements\"]))|> keep(columns:[\"_value\", \"_field\"])|> last()" }'
{
 "Series": [
  {
   "Datatypes": [
    "long",
    "double",
    "string"
   ],
   "Columns": [
    "table",
    "_value",
    "_field"
   ],
   "Values": [
    [
     "45",
     "194235952627200",
     "system_gc_eligible_in_handling"
    ],
    [
     "46",
     "-88547243030400",
     "system_gc_full_garbage"
    ],
...
    [
     "49",
     "-88547243030400",
     "system_gc_pending"
    ],
...
    [
     "67",
     "12397823209857603",
     "user_gc_unreclaim"
    ]
   ]
  }
 ]
}

Results of the svc_gc tool include warnings and negative values:

admin@ECS02n04:~> svc_gc stats btree
svc_gc v0.8.2 (svc_tools v1.7.4) Started 2020-06-12 13:16:10
 
Local node ECS Object Version: 3.4.0.1 GA (3.4.0.1-113843.05a36e2)
Local VDC: urn:storageos:VirtualDataCenterData:92zzzd3c-5aa7-4bb8-acca-789xxxyyyc41 VDC_002
WARNING got a negative value for verification_waiting_chunks_btree_gc_level_0.TOTAL --> [-2289205]
 
====> Btree Statistics
---------------------------------------------------------------------------------------------------------------------
Category | Total | Level 0 | Level 1
---------------------------------------------------------------------------------------------------------------------
Reclaimed Garbage | 115895.62 GB ( 113.18 TB) | 115895.62 GB ( 113.18 TB) | 0.00 GB ( 0.00 TB)
---------------------------------------------------------------------------------------------------------------------
Remaining Garbage | 2709.04 GB ( 2.65 TB) | 2709.04 GB ( 2.65 TB) | 0.00 GB ( 0.00 TB)
|- Full Garbage | -286148.27 GB (-279.44 TB) | -286148.27 GB (-279.44 TB) | 0.00 GB ( 0.00 TB)
|- Reclaiming | 2.36 GB ( 0.00 TB) | 2.36 GB ( 0.00 TB) | 0.00 GB ( 0.00 TB)
|- Partial Garbage | 288859.66 GB ( 282.09 TB) | 288859.66 GB ( 282.09 TB) | 0.00 GB ( 0.00 TB)
|- Eligible in Handling | 286183.50 GB ( 279.48 TB) | 286183.50 GB ( 279.48 TB) | 0.00 GB ( 0.00 TB)
---------------------------------------------------------------------------------------------------------------------
...
...

Cause

Negative counter inserted in legacy code

Resolution

ECS support has a tool available to reset negative stat counters. Open a service request with ECS support, including a reference to this knowledge article to have the symptoms reviewed and resolved.

Affected Products

ECS Software

Products

ECS, Elastic Cloud Storage
Article Properties
Article Number: 000223692
Article Type: Solution
Last Modified: 05 Apr 2024
Version:  1
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.