ECS: Space Reclamation (SR) basic information gathering

When there is a concern with the Capacity on an ECS, support can review Space Reclamation (SR) formerly known as Garbage Collection is working properly.

To take action on the data gathered, contact Dell Technologies Technical Support and open a Service Request. And reference this Knowledgebase Article, ECS Space Reclamation(SR) (Garbage Collection: GC) basic information gathering

Collecting basic Space reclamation data from ECS:

xDoctor 94 and higher now include by default a svc_collect profile to gather basic Space reclamation data.

1. Ensure the latest xDoctor 94 or higher is installed. Follow ECS: xDoctor: Manual upgrade procedure
2. Run the command from any node in the VDC and repeat for every VDC in the federation.
   Command:

   # svc_collect start -p sr_basic

   Example: The script runs several commands and prints the output on the screen, below is a reduced output sample.

admin@ecsnode1:~> svc_collect start -p sr_basic
svc_collect v1.3.5 (svc_tools v2.14.0)                 Started 2023-10-30 17:19:07

######################################################################################################
Profile Name : Space Reclamation - basic information gathering
KB : https://www.dell.com/support/kbdoc/000019395
Intent : Gather key info for space reclamation/capacity issues by TSEs
Details :
1. Scope of this procedure is a single VDC, please use seperately on all VDCs to collect a full picture
2. Discuss results with your coach when needed, and include all output into capacity related swarms
3. At the end, the output will be saved in a file under /tmp/svc_collect-sr_basic*
4. Comments for guidance added inline with every command output, compare outputs to KB000019395 for further guidance
NOTE: Some KBs/Profiles may be referenced below are still WIP
######################################################################################################


 NOTE: REPO Garbage
 1. This a breakdown of all REPO(User data garbage) by chunk occupancy range.
 2. If full garbage is too large , this indicates a problem with verification tasks. Then run 'svc_gc task verification' to dump all tasks and reference KB https://www.dell.com/support/kbdoc/000209976.
 3. If aware partial is large then partial gc is slow or blocked, correlate with partial task output below. However, if it is small, then consider lowering the threshold to 50% if not done already (ref KB 12345)

VDC Command: svc_gc stats repo -e

Local node ECS Object Version: 3.8.0.3-138685.3a0a9b6bf3a (3.8.0.3 GA)
Local VDC: urn:storageos:VirtualDataCenterData:8af5b9c3-9c0c-43b5-9402-14d181ade5bf VDC1

Query RR/0/REPO_REFERENCE_COLLECTOR_KEY ...
 * Estimated by DT urn:storageos:OwnershipInfo:52576f30-f8f3-493a-9999-6fee4494f53b__RR_0_128_0:

====> REPO Chunk Usage Breakdown (Estimated by data of 1 DT times 128)
                                    Garbage   Garbage
    Category               Chunks     in GB     in TB    Pctg%
    ----------------------------------------------------------
         Full Garbage         256     32.00      0.03     1.16
        0< ValidData<1/3     1408    161.84      0.16     5.85
      1/3<=ValidData<1/2      128      9.73      0.01     0.35
      1/2<=ValidData<2/3      768     40.20      0.04     1.45
      2/3<=ValidData        19584     68.40      0.07     2.47

    * Above size and percentage are based on garbage size

====> REPO Statistics (Estimated by data of 1 DT times 128)
    Category                       in GB    in TB   Chunks
    ------------------------------------------------------
    Total Sealed REPO  Data      2768.00     2.70    22144
    |- Valid Data                2455.83     2.40
    |- Garbage                    312.16     0.30
       |- Full Garbage             32.00     0.03      256
       |- Partial Garbage         280.16     0.27    21888
          |- Aware Partial        171.57     0.17     1536


=== truncated 



Command from node r1n7: free -m

             total       used       free     shared    buffers     cached
Mem:         63038      61139       1898        986         57      14476
-/+ buffers/cache:      46606      16431
Swap:        16383          3      16380

--------------------------------------------------

Command from node r1n8: free -m

             total       used       free     shared    buffers     cached
Mem:         63038      61347       1691       1042         70      15186
-/+ buffers/cache:      46091      16947
Swap:        16383          3      16380


Command output saved as file /tmp/svc_collect-sr_basic-VDC1-20231030_171907.txt

3. Download the output file highlighted at the end of the output and transfer it to ECS technical support for analysis.

List of commands (in sr_basic collect profile) - for reference only. 

# svc_version -c -x;
# svc_gc stats repo -e;
# svc_gc stats btree -e;
# svc_vdc capacity;
# svc_vdc trend;
# svc_gc rates reclaim -start 7d -st day;
# svc_gc rates ingest -start 3d -st day
# svc_param list -changed;
# svc_perf objmem;
# svc_gc task partial;
# deleteJobs cjtrend -start 5d -st day;
# deteteJobs trend -start 5d -st day;
# svc_gc rates verification_skip -start '10 days ago' -st day;
# svc_gc task verification;
# svc_log -f LIFECYCLE_GC_FULL_RECLAIM -files sr-chunk-lifecycle* -start 7d -sh -st day; echo;
# svc_log -f LIFECYCLE_PARTIALSR_RECLAIMED -files sr-chunk-lifecycle* -start 7d -sh -st day; echo;
# svc_gc task cleanup_job -l;
# kpi summary -start 8h -s -cas;kpi summary -start 8h -s;
# svc_dt events -start 12h | grep -c OB;
# svc_node services -sr -start 12h;
# svc_bucket list | grep -c CAS
# svc_replicate summary;svc_replicate stats
# svc_rest_cmd /vdc/sensor/status/ObjectGCObjectCleanup
# svc_exec uptime
# svc_exec free -m