ECS: GC will not remove CAS Blobs - Unable to delete CAS bucket

Summary: CAS GC Troubleshooting

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.
Check out other resources

Symptoms

GC sweeps the CAS ClipID's but not the blobs. Some time a customer tries to remove an empty CAS bucket and get an error "Bucket is not empty."

Cause

Working as designed

Resolution

Basic steps to follow for ensuring GC is working:
1. Connect to one ECS node over ssh and user admin.
2. Check whether CAS GC is enabled: 
admin@ecsnode01:~> svc_gc config
svc_gc v3.4.2 (svc_tools v2.15.0)                 Started 2024-02-02 09:30:39

Local node ECS Object Version: 3.8.0.3-138685.3a0a9b6bf3a (3.8.0.3 GA)
Local VDC: urn:storageos:VirtualDataCenterData:cf84a100-bded-4e00-b5d3-8bf5f01495af vdc_s3_j07

Current Param values:

                                                                        Default              Current             
Type     Param name                                                     Value                Value               

Repo     com.emc.ecs.chunk.gc.repo.enabled                              true                 true                
Repo     com.emc.ecs.chunk.gc.repo.verification.enabled                 true                 true                
Repo     com.emc.ecs.chunk.gc.repo.reclaimer.no_recycle_window          78 hours             78 hours            

BTREE_L1 com.emc.ecs.chunk.gc.btree.enabled                             true                 true                
BTREE_L1 com.emc.ecs.chunk.gc.btree.scanner.verification.enabled        true                 true                
BTREE_L1 com.emc.ecs.chunk.gc.btree.scanner.copy.enabled                true                 true                

BTREE_L2 com.emc.ecs.chunk.gc.btree.reclaimer.level2.enabled            true                 true                
BTREE_L2 com.emc.ecs.chunk.gc.btree.occupancyScanner.level2.enabled     true                 true                

Partial  com.emc.ecs.chunk.gc.repo.partial.enabled                      true                 true                
Partial  com.emc.ecs.chunk.gc.repo.partial.merge_chunk_threshold        89478400             89478400            
Partial  com.emc.ecs.chunk.gc.repo.partial.merge_old_chunk_threshold    89478400             89478400            

Journal  com.emc.ecs.chunk.gc.journal.enabled                           true                 true                
Journal  com.emc.ecs.prtable.gc.enabled                                 true                 true                
Journal  com.emc.ecs.prtable.gc.record_expiration                       14 days              14 days             
Journal  com.emc.ecs.chunk.gc.journal.protection_period                 14 days              14 days             

CAS      com.emc.ecs.objectgc.cas.enabled                               true                 true                
CAS      com.emc.ecs.objectgc.cas.process_update.enabled                true                 true                
CAS      com.emc.ecs.objectgc.cas.process_object.enabled                true                 true                
CAS      com.emc.ecs.objectgc.cas.process_audit.enabled                 true                 true                
CAS      com.emc.ecs.objectgc.cas.consistency_scanner.enabled           true                 true                
CAS      com.emc.ecs.objectgc.cas.process_object.dry_run                false                false               

admin@ecsnode01:~>
All returning "Current" readings should match "Default."

3. Check whether the reflection expiration is set:
UI showing Manage -> Buckets -> Reflection expiration setting


4. If it is a bucket remove issue, verify that the bucket is empty by running JCAS or another equivalent tool for running a Query:

CASScript>querySetExisting

Query Expression Properties:
        Start Time: UNBOUNDED
        End Time  : UNBOUNDED
        Type      : EXISTING

CASScript>query

Performing clip query

Query Expression Properties:
        Start Time: UNBOUNDED
        End Time  : UNBOUNDED
        Type      : EXISTING

Number of clips returned:     0
All results returned:         True


5. CAS objects have a 28-day quarantine period. Confirm that the time has lapsed. This means it can take up to 28 days for the associated blob to be removed when the clip is deleted.

If all previous steps are cleared and the bucket is still unable to be deleted, open a Service Request with Support.

 
 

Affected Products

ECS Appliance

Products

ECS Appliance, Elastic Cloud Storage
Article Properties
Article Number: 000080672
Article Type: Solution
Last Modified: 05 Feb 2024
Version:  4
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.