ECS: xDoctor: RAP075: Space Reclamation verification blocked by ERROR_OBJECT_UPDATE_MISSING or WSCritical: Current index store missing object update

Summary: Space Reclamation (formerly known as Garbage Collection) verification tasks can be blocked by missing object update.

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.
Check out other resources

Symptoms

xDoctor reporting the following error: 
   Category = checks
    Source   = logs
    Severity = ERROR
    Node     = 169.254.1.1
    Message  = Space Reclamation blocked by missing object update
    Extra    = 169.254.1.1
    RAP      = RAP075
    Solution = 522558

         or:

   Category = checks
    Source   = logs
    Severity = ERROR
    Node     = 169.254.1.1
    Message  = Current index store missing object update
    Extra    = 169.254.1.1
    RAP      = RAP075
    Solution = 522558
 
1: One ore more update sequences are missing for an ObjectID. 
Command: 
# svc_log -f "was blocked by" -sr blobsvc -rc -start "1 day ago" | grep -i repo
Example: 
2020-01-27T19:51:22,596 [TaskScheduler-BlobService-REPO_SCAN-ScheduledExecutor-009] ERROR  FileVolatileImage.java (line 2507) WSCritical: current index store missing object update. objectId 9f228fe1be1020ca7d5d174b1636b414e491f35e52e5ba72ec020b75c7b8219e prevUpdateSeq -1 currentSeq 5
2020-01-27T19:51:22,596 [TaskScheduler-BlobService-REPO_SCAN-ScheduledExecutor-009] ERROR  RepoChunkReferenceScanner.java (line 303) WSCritical: [GC] REPO_GC_VERIFICATION_TASK: REPO_GC_Verification Scanner REPO_CHUNK_REFERENCE_SCANNER-urn:storageos:OwnershipInfo:08622646-2ac2-4d98-b9db-a7e3c542cc14_b544eddc-5dfd-43ae-afcf-f5f3b647da89_OB_12_128_0: was blocked by ERROR_OBJECT_UPDATE_MISSING
2020-01-27T19:51:22,596 [TaskScheduler-BlobService-REPO_SCAN-ScheduledExecutor-009] ERROR  Scanner.java (line 276) scanner REPO_CHUNK_REFERENCE_SCANNER-urn:storageos:OwnershipInfo:08622646-2ac2-4d98-b9db-a7e3c542cc14_b544eddc-5dfd-43ae-afcf-f5f3b647da89_OB_12_128_0: caught exception:
com.emc.storageos.data.object.exception.ObjectControllerException
        at com.emc.storageos.data.object.impl.gc.verification.RepoChunk

           

2: Check for "WSCritical: Current index store missing object update" on logs

            Command:

# svc_log -f "WSCritical: current index store missing object update" -files blobsvc* -start 1d -sf | tail -4

            Example:

admin@node01:~> svc_log -f "WSCritical: current index store missing object update" -files blobsvc* -start 1d -sf | tail -4
svc_log v1.0.31 (svc_tools v2.12.3)                 Started 2023-08-30 16:17:54

Running on nodes:              <All nodes>
Time range:                    2023-08-29 16:17:54 - 2023-08-30 16:17:54
Filter string(s):              'WSCritical: current index store missing object update'
File(s) to search:                     blobsvc*

Show filename(s):              True
Show nodename(s):              True
Search reclaim logs (if any):  False

169.254.6.7 blobsvc.log.20230830-045410.gz 2023-08-30T04:41:11,439 [BlobTaskScheduler-disruptor-094] ERROR  FileVolatileImage.java (line 3160) WSCritical: current index store missing object update. objectId 407c5325ae228aed98e7fade7a5ba7731fe88aa6f9d6be7f894917d305d75a2f prevUpdateSeq -1 currentSeq 2939
169.254.6.7 blobsvc.log.20230830-045410.gz 2023-08-30T04:41:11,716 [BlobTaskScheduler-disruptor-043] ERROR  FileVolatileImage.java (line 3160) WSCritical: current index store missing object update. objectId 407c5325ae228aed98e7fade7a5ba7731fe88aa6f9d6be7f894917d305d75a2f prevUpdateSeq -1 currentSeq 2939
169.254.6.7 blobsvc.log.20230830-045410.gz 2023-08-30T04:41:14,941 [BlobTaskScheduler-disruptor-145] ERROR  FileVolatileImage.java (line 3160) WSCritical: current index store missing object update. objectId e9658f99572a0951e343f732553ed85c284b65909c37bdcbeb4870414ae21ec2 prevUpdateSeq -1 currentSeq 2436
169.254.6.7 blobsvc.log.20230830-045410.gz 2023-08-30T04:41:18,323 [BlobTaskScheduler-disruptor-085] ERROR  FileVolatileImage.java (line 3160) WSCritical: current index store missing object update. objectId e9658f99572a0951e343f732553ed85c284b65909c37bdcbeb4870414ae21ec2 prevUpdateSeq -1 currentSeq 2436
169.254.6.7 blobsvc.log.20230830-045410.gz 2023-08-30T04:41:18,685 [BlobTaskScheduler-disruptor-059] ERROR  FileVolatileImage.java (line 3160) WSCritical: current index store missing object update. objectId e9658f99572a0951e343f732553ed85c284b65909c37bdcbeb4870414ae21ec2 prevUpdateSeq -1 currentSeq 2436


3: Check how many objects are impacted.

Command:

# svc_log -f 'WSCritical: current index store missing object update' -files blobsvc.log* -start 10d -sf | awk '{print $16, $17, $18, $19, $20, $21}' | sort | uniq -c          

           Example:

admin@node01:~> svc_log -f 'WSCritical: current index store missing object update' -files blobsvc.log* -start 10d -sf | awk '{print $16, $17, $18, $19, $20, $21}' | sort | uniq -c
svc_log v1.0.31 (svc_tools v2.12.3)                 Started 2023-08-30 16:23:37

Running on nodes:              <All nodes>
Time range:                    2023-08-20 16:23:37 - 2023-08-30 16:23:37
Filter string(s):              'WSCritical: current index store missing object update'
File(s) to search:                     blobsvc.log*

Show filename(s):              True
Show nodename(s):              True
Search reclaim logs (if any):  False

      1
     30 objectId 032498fbfac7ff8a84392b51afcb3887c2625190ce5cf5137e5c067752bfaf87 prevUpdateSeq -1 currentSeq 1161
     16 objectId 04e4231efca0936a0a0eaee85a66711f0b0bc281c8af74277c90d331c7c994e2 prevUpdateSeq -1 currentSeq 2964
      3 objectId 05bc45a702b084268cd94693d4b563b7727c20a94b687e52c669a5f5d3e02c8a ceiling seq 2469 last
    147 objectId 0762d3648efbae33305f9bca99706305941ef21588a87814dfa56549e32fabf0 prevUpdateSeq -1 currentSeq 2956
      1 objectId 078d2fa4723869224c84bbf4e7eeceeb956e1fedf52732b2275e140b32fb7268 ceiling seq 2414 last
     27 objectId 07ea97959a97e31731d9d19dd616226ef3c6cb3c15309c971ca3e0ce788088d3 prevUpdateSeq -1 currentSeq 3082
    150 objectId 08b98aceb1cfc128a5d39be82d7582a10280ab5d50863be2dc46ebd3f63a2172 prevUpdateSeq -1 currentSeq 2959
    138 objectId 09fb1fc026fa27b40f57336dddd64f649785661be030407a671bfed3c91810bb prevUpdateSeq -1 currentSeq 2504

Cause

One ore more update sequences are missing for an ObjectID.

Resolution

Open a Service Request (SR) with ECS technical support and reference this KB.

Affected Products

ECS Appliance

Products

ECS Appliance, Elastic Cloud Storage
Article Properties
Article Number: 000050053
Article Type: Solution
Last Modified: 08 Oct 2025
Version:  9
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.