RecoverPoint for VMs: Consistency Group not SYNC after Failover activity

Summary: Consistency Group not SYNC after Failover activity

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.
Check out other resources

Symptoms

RecoverPoint Plugin GUI not updating correctly, as an example the following behavior may occur:

After performing a Failover activity, the Consistency group was not able to complete the SYNC.
From mango logs (/var/log/tomcat8_9/mango-debug.log): 
2024-03-23 09:45:27.911 ERROR – [http-nio-9020-exec-8] BaseCommand                              : failed to Validate Failover, command=ValidateFailoverCommand(copyId=733820340_5b3cdb93e47a8120_0, startTransfer=false, failoverParameters=FailoverParameters(applyFailoverNetworks=true), targetCopyId=GlobalCopyID [localCopyID=LocalCopyID [id=0], siteUID=SiteUID [6574370984293990697]], groupUID=GroupUID [id=733820340], groupSettingsChangesParam=GroupSettingsChangesParam [groupUID=GroupUID [id=733820340], CG_NAME,
From reaper logs (/var/log/tomcat8_9/reaper-db.log) 
2024-03-22 16:31:16.483  INFO – [pool-102-thread-1] InMemoryDBManageServiceImpl              : Initializing repository for vm
2024-03-22 16:31:16.484 ERROR – [pool-102-thread-1] InitDBManagerService                     : initializing with ProtectedVmsOutOfSync has failed, by exception: com.arangodb.springframework.ArangoUncategorizedException: Response: 500, Error: 1104 - IO error: No space left on deviceWhile appending to file: /var/lib/arangodb3/engine-rocksdb/journals/3721104.log: No space left on device; nested exception is com.arangodb.ArangoDBException: Response: 500, Error: 1104 - IO error: No space left on deviceWhile appending to file: /var/lib/arangodb3/engine-rocksdb/journals/3721104.log: No space left on device

Cause

ArangoDB experiences an I/O failure, resulting in a 'No space left on device' error message in the logs. This specific issue occurs during high I/O operations with ArangoDB and is related to a memory leak.

Resolution

Workaround:

In order to resolve this issue, please apply the script below:
  • Access RPA1 and RPA2 for each cluster using SSH.
  • Login with admin account
  • Select options:
    • [2] Setup
    • [8] Advanced options
    • [4] Run script
    • Paste script:
MzRiMmJkZWVlMmRkMmU5ZTRkYjc1YjlmNmMxMmU0MzIKdW5saW1pdGVkCm5vdF9yZXN0cmljdGVk
ClRoZSBpZCBvZiB0aGUgc2NyaXB0IGlzOlJQRUUtMzY2MgpyZXN0YXJ0IGFyYW5nbwpJYW4gUmFu
c3Ryb20KIyEvYmluL2Jhc2gKcHJpbnRmICcjIS9iaW4vYmFzaFxuXG5sb2dfZmlsZT0iL3Zhci9s
b2cvdG9tY2F0OS9yZWFwZXItZGIubG9nIlxudGV4dF90b19zZWFyY2hfZm9yPSJObyBzcGFjZSBs
ZWZ0Ilxuc2VydmljZV9uYW1lPSJhcmFuZ29kYjMuc2VydmljZSJcblxuZWNobyAiJChkYXRlKSAt
LSBFeGVjdXRpb24gc3RhcnRlZC4gQ2hlY2tpbmcgZm9yIGFyYW5nb2RiMyBtZW1vcnkgaXNzdWUg
aW4gJGxvZ19maWxlIlxudGFpbCAtbiAxMDAgIiRsb2dfZmlsZSIgfCBncmVwIC1xICIkdGV4dF90
b19zZWFyY2hfZm9yIlxuaWYgWyAkPyAtZXEgMCBdOyB0aGVuXG4gICAgIyBJZiB0aGUgdGV4dCBp
cyBmb3VuZCwgcmVzdGFydCB0aGUgc2VydmljZVxuICAgIGVjaG8gIiQoZGF0ZSkgLS0gRm91bmQg
QXJhbmdvRGIgaXNzdWUuIFJlc3RhcnRpbmcgJHNlcnZpY2VfbmFtZSJcbiAgICBzdWRvIHN5c3Rl
bWN0bCByZXN0YXJ0ICIkc2VydmljZV9uYW1lIlxuZWxzZVxuICAgICAgICBlY2hvICIkKGRhdGUp
IC0tIE5vIGFyYW5nb2RiMyBpc3N1ZSBmb3VuZC4gRXhpdGluZyB0aGUgc2NyaXB0LiJcbmZpJyA+
PiAvaG9tZS9rb3Mva2JveC9zcmMvaW5zdGFsbGF0aW9uL2Rpc3RyaWJ1dGlvbi9hcmFuZ29kYl9y
ZXN0YXJ0LnNoCmNobW9kIDc3NyAvaG9tZS9rb3Mva2JveC9zcmMvaW5zdGFsbGF0aW9uL2Rpc3Ry
aWJ1dGlvbi9hcmFuZ29kYl9yZXN0YXJ0LnNoCmVjaG8gIiovMzAgKiAqICogKiAgICByb290ICAg
IC9ob21lL2tvcy9rYm94L3NyYy9pbnN0YWxsYXRpb24vZGlzdHJpYnV0aW9uL2FyYW5nb2RiX3Jl
c3RhcnQuc2ggPj4gL3RtcC9yZXN0YXJ0X2FyYW5nb2RiLmxvZyIgPj4gL2V0Yy9jcm9udGFiCg==
#
  • Apply the script to RPA1 and RPA2 on each RecoverPoint Cluster.
  • Reboot RPA after applying the script.
  • Do one RPA at a time.
Resolution:

Dell Technologies Engineering is currently investigating this issue. A permanent fix is still in progress. Contact the Dell Technologies Customer Support Center or your service representative for assistance and reference this solution ID.
 

Article Properties
Article Number: 000225007
Article Type: Solution
Last Modified: 30 Aug 2024
Version:  2
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.