Article Number: 541569

printer Print mail Email

Recoverpoint for VM: CG stuck in pausing on old image and Replication process repeatedly crashes after trying to exit image access.

Summary: CG stuck in pausing on old image and Replication process repeatedly crashes after trying to exit image access.

Primary Product: RecoverPoint for Virtual Machines

Product: RecoverPoint for Virtual Machines more...

Last Published: 21 Apr 2020

Article Type: Break Fix

Published Status: Online

Version: 7

Recoverpoint for VM: CG stuck in pausing on old image and Replication process repeatedly crashes after trying to exit image access.

Article Content

Issue


After Image Access/Test Copy CG stuck in pausing on old image and Replication process repeatedly crashes after trying to exit image access.
Symptom: 
Any operation on the affected group copy results in a deadlock and Replication process crash.
Journal status report as inaccessible.

Symptoms found in the logs: 
Replication logs showed CG is stuck in e_distributorPausing and deadlocks appear after trying to exit image access:
2020/01/28 00:45:44.991 - #2 - 20242/19532 - DistributorGroupHandler::traceOnTimer: dist-trace kVolSlot=346373634 copy=1 grid=0 e_distributorPausing -init -regulated
...
2020/01/28 00:58:10.692 - #2 - 20242/19532 - DistributorGroupHandler::traceOnTimer: dist-trace kVolSlot=346373634 copy=1 grid=0 e_distributorPausing -init -regulated
...
2020/01/28 01:04:55.845 - #2 - 16732/16542 - DistributorGroupHandler::doOpen: end dist-trace kVolSlot=346373634 copy=1 grid=0 e_distributorPausing -init
+ streams usage (blockSize 120000000 maxJournalSize 1200000000000)
- pool block = 3 free out of 712
...
+ distribution task e_distributeBackward startTime=(distributorSession=18,timeStamp=(transactionCounter=1229026,timeCounter=1580045017872026,dataCounter=131314324)) endTime=(distributorSession=16,timeStamp=(transactionCounter=24104682,timeCounter=1580019732309495,dataCounter=2393824946)) sourceFrom=StreamPointer(streamID=11798064549800706049 blockID=284 offset=34696016) sourceTo=StreamPointer(streamID=11798064549800706049 blockID=438 offset=116659781) destStart=StreamPointer(streamID=2618831595494178817 blockID=309 offset=68098848) destFrom=StreamPointer(streamID=2618831595494178817 blockID=709 offset=113772930)
...
there are 197 undo snapshots
streamId=11798064549800706049
start time (24104682/1580019732309495/2393824946) end time (24107889/1580019788235757/2394198014) Type e_gridRegularSnapshot policy e_allwaysConsolidate size 183755124 uncompressed size 191012529 saved space 0 MetaData NoOption start (block=438 offset=116659781) end (block=441 offset=60414905) id Option(73087) approved 0 event NoOptione_NonConsolidatedSnapshot
...
2020/01/28 01:04:57.004 - #2 - 16665/16542 - Distributor_AO_IMPL::resumeDistribution: enter groupID = (groupCopyRID=(kVolSlot=346373634,globalCopyID=GlobalCopy(SiteUID(0x11d65f1a7e028d45) 1) ),gridCopyID=0) sessionID = 16262658130653280228
2020/01/28 01:04:57.004 - #2 - 16665/16542 - AsyncDistPhase2::stop: m_groupId = (groupCopyRID=(kVolSlot=346373634,globalCopyID=GlobalCopy(SiteUID(0x11d65f1a7e028d45) 1) ),gridCopyID=0)
...
2020/01/28 01:24:52.576 - #1 - 16562/16542 - DLManager: deadlock suspected at N6Kashya19Distributor_AO_IMPLE, pid=16729/16542, config=(alertThreshold=10,alertLevel=1,killThreshold=0,coreThreshold=0,full=1), deltaTime=1069, cmd=N6Kashya46Distributor_AO_IMPL_handleTimer_48_CmdConcreteE
2020/01/28 01:24:52.576 - #1 - 16562/16542 - DLManager: deadlock suspected at N6Kashya26ReplicationControl_AO_IMPLE, pid=16665/16542, config=(alertThreshold=10,alertLevel=1,killThreshold=0,coreThreshold=0,full=1), deltaTime=1071, cmd=N6Kashya33SerializableMethodRequestRefCountE
2020/01/28 01:25:04.551 LOG STARTED HERE
...

Cause
Not enough free journal blocks for distribution and trying to distribute the oldest undo snapshot.
Resolution
Workaround:
A solution exists for this issue but intervention from Dell EMC technical support personnel is required. Contact the Dell EMC Customer Support Center or your service representative for technical assistance and reference this Dell EMC knowledgebase solution ID.

Resolution:

This issue is addressed in the RecoverPoint 5.2.2.1 version.

To determine whether an upgrade is appropriate for your environment, contact the Dell EMC Customer Support Center or your service representative and reference this solution ID.

Notes

Issue


After Image Access/Test Copy CG stuck in pausing on old image and Replication process repeatedly crashes after trying to exit image access.
Symptom: 
Any operation on the affected group copy results in a deadlock and Replication process crash.
Journal status report as inaccessible.

Symptoms found in the logs: 
Replication logs showed CG is stuck in e_distributorPausing and deadlocks appear after trying to exit image access:
2020/01/28 00:45:44.991 - #2 - 20242/19532 - DistributorGroupHandler::traceOnTimer: dist-trace kVolSlot=346373634 copy=1 grid=0 e_distributorPausing -init -regulated
...
2020/01/28 00:58:10.692 - #2 - 20242/19532 - DistributorGroupHandler::traceOnTimer: dist-trace kVolSlot=346373634 copy=1 grid=0 e_distributorPausing -init -regulated
...
2020/01/28 01:04:55.845 - #2 - 16732/16542 - DistributorGroupHandler::doOpen: end dist-trace kVolSlot=346373634 copy=1 grid=0 e_distributorPausing -init
+ streams usage (blockSize 120000000 maxJournalSize 1200000000000)
- pool block = 3 free out of 712
...
+ distribution task e_distributeBackward startTime=(distributorSession=18,timeStamp=(transactionCounter=1229026,timeCounter=1580045017872026,dataCounter=131314324)) endTime=(distributorSession=16,timeStamp=(transactionCounter=24104682,timeCounter=1580019732309495,dataCounter=2393824946)) sourceFrom=StreamPointer(streamID=11798064549800706049 blockID=284 offset=34696016) sourceTo=StreamPointer(streamID=11798064549800706049 blockID=438 offset=116659781) destStart=StreamPointer(streamID=2618831595494178817 blockID=309 offset=68098848) destFrom=StreamPointer(streamID=2618831595494178817 blockID=709 offset=113772930)
...
there are 197 undo snapshots
streamId=11798064549800706049
start time (24104682/1580019732309495/2393824946) end time (24107889/1580019788235757/2394198014) Type e_gridRegularSnapshot policy e_allwaysConsolidate size 183755124 uncompressed size 191012529 saved space 0 MetaData NoOption start (block=438 offset=116659781) end (block=441 offset=60414905) id Option(73087) approved 0 event NoOptione_NonConsolidatedSnapshot
...
2020/01/28 01:04:57.004 - #2 - 16665/16542 - Distributor_AO_IMPL::resumeDistribution: enter groupID = (groupCopyRID=(kVolSlot=346373634,globalCopyID=GlobalCopy(SiteUID(0x11d65f1a7e028d45) 1) ),gridCopyID=0) sessionID = 16262658130653280228
2020/01/28 01:04:57.004 - #2 - 16665/16542 - AsyncDistPhase2::stop: m_groupId = (groupCopyRID=(kVolSlot=346373634,globalCopyID=GlobalCopy(SiteUID(0x11d65f1a7e028d45) 1) ),gridCopyID=0)
...
2020/01/28 01:24:52.576 - #1 - 16562/16542 - DLManager: deadlock suspected at N6Kashya19Distributor_AO_IMPLE, pid=16729/16542, config=(alertThreshold=10,alertLevel=1,killThreshold=0,coreThreshold=0,full=1), deltaTime=1069, cmd=N6Kashya46Distributor_AO_IMPL_handleTimer_48_CmdConcreteE
2020/01/28 01:24:52.576 - #1 - 16562/16542 - DLManager: deadlock suspected at N6Kashya26ReplicationControl_AO_IMPLE, pid=16665/16542, config=(alertThreshold=10,alertLevel=1,killThreshold=0,coreThreshold=0,full=1), deltaTime=1071, cmd=N6Kashya33SerializableMethodRequestRefCountE
2020/01/28 01:25:04.551 LOG STARTED HERE
...

Cause
Not enough free journal blocks for distribution and trying to distribute the oldest undo snapshot.
Resolution

Workaround:
A solution exists for this issue but intervention from Dell EMC technical support personnel is required. Contact the Dell EMC Customer Support Center or your service representative for technical assistance and reference this Dell EMC knowledgebase solution ID.

Resolution:

This issue is addressed in the RecoverPoint 5.2.2.1 version.

To determine whether an upgrade is appropriate for your environment, contact the Dell EMC Customer Support Center or your service representative and reference this solution ID.

Notes

Article Attachments

Attachments

Attachments

Article Properties

First Published

Thu Feb 27 2020 03:36:37 GMT

First Published

Thu Feb 27 2020 03:36:37 GMT

Rate this article

Accurate
Useful
Easy to understand
Was this article helpful?
0/3000 characters