RecoverPoint: Replication Process Crashes Causing Journal Corruption and Full Sweeping
Summary: Replication Process Crashes Causing Journal Corruption and Full Sweeping a Consistency Group
Symptoms
Due to a code issue, a distribution error can cause journal loss, leading to a Consistency Group (CG) full sweeping and losing all journal entries.
This impacts a standard replica copy, but not Data Domain or XtremIO Arrays.
From within the Replication logs of the Site Control RecoverPoint Appliance, the following is witnessed:
2019/05/25 00:00:43.684 - #2 - 7872/7573 - DistributorGroupHandler::tryDistributeForward: highPerf m_GroupGridCopyRID = (groupCopyRID=(kVolSlot=XXXXXX,globalCopyID=GlobalCopy(SiteUID(0XXXXXXX..a) 0) ),gridCopyID=0)... 2019/05/25 00:00:46.427 - #2 - 7691/7573 - ReplicationControl_AO_IMPL::closePipe_i: enter groupCopyID = GroupCopy(XXXXXXXXXX SiteUID() 0) destCopyID = GlobalCopy(SiteUID(0x0XXXXXX..a) 0)... 2019/05/25 00:00:46.435 - #2 - 7691/7573 - DistributorGroupHandler::closeSession: close session free a_Data.m_bPartialReleaseStarted = 1 a_Data.m_bPartialReleaseSessionStarted = 1 ... a_Data.m_InitDoSnapHead = StreamPointer(streamID=13060353732993089537 blockID=1839 offset=261304942) ... ... 2019/05/25 00:00:46.575 - #2 - 7704/7573 - ReplicationControl_AO_IMPL::openPipe_i: groupCopyID = GroupCopy(XXXXXX SiteUID(0xxxxxx..a) 0)... 2019/05/25 00:01:16.552 - #2 - 10582/7573 - Fetcher_AO_IMPL::fetch_i: ... m_readLoc = StreamPointer(streamID=13060353732993089537 blockID=1839 offset=318942206) 2019/05/25 00:01:16.621 - #2 - 7869/7573 - Fetcher_AO_IMPL::read: a_startPtr = StreamPointer(streamID=13060353732993089537 blockID=1839 offset=318942206) a_isFastForward = 0 a_fastForwardTimeStamp = NoOption 2019/05/25 00:01:16.634 - #0 - 10582/7573 - PersistentObjectVersion: errno=0 Call to deserialization with unknown version=62, current version =16
Cause
During initiation, a pipe closes, the replication process might free journal blocks that the fetcher is not aware of, interrupting a snapshot and leading to corruption.
Resolution
Workaround: On every RecoverPoint Appliance (RPA) at each cluster, change the following tweak from within either the boxmgmt or admin user, depending on your version of RecoverPoint: t_maxHighPerfGroups = 0
The default value is 3 and should be changed to 0.
After this has been done, each RecoverPoint Appliance must be rebooted to ensure that the change takes affect.
Resolution: This issue is resolved in RecoverPoint version 5.1.4.1(5.1 SP4 Patch 1 and later).