RecoverPoint and RecoverPoint for VMs: CLI session ends abnormally with "Killed" message when commands are run.
Summary: RecoverPoint: CLI session ends abnormally with "Killed" message; this message may not appear if the session is run through PuTTY, as PuTTY may close the window.
Symptoms
Many combined CLI changes in a large-scale environment can cause this issue. Running commands in large environments may cause CLI sessions to close abnormally.
If the window remains open, the user may notice a "Killed" message:
Cluster1> add_replication_volume group=CG copy=Copy1 replication_set=RSet1 name=device_DEV01 Cluster1> add_replication_volume group=CG copy=Copy1 replication_set=RSet2 name=device_DEV02 Cluster1> add_replication_volume group=CG copy=Copy1 replication_set=RSet3 name=device_DEV03 Killed
The following may appear in CLI logs (extracted.*/files/home/kos/cli/result.xx.log):
2018/07/29 21:32:48.614 - #2 - 11940/11940 - CLI: Handling command: create_replication_set 2018/07/29 21:32:48.614 - #2 - 11940/11940 - CLI: In CommandHandler::handleCommand(..) 2018/07/29 21:32:48.614 - #2 - 11940/11940 - CLI: In CreateReplicationSetHandler::initDataMembers() 2018/07/29 21:32:48.614 - #2 - 11940/11940 - CLI: In CommandHandler::initDataMembers() 2018/07/29 21:32:48.614 - #2 - 11940/11940 - CLI: m_settingsCache is relevant, returning 2018/07/29 21:32:48.614 - #2 - 11940/11940 - CLI: Calling Mgmt::getState() 2018/07/29 21:32:49.949 - #2 - 11944/11940 - MemoryShield: totalMemory = 827.188 [MB] resMemory = 331.613 [MB] shrMemory = 73.7656 [MB] 2018/07/29 21:32:49.949 - #0 - 11944/11940 - MemoryShield: errno=0 memory low watermark exceeded, current=347721728 totalMemory = 827.188 [MB] resMemory = 331.613 [MB] shrMemory = 73.7656 [MB], limit=283115520 2018/07/29 21:32:49.949 - #0 - 11944/11940 - MemoryShield: errno=0 memory high watermark exceeded, current=347721728 totalMemory = 827.188 [MB] resMemory = 331.613 [MB] shrMemory = 73.7656 [MB], limit=314572800 2018/07/29 21:32:49.949 - #0 - 11944/11940 - MemoryShield: errno=0 memory shield discovered an error !!! 2018/07/29 21:32:49.949 - #2 - 11944/11940 - DLManager: dumpDetectors 2018/07/29 21:32:49.949 - #2 - 11944/11940 - DLManager: running status for N6Kashya29MemoryShieldScheduler_AO_IMPLE, pid=11944/11940, config=(alertThreshold=10,alertLevel=0,killThreshold=7200,coreThreshold=0,full=1), deltaTime=0, cmd=N6Kashya59MemoryShieldScheduler_AO_IMPL_printStatistics_6_CmdConcreteE 2018/07/29 21:32:49.949 - #0 - 11944/11940 - MemoryShield: errno=0 Assertion failed: false Line 285 File MemoryShield.cc PID: 11940 Info: memory limit exceeded, no core END OF LOGGER
Cause
Due to the size of the environment and many changes being combined within the same CLI session, the CLI process reaches its memory limit and crashes.
For example, when creating a Consistency Group, many changes are combined in the edit_group_settings command as a single change. Another example is "heavier" commands like get_group_statistics.
Resolution
Workaround:
The following signed script can be run on the Cluster IP using putty/ssh > [2] Setup -> [8] Advanced options -> [4] Run script.
ZWE0M2YwMTIzYzg5ZWRhZjk2MzIwYzAzNzA0MzI5YzIKdW5saW1pdGVkCm5vdF9yZXN0cmljdGVk ClRoZSBpZCBvZiB0aGUgc2NyaXB0IGlzOjAwMDE2NzExNQpDcmVhdGUgYW4gb3V0cHV0IGZpbGUg Zm9yIGdyb3VwIHN0YXRpc3RpY3MgaW4gbGFyZ2UgZW52aXJvbm1lbnRzCk5pdiBOZXRhbmVsaQoj IS9iaW4vY3NoCmVjaG8gZ2V0X2dyb3VwX3N0YXRpc3RpY3MgLW4gPiAvaG9tZS93d3cvaW5mby9H cm91cF9TdGF0cy5vdXRwdXQKY2htb2QgK3IgL2hvbWUvd3d3L2luZm8vR3JvdXBfU3RhdHMub3V0 cHV0CmZvcmVhY2ggQ0cgKCBgc3UgYWRtaW4gLWMgImdldF9ncm91cHMgLW4iIHwgZWdyZXAgIl4g IFtbOmFsbnVtOl1dIiB8IHNlZCBzLVwgXCAtLWcgfCBzZWQgcy1cOi0tZ2AgKQojICBlY2hvICRD RwogIHN1IGFkbWluIC1jICJnZXRfZ3JvdXBfc3RhdGlzdGljcyBncm91cD0kQ0ciID4+IC9ob21l L3d3dy9pbmZvL0dyb3VwX1N0YXRzLm91dHB1dAplbmQKc2V0ZW52IEVUSCBgZ3JlcCBldGggL2hv bWUva29zL2NvbnRyb2wvYWNxdWlyZUlQLnRjc2ggfCBhd2sgJ3twcmludCAkN30nIHxncmVwIGV0 aGAKc2V0ZW52IElQIGBpcCBhZGQgfCBncmVwICJnbG9iYWwgJEVUSCIgfCBoZWFkIC1uMXxhd2sg J3twcmludCAkMn0nIHwgc2VkICJzL1wvLiovLyJgCmVjaG8gR3JvdXAgU3RhdGlzdGljcyBmaWxl IGF2YWlsYWJsZSBodHRwczovLyRJUC9pbmZvCmV4aXQgMAo= #
The output file is created on https://{RPA-IP}/info/Group_Stats.output.
Workaround 2:
A solution exists for this issue, but intervention from Dell Technologies technical support personnel is required. Support personnel must access your Site Control appliances for an internal change. Contact the Dell Technologies Customer Support Center or your service representative for technical support and reference this Dell Technologies knowledgebase solution ID.