RecoverPoint for VMs: Splitter in the 5.3.4.1 and 6.0.1.x generate DCUI login requests

After upgrading RecoverPoint splitter/Kdriver VIB to 5.3.4.1 version, kdriver will generate many DCUI login requests on vCenter.

From /scratch/log/kdriver.log.* on the affected ESXi host:

2024/08/21 14:39:51.268 - #2 - 4561369/4561352 - HostIPDiscoveryMgr_AO_IMPL:  rpEsxInfoScan: /opt/emc/rp/kdriver/bin/rp_rpa_discovery.sh --scan-props & executed successfully
2024/08/21 14:40:26.444 - #2 - 4561369/4561352 - HostIPDiscoveryMgr_AO_IMPL:  rpEsxInfoScan: /opt/emc/rp/kdriver/bin/rp_rpa_discovery.sh --scan-props & executed successfully

From var/log/hostd.log on the affected ESXi host:

2024-08-18T23:32:18.395Z info hostd[2101236] [Originator@6876 sub=Default opID=esxcli_name] Accepted password for user dcui from 127.0.0.1
2024-08-18T23:32:18.395Z warning hostd[2101236] [Originator@6876 sub=Vimsvc opID=esxcli_name] Refresh function is not configured.User data can't be added to scheduler.User name: dcui
2024-08-18T23:32:18.395Z info hostd[2101236] [Originator@6876 sub=Vimsvc.ha-eventmgr opID=esxcli_name] Event 125221 : User dcui@127.0.0.1 logged in as pyvmomi Python/3.8.18 (VMkernel 7.0.3 x86_64)
2024-08-18T23:32:18.466Z info hostd[2100837] [Originator@6876 sub=Solo.VmwareCLI opID=esxcli_name user=dcui] Dispatch list
2024-08-18T23:32:18.475Z info hostd[2100837] [Originator@6876 sub=Solo.VmwareCLI opID=esxcli_name user=dcui] Dispatch list done

After installing or upgrading the RecoverPoint splitter to 6.0.1.x version, it will generate many Direct Console User Interface (DCUI) logins requests on vCenter.

From /scratch/log/iofilterd-emcsplitter.log on the affected ESXi host:

2024-08-16T08:24:40.578Z In(14) iofilterd-emcsplitter[16847250]: spl_run_cmd: running command 'VI_USERNAME=dcui esxcli system settings advanced list -o "/UserVars/RP_IP_Discovery_8" | grep "\s\s\sString Value:" | awk -F "String Value: " {'print $2'}'
2024-08-16T08:24:41.145Z In(14) iofilterd-emcsplitter[16847250]: spl_run_cmd: running command 'VI_USERNAME=dcui esxcli system settings advanced list -o "/UserVars/RP_IP_Discovery_9" | grep "\s\s\sString Value:" | awk -F "String Value: " {'print $2'}'
2024-08-16T08:24:41.473Z In(14) iofilterd-emcsplitter[16847250]: spl_run_cmd: running command 'VI_USERNAME=dcui esxcli system settings advanced set -o "/UserVars/emcsplitter_clusters" -s ""'
2024-08-16T08:24:41.720Z In(14) iofilterd-emcsplitter[16847250]: spl_run_cmd: running command 'VI_USERNAME=dcui esxcli system settings advanced list -o "/UserVars/RP_IP_Discovery_10" | grep "\s\s\sString Value:" | awk -F "String Value: " {'print $2'}'

From var/log/hostd.log on the affected ESXi host:

 

2024-08-16T08:24:40.538Z In(166) Hostd[2099945]: [Originator@6876 sub=Vimsvc.ha-eventmgr opID=esxcli-ed-6cca sid=52e328a0 user=dcui] Event 2652729 : User dcui@127.0.0.1 logged out (login time: Friday, 16 August, 2024 08:24:40 AM, number of API invocations: 7, user agent: pyvmomi Python/3.8.16 (VMkernel; 8.0.1; x86_64))
2024-08-16T08:24:41.059Z In(166) Hostd[2099964]: [Originator@6876 sub=Vimsvc.HaSessionManager opID=esxcli-hostname sid=5284e077] Accepted password for user dcui from 127.0.0.1 - session=5284e077-ac72-8d89-47b2-38feba5f8354
2024-08-16T08:24:41.059Z Wa(164) Hostd[2099964]: [Originator@6876 sub=Vimsvc opID=esxcli-hostname sid=5284e077] Refresh function is not configured.User data can't be added to scheduler.User name: dcui
2024-08-16T08:24:41.059Z In(166) Hostd[2099964]: [Originator@6876 sub=Vimsvc.ha-eventmgr opID=esxcli-hostname sid=5284e077] Event 2652730 : User dcui@127.0.0.1 logged in as pyvmomi Python/3.8.16 (VMkernel; 8.0.1; x86_64)
2024-08-16T08:24:41.109Z In(166) Hostd[2099943]: [Originator@6876 sub=Solo.VmwareCLI opID=esxcli-8f-6cd7 sid=5284e077 user=dcui] Dispatch system.settings.advanced.list
2024-08-16T08:24:41.111Z In(166) Hostd[2099943]: [Originator@6876 sub=Solo.VmwareCLI opID=esxcli-8f-6cd7 sid=5284e077 user=dcui] Dispatch system.settings.advanced.list done
2024-08-16T08:24:41.114Z In(166) Hostd[2099957]: [Originator@6876 sub=Vimsvc.ha-eventmgr opID=esxcli-8f-6cd8 sid=5284e077 user=dcui] Event 2652731 : User dcui@127.0.0.1 logged out (login time: Friday, 16 August, 2024 08:24:41 AM, number of API invocations: 7, user agent: pyvmomi Python/3.8.16 (VMkernel; 8.0.1; x86_64))
2024-08-16T08:24:41.630Z In(166) Hostd[2099938]: [Originator@6876 sub=Vimsvc.HaSessionManager opID=esxcli-hostname sid=520464cb] Accepted password for user dcui from 127.0.0.1 - session=520464cb-08ef-ad94-d4dd-4d02abaf0937
2024-08-16T08:24:41.630Z Wa(164) Hostd[2099938]: [Originator@6876 sub=Vimsvc opID=esxcli-hostname sid=520464cb] Refresh function is not configured.User data can't be added to scheduler.User name: dcui

This login and logout calls are multiplied by the number of hosts the environment has. It generates multiple events on the vCenter, which can potentially fill up the /seat partition causing vCenter to become unresponsive.

5.3.4.1 version

In RecoverPoint version 5.3.4.1, a new script called rp_rpa_discovery.sh was added to the splitter/kdriver VIB. This script is designed to retrieve ESXi host details on a scheduled basis, running every 35 s by default.

The script uses DCUI calls to gather the necessary values for each host, generating frequent login and logout requests for each host in the vCenter. Each command run on ESXi is logged on vCenter's database, and the storage partition containing the database (/seat) can become full. This can cause vCenter to behave abnormally.

The impact of this issue depends on the number of hosts running on the vCenter. The more hosts there is, the quicker the partition can fill up, potentially causing the VXPD service to crash.

This behavior affects hosts running version 7.0.x.

6.0.1.x version

In RecoverPoint version 6.0.1.x, RecoverPoint uses esxcli command to with DCUI to fetch system information. These calls to gather the necessary values for each host, generating frequent login and logout requests for each host in the vCenter. Each command run on ESXi is logged on vCenter's database, and the storage partition containing the database (/seat) can become full. This can cause vCenter to behave abnormally.

Unlike version RecoverPoint 5.3.4.1, no longer manage the execution time of each command.

Workaround:

A - For RecoverPoint for Virtual Machines version 5.3.4.1 and vSphere 7.0.x:

Option 1:

In order to reduce the number of DCUI calls, perform the following steps:
 

1. Open the ssh session to the ESXi.
2. Perform the command below.

sed -i 's/t_RpEsxInfoScanInterval                     = 35000000  # 35 seconds/t_RpEsxInfoScanInterval                     = 15770000000000 # 6 months As per Dell KB123456 # OLD value is 35000000  # 35 seconds/' /etc/config/emc/rp/kdriver/tweak/tweak.params.splitter

3. Validate that value is changed to t_RpEsxInfoScanInterval value to 15770000000000 

cat /etc/config/emc/rp/kdriver/tweak/tweak.params.splitter | grep t_RpEsxInfoScanInterval

4. Restart the splitter 

ps | grep kdriver

pkill -9 kdriver

Option 2:

There is a second option to perform the workaround, Dell Technologies can provide a script to make the changes automatically. 

1. Contact RecoverPoint support to get the script, there are two options one is written in python and other in bash. Script name is kdriver_scantime_change.sh or kdriver_scantime_change.py.
2. Place the script under /tmp on the affected ESXi host.
3. Run command:

   chmod +x kdriver_scantime_change.sh

   
   Or 

   chmod +x kdriver_scantime_change.py

4. Now run the script with the command:

   ./kdriver_scantime_change.sh

   
   Or 

   python kdriver_scantime_change.py

The change makes the scan to run each 6 months, instead of running every 35 s. This change should not impact any communication between RecoverPoint and ESXi hosts.


B - For RecoverPoint for Virtual Machines version 6.0.1.x and vSphere 8.0.x:

Broadcom requests that vCenter sizing must follow their recommendations.

https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.install.doc/GUID-077C7523-E0EA-4922-8D48-C026916323C4.html
 

If you must increase the /seat partition follow the instruction on the Broadcom article:

https://knowledge.broadcom.com/external/article/316602/increasing-the-disk-space-for-the-vcente.html

If the partition is 100% used, see the Broadcom article for cleanup instructions.

https://knowledge.broadcom.com/external/article/318931/storageseat-disk-100-full-on-vcenter-ser.html

C - For RecoverPoint for Virtual Machines version 5.3.4.1 and vSphere 8.0.x
 

This issue is addressed in the RecoverPoint for VMs splitter 5.3.4.1.HF2 version. 

To determine whether an upgrade is appropriate for your environment, contact the Dell Technologies Customer Support Center or your service representative and reference this solution ID.