PowerProtect: Virtual Machine Restore Reported as Success but Has Failed

• VM restores reported as Success but has failed. 
• PowerProtect Data Manager version is 19.16 or 19.17
• vproxyd.service (embedded VM Direct Engine) or vbackupd.service (external VM Direct Engine) is observed as restarting at the time of the issue.

When the restore was completed, the vproxyd-daemon or vbackupd-daemon log indicates that the Memory Monitor had restarted due to the internal memory limit being reached.

In the log extract below, the requested shutdown is observed and the check for any running backups or restores. However, despite the report indicating "There are 0 sessions running", a restore was active then.

Log: vproxyd-daemon.log | vbackupd-daemon.log

Location: /opt/emc/vproxy/runtime/logs/vproxyd/ | /opt/emc/vproxy/runtime/logs/vbackupd/

2024-10-18T08:38:09.881Z DEBUG:  Request #53366: Responded to client.
2024-10-18T08:38:11.307Z DEBUG:  Request #53367: Checking HTTP headers of REST request...
2024-10-18T08:38:11.307Z DEBUG:  New get status request: ALL
2024-10-18T08:38:11.307Z INFO:   The state of the vProxy is: READY
2024-10-18T08:38:11.418Z DEBUG:  Retrieving process status successful
2024-10-18T08:38:11.418Z DEBUG:  Request #53367: Responded to client.
2024-10-18T08:38:29.494Z DEBUG:  Memory monitor: 1369 MB at 24.00 hours. Memory limit is 1024 MB. Minimum alive time is 24.00 hours.
2024-10-18T08:38:29.494Z DEBUG:  Memory monitor: Running forced garbage collection ...
2024-10-18T08:38:29.494Z DEBUG:  Memory monitor: 1369 MB before GC.
2024-10-18T08:38:29.496Z DEBUG:  Memory monitor: 1366 MB after GC. Saved 18446744073709551613 MB.
2024-10-18T08:38:29.496Z WARN:   Memory monitor: Memory used (1366 MB) exceeds the specified threshold (1024 MB)
2024-10-18T08:38:29.496Z DEBUG:  Memory monitor: There are 0 sessions running on the backup engine.
2024-10-18T08:38:29.496Z DEBUG:  Memory monitor: There are 0 sessions running on the flr engine.
2024-10-18T08:38:29.496Z DEBUG:  Memory monitor: There are 0 sessions running on the recover engine.
2024-10-18T08:38:29.496Z DEBUG:  Memory monitor: There are 0 sessions running on the vsessions engine.
2024-10-18T08:38:29.496Z NOTICE: Memory monitor: Out of memory. Shutting down ...
2024-10-18T08:38:29.496Z INFO:   Memory monitor: Monitoring has ended.
2024-10-18T08:38:29.497Z INFO:   REST listener has shut down.
2024-10-18T08:38:29.497Z INFO:   REST service has ended.

As of PowerProtect Data Manager 19.16, there is an integrated memory manager that restarts the vbackupd or vproxyd service on the VM Direct Engines if required. This process should ensure that no backup or restore is running before carrying out this maintenance action. However, PowerProtect Data Manager versions 19.16 and 19.17 have an issue that allowed the service to restart even when a restore was active.
 

As a result, the restore job would fail, but the UI would report it as a success.
 

This issue has been resolved as of PowerProtect Data Manager 19.18 and later versions.

Resolution:

Upgrade to 19.18 or above

Temporary workaround:

Increase the default memory limit on the VM Direct Engines. There should be no VM backups running when these steps are carried out.

1. SSH to PowerProtect Data Manager and switch to the root user, or use sudo for elevated privileges.

2. Back up the environmental file

cp -a /opt/emc/vproxy/unit/vproxy.env /opt/emc/vproxy/unit/vproxy.env.bak

3. Edit the parameter file to increase the VPROXY_MEMORY_LIMIT_MB variable and save changes. 

vim /opt/emc/vproxy/unit/vproxy.env

Increase the value for VPROXY_MEMORY_LIMIT_MB from 1024 to 2048.

4. Restart vproxyd for changes to take effect, when no VM backup or restore is running.

systemctl restart vproxyd.service