PowerProtect Data Manager: Search Engine VM is Stuck in Restart Mode

Search Engine VM cannot be started, and is stuck in restart mode.

From PowerProtect Data Manager UI, cluster is not operational as one or more nodes are in a failed state.

Figure 1: Screenshot of Search Engine Error Message

The root cause of the issue is that the /tmp directory is full on the search engine VM which affects the operating system. 

In version 19.12 of PowerProtect Data Manager, the size of the /tmp directory in the search engine is not monitored. This issue is addressed in 19.13 release.

Workaround:

1.  SSH to the PowerProtect Data Manager and elevate to root user.
2. Get the Search Engine VM credentials with the following commands

source /opt/emc/vmdirect/unit/vmdirect.env  && 
/opt/emc/vmdirect/bin/infranodemgmt get -secret

3. Connect to the vCenter and open the Remote Console for the Search Node VM. Use Single-User Mode to log in. Dell article 181219, PowerProtect Software: How to reset PPDM root password from single user mode provides steps on how to access the VM in single-user mode.
4. Check if the /tmp directory is full. Run the following commands:

• Run the following command to check the available space on the /tmp directory:

df -h /tmp

• Run the following command find top 10 largest files in /tmp directory:

sudo find /tmp -type f -printf '%s %p\n' | sort -nr | head -10

5. Remove the top largest files and then run the following command again to see whether there is enough space on the /tmp directory.

df -h /tmp

6. Restart the Search engine VM. 
7. Check the Search Engine state on PowerProtect Data Manager and ensure that it is operational.