PPDM - Application aware VM restore failed with "Unable to reset disks to read/write mode after VSS quiesce operation set them to read-only"
Summary: Application aware VM restore failed with "Unable to reset disks to read/write mode after VSS quiesce operation set them to read-only" due to the lack of the memory on the search node.
Symptoms
Unable to reset disks to read-write mode after VSS quiesce operation set them to read-only
In "PowerProtect Data Manager_*******.log"
vmdm.log: 2024-06-19T01:51:39.916Z ERROR [] [request-dispatch-update-91] [00000000-0000-4000-a000-000000000000][][][TRACE_ID:b78abd00d05a58d4;JOB_ID:af29df7e344de2ee][] [c.e.b.v.r.a.BaseRestoreOp.doFailureStatus(561)] - Recover VM failed. vmdm.log: 2024-06-19T01:51:39.917Z ERROR [] [request-dispatch-update-91] [00000000-0000-4000-a000-000000000000][][][TRACE_ID:b78abd00d05a58d4;JOB_ID:af29df7e344de2ee][] [c.e.b.v.r.a.BaseRestoreOp.getSessionError(740)] - VM Proxy restore job session failed. Session '8fbda571-ab2d-4597-8fe7-4791993b836d' is unsuccessful: Unable to reset disks to read-write mode after VSS quiesce operation set them to read-only.
Vmdirect_******.log reported below WARNING, indicating the lack of memory, before throwing out the ERROR to reset disks to read/write mode.
2024-06-19 09:36:57+0800 164 WARNING ppdm.local 58736 processmanager.go(336) main.(*ProcessManager).isThresholdMemoryConsumptionReached Available system memory (364 MB) is less than minimum required system resources (800 MB)
2024-06-19 09:36:57+0800 165 WARNING ppdm.local 58736 processmanager.go(251) main.(*ProcessManager).waitForCurrentSearchClientProcessesToComplete Max number of vm-file-extractor processes are running. Cancelling run of vm-file-extractor for vm EC020DB01 after 10 minutes
2024-06-19 09:36:57+0800 166 WARNING ppdm.local 58736 main.go(323) main.main Max number of vm-file-extractor processes are running. Cancelling the process after 10 minutes as not enough system resources are available. The metadata extraction and indexing of this VM will be handled later by the offline-indexing process.
2024-06-19 09:36:57.001+0800 167 ERROR ppdm.local 58736 main.go(838) main.exitWithError abort extraction with Max number of vm-file-extractor processes are running. Cancelling the process after 10 minutes as not enough system resources are available. The metadata extraction and indexing of this VM will be handled later by the offline-indexing process.
2024-06-19 09:36:57.001+0800 169 ERROR ppdm.local 58736 main.go(848) main.exitWithError {"State":"Skipped","Error":"Max number of vm-file-extractor processes are running. Cancelling the process after 10 minutes as not enough system resources are available. The metadata extraction and indexing of this VM will be handled later by the offline-indexing process.","LogFile":"","DetailedStatus":"{\"State\":\"Skipped\",\"Error\":\"\",\"DiskStatus\":[]}"}
2024-06-19 09:36:57.001+0800 170 ERROR ppdm.local 58736 main.go(855) main.exitWithError abort extraction of VM metadata for asset EC020DB01 (vm-53132)
RecoverVmSessions-b1d958e1-f05b-46a8-af3c-d7d1e5840631.log: 2024-06-19T01:36:57.092Z WARN: [aaa7a1018b9ad483;86443860fffb28d7] VM File Extractor plugin was unsuccessful. Max number of vm-file-extractor processes are running. Cancelling the process after 10 minutes as not enough system resources are available. The metadata extraction and indexing of this VM will be handled later by the offline-indexing process.
RecoverVmSessions-b1d958e1-f05b-46a8-af3c-d7d1e5840631.log: 2024-06-19T01:36:57.092Z ERROR: [aaa7a1018b9ad483;86443860fffb28d7] Unable to repair disk properties which were modified by the VSS Quiesce operation. VM File Extractor plugin was unsuccessful. Max number of vm-file-extractor processes are running. Cancelling the process after 10 minutes as not enough system resources are available. The metadata extraction and indexing of this VM will be handled later by the offline-indexing process.
RecoverVmSessions-b1d958e1-f05b-46a8-af3c-d7d1e5840631.log: 2024-06-19T01:36:57.092Z ERROR: [aaa7a1018b9ad483;86443860fffb28d7] Unable to reset disks to read-write mode after VSS quiesce operation set them to read-only.Cause
The process vm-file-extractor cannot be run due to the lack of the memory on the PowerProtect Data Manager search node. The minimum requirement of the free memory is 800 MB. The failure of running vm-file-extractor caused the read-only bit not set properly. Thus, leading to the error "Unable to reset disks to read/write mode after VSS quiesce operation set them to read-only."
Resolution
1. Log in to the PowerProtect Data Manager appliance using SSH, switch to the root user, and run below commands to get the password of the PowerProtect Data Manager search node.
source /opt/emc/vmdirect/unit/vmdirect.env && /opt/emc/vmdirect/bin/infranodemgmt get -secret -node_type SearchNode
2. Log in to each search node using the credentials got in step 1.
3. Run below command to check the memory usage of the search node.
free -m
4. If the free memory is indeed less than 800 MB, add more memory to the search node, or reboot the search node when PowerProtect Data Manager is idle to free up the allocated memory.
5. Rerun the VM restore.