The PowerProtect Data Manager is used to protect the vSphere environment with the Transparent Snapshot Data Mover (TSDM) solution. A few VM Asset backups started to fail, when they were completing successfully in the past. The failure details indicate the vCenter task fails with a TSDM error indicating "DPD Sync in progress." The ABV0016 and ABV0006 details show:
ABV0016: VM Direct engine 'localhost' is unable to back up the virtual machine 'my-vm-name' on vCenter 'my-vcernter-name' because of a vCenter API issue. The VM Direct engine cannot complete the backup of this virtual machine because of a vCenter API operation did not complete successfully. To troubleshoot this issue: 1) Verify that the vCenter can be reached. 2) If the vCenter is busy, wait for some activities to complete and then retry the operation. 3) Verify that the virtual machine exists on the vCenter. Read Less Unable to cleanup from earlier backup attempt. Unable to cleanup session 'Backup:6755cb99-3de5-4731-8b6f-6c64bb809f0b' from virtual machine 'my-vm-name' (MORef = 'vm-241'). Unable to cancel TSDM / LWD snapshot. The vCenter task for 'Cancel SDM Snapshot' completed with state 'error': DPD Sync in progress.
ABV0006: VM Direct engine 'localhost' is unable to back up the virtual machine 'my-vm-name' on vCenter 'my-vcernter-name' because of a vCenter API issue. The VM Direct engine cannot complete the backup of this virtual machine because of a vCenter API operation did not complete successfully. To troubleshoot this issue: 1) Verify that the vCenter can be reached. 2) If the vCenter is busy, wait for some activities to complete and then retry the operation. 3) Verify that the virtual machine exists on the vCenter. Read Less vCenter task 'task-943698' failed: VimFault: 'dp.vpx.fault.SnapshotInProgress.summary': DPD Snapshot in progress
... YYYY-MM-DD HH:MM:SS INFO: [acae20b6ae1ca9c8;aac63ea8d0fb1d9e] }, YYYY-MM-DD HH:MM:SS INFO: [acae20b6ae1ca9c8;aac63ea8d0fb1d9e] "Status": { YYYY-MM-DD HH:MM:SS INFO: [acae20b6ae1ca9c8;aac63ea8d0fb1d9e] "State": "Failure", YYYY-MM-DD HH:MM:SS INFO: [acae20b6ae1ca9c8;aac63ea8d0fb1d9e] "Description": "Backup is unsuccessful: Unable to cleanup from earlier backup attempt. Unable to cleanup session 'Backup:6755cb99-3de5-4731-8b6f-6c64bb809f0b' from virtual machine 'vm-vm-name' (MORef = 'vm-241'). Unable to cancel TSDM / LWD snapshot. The vCenter task for 'Cancel SDM Snapshot' completed with state 'error': DPD Sync in progress", YYYY-MM-DD HH:MM:SS INFO: [acae20b6ae1ca9c8;aac63ea8d0fb1d9e] "Progress": 0, YYYY-MM-DD HH:MM:SS INFO: [acae20b6ae1ca9c8;aac63ea8d0fb1d9e] "Error": { YYYY-MM-DD HH:MM:SS INFO: [acae20b6ae1ca9c8;aac63ea8d0fb1d9e] "Text": "Unable to cleanup from earlier backup attempt. Unable to cleanup session 'Backup:6755cb99-3de5-4731-8b6f-6c64bb809f0b' from virtual machine 'vm-vm-name' (MORef = 'vm-241'). Unable to cancel TSDM / LWD snapshot. The vCenter task for 'Cancel SDM Snapshot' completed with state 'error': DPD Sync in progress", YYYY-MM-DD HH:MM:SS INFO: [acae20b6ae1ca9c8;aac63ea8d0fb1d9e] "Code": 0, YYYY-MM-DD HH:MM:SS INFO: [acae20b6ae1ca9c8;aac63ea8d0fb1d9e] "Classifications": [], YYYY-MM-DD HH:MM:SS INFO: [acae20b6ae1ca9c8;aac63ea8d0fb1d9e] "Retryable": false, YYYY-MM-DD HH:MM:SS INFO: [acae20b6ae1ca9c8;aac63ea8d0fb1d9e] "Remedy": null YYYY-MM-DD HH:MM:SS INFO: [acae20b6ae1ca9c8;aac63ea8d0fb1d9e] }, ...
... YYYY-MM-DD HH:MM:SS INFO: [a3f45e322add25ec;bb6e18b9d1e8178c] }, YYYY-MM-DD HH:MM:SS INFO: [a3f45e322add25ec;bb6e18b9d1e8178c] "Status": { YYYY-MM-DD HH:MM:SS INFO: [a3f45e322add25ec;bb6e18b9d1e8178c] "State": "Failure", YYYY-MM-DD HH:MM:SS INFO: [a3f45e322add25ec;bb6e18b9d1e8178c] "Description": "Backup is unsuccessful: vCenter task 'task-943698' failed: VimFault: 'dp.vpx.fault.SnapshotInProgress.summary': DPD Snapshot in progress, TSDM snapshot UUID = '52031978-18dd-444a-37fd-9c070de84cc5'", YYYY-MM-DD HH:MM:SS INFO: [a3f45e322add25ec;bb6e18b9d1e8178c] "Progress": 0, YYYY-MM-DD HH:MM:SS INFO: [a3f45e322add25ec;bb6e18b9d1e8178c] "Error": { YYYY-MM-DD HH:MM:SS INFO: [a3f45e322add25ec;bb6e18b9d1e8178c] "Text": "vCenter task 'task-943698' failed: VimFault: 'dp.vpx.fault.SnapshotInProgress.summary': DPD Snapshot in progress, TSDM snapshot UUID = '52031978-18dd-444a-37fd-9c070de84cc5'", YYYY-MM-DD HH:MM:SS INFO: [a3f45e322add25ec;bb6e18b9d1e8178c] "Code": 0, YYYY-MM-DD HH:MM:SS INFO: [a3f45e322add25ec;bb6e18b9d1e8178c] "Classifications": [], YYYY-MM-DD HH:MM:SS INFO: [a3f45e322add25ec;bb6e18b9d1e8178c] "Retryable": false, YYYY-MM-DD HH:MM:SS INFO: [a3f45e322add25ec;bb6e18b9d1e8178c] "Remedy": null YYYY-MM-DD HH:MM:SS INFO: [a3f45e322add25ec;bb6e18b9d1e8178c] }, ...
The SDM feature should be disabled and enabled on the affected VM to clear the orphan vSphere SDM snapshot task. Here are some of the available methods to disable and enabled the SDM on the affected VM.
Change the affected VM asset protection mechanism.
Remove and add the affected VM from the Protection Policy.
The PowerProtect Data Manager vmconfig command-line utility can be used to change the SDM feature setting for the affected VM.
ssh admin@my-ppdm-name
sudo su
source /opt/emc/vproxy/unit/vproxy.env
cd /opt/emc/vproxy/bin/
./vmconfig -v <vcenter-ip-fqdn> -u '<vcenter-user-name>' -p '<vcenter-user-pwd>' -l dns -k <VM DNS name> -c info
./vmconfig -v <vcenter-ip-fqdn> -u '<vcenter-user-name>' -p '<vcenter-user-pwd>' -l dns -k <VM DNS name> -c disable-sdm
./vmconfig -v <vcenter-ip-fqdn> -u '<vcenter-user-name>' -p '<vcenter-user-pwd>' -l dns -k <VM DNS name> -c enable-sdm