Cloud Disaster Recovery (CDR): VM Protection Upload Fails for Virtual Machines Added as A Part of A Container Client in Avamar
Summary: This KB walks you through VM protection upload failure for Virtual Machines added as a container client in Avamar.
Symptoms
When Virtual Machines are added to Avamar as a container client, the upload for the VMs may fail with the following error:
ERROR [2019-05-05 11:01:43,002] [ThreadPoolTaskExecutor-18] [JobExecutionUtil:logJobErrorsWithStackTrace:29]: Error in job ProtectionFlow: com.emc.cloud_dr.cdr.commons.data_mover.api.DataMoverBaseException: Failed to upload segment 1401 ! com.emc.cloud_dr.cdr.commons.common_models.copies_storage.CopiesStorageException: Copies Storage returned unexpected error code (137) ! at com.emc.cloud_dr.cdr.cdra.dd_adapter.DdAdapter.callDdAdapterExecutable(DdAdapter.java:90) ! at com.emc.cloud_dr.cdr.cdra.dd_adapter.DdAdapter.getDiffs(DdAdapter.java:193) ! at com.emc.cloud_dr.cdr.cdra.dd_adapter.DdAdapter.getDiffs(DdAdapter.java:171) ! at com.emc.cloud_dr.cdr.commons.common_logic.copies_storage.impl.data_domain.CopiesStorageImplDataDomain.getDiffs(CopiesStorageImplDataDomain.java:110) ! at com.emc.cloud_dr.cdr.commons.common_logic.dd_adapter.DiffFileTask.call(DiffFileTask.java:18) ! at com.emc.cloud_dr.cdr.commons.common_logic.dd_adapter.DiffFileTask.call(DiffFileTask.java:8) ! ... 4 common frames omitted ! Causing: java.io.IOException: Expected to read: 4194304 but read 63712 ! at com.emc.cloud_dr.cdr.commons.data_mover.impl.UploadUtils.readInputStreamToBuffer(UploadUtils.java:25) ! ... 7 common frames omitted ! Causing: com.emc.cloud_dr.cdr.commons.data_mover.api.DataMoverBaseException: Error reading from pipe ! at com.emc.cloud_dr.cdr.commons.data_mover.impl.UploadUtils.readInputStreamToBuffer(UploadUtils.java:32) ! at com.emc.cloud_dr.cdr.commons.data_mover.impl.PipedFileSegmentedDiffUploadTask.readToBuffer(PipedFileSegmentedDiffUploadTask.java:122) ! at com.emc.cloud_dr.cdr.commons.data_mover.impl.PipedFileSegmentedDiffUploadTask.call(PipedFileSegmentedDiffUploadTask.java:70) ! ... 5 common frames omitted
Cause
Cloud Disaster Recovery (CDR) does not support Virtual Machines added as part of a Container to the Avamar Server on CDR version 18.3 or earlier. This is supported in the upcoming CDR release.
When Virtual Machines on the Avamar Server are added for Protection to CDR, they must be added or imported individually instead of as the entire container.
Resolution
In order to address this issue, retire the virtual machines added as a container and readd them individually to Avamar.
From the Administration tab on the Avamar UI, 'Retire' the container which was imported to the Avamar and contains the Virtual Machines.
- Go to Navigation > Administration.
- From the left pane, expand the vCenter domain in question and select the Virtual Machines sub-domain.
- Identify the VM Container that was added to Avamar.
- Right click the container and select "Retire"
Note: The option for backup retention is offered when 'Retire' is selected. Selections can be made on the appropriate option to retain the existing backups for these Virtual Machines as per their current retention Policy.
- Right click the 'Virtual Machines' sub domain and select 'New Client'. Readd the Virtual machines by selecting them individually instead of selecting the entire container.
- Add these VMs to the Backup Policy which has "Enable Cloud" option selected.
Retry a backup and monitor the backup upload on CDRS UI to verify if it works as expected.