Avamar: VM Image level Backup fails with 10026 Error to Data Domain
Summary: Backing up a VMware Image fails with 10026 or 10055, "Invalid length failed when overwriting to a file in a container for backup"
Symptoms
At the end of a VM backup with Data Domain it fails with FCOW error related to change block offset:
2016-11-10 19:00:39 avtar FATAL <0000>: Failed to send data to file container for FCOW, Invalid offset failed overwriting to a file in a container for backup, changed block begin offset 133272572928 is not a multiple of the fixed segment size 8192, DDR result code: 4914, desc: Invalid offset
2016-11-10 19:00:39 avtar FATAL <0000>: <10621>Failed to overwrite data to stream, due to send data failure for size 262144 for FCOW for stream index 15, DDR result code: 0 desc: Error not set
2016-11-10 19:00:39 avtar FATAL <40009>: DDR encountered errors.
2016-11-10 19:00:39 avtar Info <9772>: Starting graceful (staged) termination, DDR_ERROR event received (fatal severity) (wrap-up stage)
2016-11-10 19:00:39 avtar Info <0000>: Entering the 'final' phase of termination, DDR_ERROR need to exit)
2016-11-10 19:00:39 avtar Info <15092>: Container final statistics:
container file name: C884337993FD57BF9CB6B44AB276E861ABA0F5F8
total seg count: 22,599,398
redundant seg count: 22,560,065
2016-11-10 19:00:39 avtar FATAL <5155>: Backup aborted due to earlier errors. No backup created on the server.
Or
2022-07-13 10:39:52 avtar FATAL <0000>: Failed to close file container for backup for FCOW, Invalid length failed when overwriting to a file in a container for backup, length 1677312 is not a multiple of the fixed segment size 8192, DDR result code: 4915, desc: Invalid length
2022-07-13 10:39:52 avtar FATAL <0000>: Failed closing write stream for FCOW, index: 33, close container failed, exit code: 157.
Cause
Resolution
Configure the Virtual Machine and edit the settings.
Check each disk size for a decimal size.
Adjust the size of the disk by ROUNDING UP to the next whole number.
Example:
The disk appears as size 127999.3337 MB.
Change this to 128000 MB.
Also the goav vm validate-size command can be used to check for this issue.
For more information about validate-size feature, see the Additional Info section of this article or GoAV documentation.
-
Using GoAV, run the command:
./goav vm validate-size
-
To resize the virtual machine:
-
In the vSphere client, locate the virtual machine.
-
Edit the virtual machine and select the identified disk using the tool.
-
Change the units to MB, and from the GoAV tool set the size to RESIZE TO.
-
-
Start a new backup in Avamar. After the backup is complete, perform a new image restore on the latest backup to confirm the issue is resolved.
See VMware KB: https://knowledge.broadcom.com/external/article?articleNumber=377962
Additional Information
More information about GoAV validate-size feature:
This feature detects virtual machines with virtual disk sizes that are not multiples of 8192 KB or 1,048,576 KB (1 MB). If found, it recommends a newer, larger size to increase the virtual disk.
./goav vm validate-size --help
Validate Virtual Machine disk size by checking each disk for the VM on vCenter.
Verifies that disk sizes are properly divisible by 1 MB and 8 KB.
Usage:
goav vm validate-size [flags]
Examples:
./goav vm validate-size --name testing8kb-invalidlength --vc myvcenter.fqdn.tld
DEVICE KEY | VMDK | DISK SIZE | 8 KB | 1 MB | RESIZE TO
-------------+---------------------------------------------------------------------+-----------+-------+-------+------------
2001 | [localDatastore] testing8kb-invalidlength/invalidSize.vmdk | 1536000 | false | false | 2 MB
Flags:
-h, --help help for validate-size
--hide-valid Set output to hide valid VMs
--name string VM Name
--prompt Prompts to select vCenters
--vc string Vcenter Name
Global Flags:
-d, --debug debug output
-f, --force Ignore Configuration
-n, --noheader Do not display header