could you give some more details avamar version etc. ? and a little more from the log

I'm running DDOS 5.4.0.7 and Avamar 7.0.2-47.

There really wasn't much to the logs. That was the problem. However, this occurred on image backups, Exchange backups and replication.

You can verify if this is set. SSH into your DD and from the command line run:

ddboost option show

The results look something like this.

Option                           Value

------------------------------   --------

distributed-segment-processing   enabled

virtual-synthetics               enabled

fc                               disabled

------------------------------   --------

This is from Exchange

2015-01-19 22:28:59 avtar Error <0000>: flushPendingExtents: encountered an error writing existing file data via synthesis now (Log #1)

2015-01-19 22:28:59 avtar Error <0000>: Function not supported failed when writing to a file in a container for backup, length 76 bytes at offset 22632175740, remaining bytes 76 (Log #1)

2015-01-19 22:28:59 avtar FATAL <5352>: MAIN: Unhandled standard exception failed when writing to a file in a container for backup, length 76 bytes at offset 22632175740, remaining bytes 76 (Log #1)

2015-01-19 22:29:00 avexvss Error <7759>: Unable to successfully process snapview workorder (Log #4)

2015-01-19 22:29:10 avexvss Error <13077>: Unable to complete backup. (Log #4)

This is from an Image Backup

2015-01-20 00:02:57 avtar FATAL <0000>: DDR encountered errors. (Log #2)

2015-01-20 00:02:57 avtar Error <0000>: flushPendingExtents: encountered an error writing existing file data via synthesis now (Log #2)

2015-01-20 00:02:57 avtar FATAL <0000>: Failed begin data slices to container, Function not supported failed when beginning a file in a container for backup, DDR result code: 5045, desc: Function not supported (Log #2)

2015-01-20 00:02:57 avtar FATAL <10621>: Failed to overwrite data to stream, due to begin data slice failure at offset 1179648 for stream index 13, DDR result code: 0 desc: Error not set (Log #2)

2015-01-20 00:03:00 avvcbimage Error <0000>: [IMG0010] PAX stream append ([XXXXXXXX-Storage2] XXXXXXXX/XXXXXXXX.vmdk) at sector offset 55168 length 128 sectors=65536 failed (32) (Log #1)

2015-01-20 00:03:00 avtar Error <0000>: flushPendingExtents: encountered an error writing existing file data via synthesis now (Log #2)

2015-01-20 00:03:00 avtar FATAL <0000>: Failed begin data slices to container, Function not supported failed when beginning a file in a container for backup, DDR result code: 5045, desc: Function not supported (Log #2)

2015-01-20 00:03:00 avtar FATAL <10622>: Terminating avtar because it failed to overwrite data to stream, due to begin data slice failure at offset 1245184 for stream index 13, DDR result code: 0 desc: Error not set, [stage:RSTAGE_WRAPUP/1 excode:EXIT_KILLED/166 skipwrap:f] (Log #2)

2015-01-20 00:03:00 avvcbimage Error <0000>: [IMG0010] pax_container::endfile(virtdisk-flat.vmdk,28246016) returned problem:32 (Log #1)

2015-01-20 00:03:00 avvcbimage Error <0000>: [IMG0010] pax_container::enddir returned problem:32 (Log #1)

2015-01-20 00:03:00 avvcbimage Error <0000>: [IMG1003] Backup of [XXXXXXXXX-Storage2] XXXXXXXX/XXXXXXXX.vmdk failed (Log #1)

2015-01-20 00:03:20 avvcbimage Error <0000>: [IMG0011] Avtar exited with 'code 166: session cancelled by Server' (Log #1)

2015/01/20-00:03:14.41660 VcbImageBackupAssistThread DEBUG-INTERNAL ERROR <0001> assert error (uapp::staging().isExitCancel() || list == __null || skipped_ntfs_chunks), /local/jenkins/workspace/client_Hermosa_SLES11SP3-64/abs2/work/src/avclient/filepipe.cpp line 98 (Log #1)

This is from a replication job

2015-01-19 22:43:01 avtar Error <0000>: DDRIO_Write::SynthesizeFile: ddp_synthesize_file failed (Log #2)

2015-01-19 22:43:02 avtar Error <18786>: Replication - failed to join slice /STAGING/40fcf057cabb0b5bd5248906e4019330cf5e9463/BACKUP-89C4E876F79D3D5383AE2FBD8004495ADCAEFD1F-1D03381BAB1CEA2/7C0A3A89AD190F41D8ADBEF18FED281607144955.00 (#0) to offset 0 for 2,147,483,648 bytes, id: 8 (Log #2)

2015-01-19 22:43:03 avtar Error <18797>: Replication failed - The replicator returned an error. (Log #2)

2015-01-19 22:43:03 avtar Error <18773>: MReplication failed -- backup: 1D03381BAB1CEA2, from xxxxxxxxxxx.xxxxxx.com(1):avamar-1390414106 to xxxxxxxxxxx.xxxxxx.com(1):avamar-1390426309, DDR result code: 4912, desc: No error description available. (Log #2)

2015-01-19 22:43:06 avtar FATAL <12527>: Replication failed (Log #2)

2015-01-19 22:43:06 avtar FATAL <0000>: DDR encountered errors. (Log #2)

2015-01-19 22:43:06 avtar Error <6655>: Replicate interrupted. (Log #2)

2015/01/20-04:43:06.62103 BuilderMgr0 DEBUG-INTERNAL ERROR <0001> assert error (result == MSG_ERR_NONE), /local/jenkins/workspace/client_Hermosa_SLES11-64/abs2/work/src/avclient/ade/adelib/ddrbuilder.cpp line 333 (Log #2)

Did you get a chance to discuss the issue with Avamar support ? This looks like a bug, but it's better that support gives the final word.

ah! Nice.

It's not a bug. It was human error. Support found the problem, but ironically support also caused it.

While support was working on a bug in version 5.4.0.7 regarding the the filesys cleanup job, it was necessary to disable virtual synthetics. When the filesys was restarted, virtual synthetics was never re-enabled.

The reason I posted this was because I had the incident open for the better part of two days before they got it corrected.

I have the same issue however I checked my Datadomain DDBOOST option show and the virtual-sythentics as well as distributed-segment-processing are enabled and fc was disabled, but i am still getting these issues on some replication jobs, using policy replication on Avamar version 7.0.1-61_HF190109_1. Has anyone else seen this problem or know what else may be going on? Thank You.