I wondered if anyone may have seen this problem before we log a case on it...
We have an old Networker 7.6.5 server that backs up all VMs fine. The VMs we are having an issue with only backup once a week here. vCenter is 5.1 and VMs we have seen this with have a mix of current and older VMtools.
We have a new Networker 8.2.1 server which we have began moving VM backups to (using VADP to tape, not VBA) and have so far moved about 20 VMs which backup successfully via proxy server on SAN; however 4 of the VMs have had issues during testing in that the they do not release the snapshot properly so require a manual consolidation afterwards; this problem is tempermental in that it happens maybe 50% of the time. We have also less frequently seen times when the snapshot is taken but then no data is moved at all until the backup fails when the timeout is reached, however if it retries with a new snapshot that writes data successfully but then when trying to delete/consolidate has the same problem again.
We have tried setting timeout to 0 and eventually had to kill off a group that failed to move any data; it shows snapshot being taken but no attempt to delete it. Manually deleting this in vCenter gives the same error which suggests this part of the problem is independent of Networker so we assume the problem exists during Networker taking the snapshot.
We are currently gathering debug logging and will open a ticket but curious if anyone has seen anything similar before.
After extensive testing we have came to the conclusion that it is a compatibility issue between VDDK 5.5 and vCenter 5.1. This means that any version of Networker that includes VDDK 5.5 (Networker 8.1 SP1 upwards) will not work with vCenter 5.1 at least until a fix is found.
EMC Support give a fix of upgrading vCenter to 5.5 which is currently not feasible for us. This case remains open with them to produce a fix iner the current version.
We have downgraded to 220.127.116.11 as this is the last version that packages VDDK 5.0 with it (VDDK 5.0 U2), at that point everything works fine again (we have turned a blind eye to the Software Compatibility Guide saying minimum supported version with vCenter is 5.1
Unfortunately we have found that on this older VDDK we cannot restore cleanly from the GUI, however we have built an additional 8.2.1 proxy for restores only and this allows flawless restores of VMs. We will maintain this until the end of the year when we can upgrade the vCenter.
We have been carrying out further testing and it appears that the ESX server in question will only run two backups simultaneously (we are currently testing to see if this is specific to this ESX server or all are affected).