Vss error win2012 r2 - VSS OTHER

Hi,

we got a few Win2012 r2 machine to be backed up, same build VM template.

We do a FS backup all files, but also tried single file.

First machine backs up just fine, remaining 3 machines get this error:

VSS OTHER: ERROR: VSS failed to process snapshot: 90108:save: Unable to save the SYSTEM STATE save sets: cannot create the snapshot.

Now i looked at vssadmin list writers and all writers including system writer are state[1] and look fine.

Event/App logs show no VSS errors besides a notification - vss timeout 3min after Networker Error (as expected).

Applogs show this Networker error:

ID 1 Could not perform a VSS save operation. unable to save the SYSTEM STATE save sets:cannot create the snapshot.

It looks like Networker doesnt call the system writer for some reason for this save set?

Now i ran DBG9 from NMC and get this (i extracted the VSS part):

07/26/18 03:59:04.274782 client name = a0j3us042xvm005, client res name = A0J3US042XVM005 07/26/18 03:59:04.277777 WriteTracker: nsrwriter_start(3460:3364) -->. 07/26/18 03:59:04.277777 WriteTracker: nsrwriter_start(3460:3364) CreateMutexW successful err 0, starting to wait.... 07/26/18 03:59:04.277777 Traditional save workflow acquired nsrwriter lock. 07/26/18 03:59:04.277777 WriteTracker: nsrwriter_start(3460:3364) wait successful on hWriterMutex. 07/26/18 03:59:04.278775 WriteTracker: nsrwriter_start(3460:3364) <-- TRUE. 07/26/18 03:59:04.278775 sow_init_save_sysset() running in save.exe or savepnpc.exe 07/26/18 03:59:04.280779 VSS .. COM library initialized.07/26/18 03:59:04.281779 VSS .. CoInitializeSecurity succeeded .. 07/26/18 03:59:04.282779 PopulateSupportedWriterList() Enter./n07/26/18 03:59:04.282779 lgwstat: Failed to open file/directory: \\?\C:\Program Files\EMC NetWorker\nsr\bin\writerlist.txt07/26/18 03:59:04.282779 PopulateDefaultSupportedWriterList() Enter. 07/26/18 03:59:04.282779 PopulateDefaultSupportedWriterList() Exit. 07/26/18 03:59:04.282779 PopulateSupportedWriterList() Exit. 07/26/18 03:59:04.282779 CVssSysReqBackup::Init(): Successfully updated writer list. 07/26/18 03:59:04.282779 VSS .. Setting context to VSS_CTX_BACKUP .. 07/26/18 03:59:05.229834 IsWriterSupported() Enter. 07/26/18 03:59:05.229834 VSS Writer {e8132975-6f93-4464-a53e-1050253ae220} is supported 07/26/18 03:59:05.229834 IsWriterSupported() Exit. 07/26/18 03:59:05.229834 Adding ��   volume in the volume snapshot map. 07/26/18 03:59:05.229834 Adding ��   volume in the volume snapshot map. 07/26/18 03:59:05.230833 IsWriterSupported() Enter. 07/26/18 03:59:05.230833 VSS Writer {d61d61c8-d73a-4eee-8cdd-f6f9786b7124} is supported 07/26/18 03:59:05.230833 IsWriterSupported() Exit. 07/26/18 03:59:05.230833 IsWriterSupported() Enter. 07/26/18 03:59:05.230833 VSS Writer {75dfb225-e2e4-4d39-9ac9-ffaff65ddf06} is supported 07/26/18 03:59:05.230833 IsWriterSupported() Exit. 07/26/18 03:59:05.230833 IsWriterSupported() Enter. 07/26/18 03:59:05.230833 VSS Writer {0bada1de-01a9-4625-8278-69e735f39dd2} is supported 07/26/18 03:59:05.230833 IsWriterSupported() Exit. 07/26/18 03:59:05.230833 IsWriterSupported() Enter. 07/26/18 03:59:05.230833 VSS Writer {e8132975-6f93-4464-a53e-1050253ae220} is supported 07/26/18 03:59:05.230833 IsWriterSupported() Exit. 07/26/18 03:59:05.230833 IsWriterSupported() Enter. 07/26/18 03:59:05.230833 VSS Writer {be000cbe-11fe-4426-9c58-531aa6355fc4} is not supported 07/26/18 03:59:05.230833 IsWriterSupported() Exit. 07/26/18 03:59:05.230833 IsWriterSupported() Enter. 07/26/18 03:59:05.230833 VSS Writer {afbab4a2-367d-4d15-a586-71dbb18f8485} is supported 07/26/18 03:59:05.230833 IsWriterSupported() Exit. 07/26/18 03:59:05.230833 IsWriterSupported() Enter. 07/26/18 03:59:05.230833 VSS Writer {542da469-d3e1-473c-9f4f-7847f01fc64f} is supported 07/26/18 03:59:05.230833 IsWriterSupported() Exit. 07/26/18 03:59:05.230833 IsWriterSupported() Enter. 07/26/18 03:59:05.230833 VSS Writer {a6ad56c2-b509-4e6c-bb19-49d8f43532f0} is supported 07/26/18 03:59:05.230833 IsWriterSupported() Exit. 07/26/18 03:59:05.230833 IsWriterSupported() Enter. 07/26/18 03:59:05.230833 VSS Writer {4969d978-be47-48b0-b100-f328f07ac1e0} is supported 07/26/18 03:59:05.230833 IsWriterSupported() Exit. 07/26/18 03:59:05.230833 IsWriterSupported() Enter. 07/26/18 03:59:05.230833 VSS Writer {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f} is supported 07/26/18 03:59:05.230833 IsWriterSupported() Exit. 07/26/18 03:59:09.694089 Found @F�  volume in the volume snapshot map. 07/26/18 03:59:09.694089 add_associated_critical_volumes: print out of critical volume list after call to add_associated_critical_volumes. 07/26/18 03:59:09.694089 critical volume = . 07/26/18 03:59:09.695089 add_associated_critical_volumes: print out of critical volume list after call to add_associated_mount_points. 07/26/18 03:59:09.695089 critical volume = . 07/26/18 03:59:09.695089 add_associated_critical_volumes: print out of critical volume list after call to add_associated_dynamic_disk_volumes. 07/26/18 03:59:09.695089 critical volume = . 07/26/18 03:59:09.695089 add_associated_critical_volumes: print out of critical volume list after call to add_associated_mount_points. 07/26/18 03:59:09.695089 critical volume = . 07/26/18 03:59:09.696090 Adding @D�  volume in the volume snapshot map. 07/26/18 03:59:09.722091 WINDOWS ROLES AND FEATURES: sow_init_save_sysset_walker() called 07/26/18 03:59:09.723091 No components found for writer=BITS Writer07/26/18 03:59:09.778094 VSS USER DATA: sow_init_save_sysset_walker() called 07/26/18 03:59:09.778094 IsWriterSupported() Enter. 07/26/18 03:59:09.778094 VSS Writer {542da469-d3e1-473c-9f4f-7847f01fc64f} is supported 07/26/18 03:59:09.778094 IsWriterSupported() Exit. 07/26/18 03:59:09.778094 IsWriterSupported() Enter. 07/26/18 03:59:09.778094 VSS Writer {afbab4a2-367d-4d15-a586-71dbb18f8485} is supported 07/26/18 03:59:09.778094 IsWriterSupported() Exit. 07/26/18 03:59:09.778094 IsWriterSupported() Enter. 07/26/18 03:59:09.778094 VSS Writer {be000cbe-11fe-4426-9c58-531aa6355fc4} is not supported 07/26/18 03:59:09.778094 IsWriterSupported() Exit. 07/26/18 03:59:09.778094 IsWriterSupported() Enter. 07/26/18 03:59:09.778094 VSS Writer {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f} is supported 07/26/18 03:59:09.778094 IsWriterSupported() Exit. 07/26/18 03:59:09.778094 IsWriterSupported() Enter. 07/26/18 03:59:09.778094 VSS Writer {4969d978-be47-48b0-b100-f328f07ac1e0} is supported 07/26/18 03:59:09.778094 IsWriterSupported() Exit. 07/26/18 03:59:09.778094 IsWriterSupported() Enter. 07/26/18 03:59:09.778094 VSS Writer {a6ad56c2-b509-4e6c-bb19-49d8f43532f0} is supported 07/26/18 03:59:09.778094 IsWriterSupported() Exit. 07/26/18 03:59:09.778094 IsWriterSupported() Enter. 07/26/18 03:59:09.778094 VSS Writer {e8132975-6f93-4464-a53e-1050253ae220} is supported 07/26/18 03:59:09.778094 IsWriterSupported() Exit. 07/26/18 03:59:09.778094 IsWriterSupported() Enter. 07/26/18 03:59:09.778094 VSS Writer {0bada1de-01a9-4625-8278-69e735f39dd2} is supported 07/26/18 03:59:09.778094 IsWriterSupported() Exit. 07/26/18 03:59:09.778094 IsWriterSupported() Enter. 07/26/18 03:59:09.779094 VSS Writer {75dfb225-e2e4-4d39-9ac9-ffaff65ddf06} is supported 07/26/18 03:59:09.779094 IsWriterSupported() Exit. 07/26/18 03:59:09.779094 IsWriterSupported() Enter. 07/26/18 03:59:09.779094 VSS Writer {d61d61c8-d73a-4eee-8cdd-f6f9786b7124} is supported 07/26/18 03:59:09.779094 IsWriterSupported() Exit. 07/26/18 03:59:09.779094 No components found for writer=Shadow Copy Optimization Writer07/26/18 03:59:09.779094 VSS OTHER: sow_init_save_sysset_walker() called 07/26/18 03:59:09.779094 IsWriterSupported() Enter. 07/26/18 03:59:09.779094 VSS Writer {542da469-d3e1-473c-9f4f-7847f01fc64f} is supported 07/26/18 03:59:09.779094 IsWriterSupported() Exit. 07/26/18 03:59:09.779094 IsWriterSupported() Enter. 07/26/18 03:59:09.779094 VSS Writer {afbab4a2-367d-4d15-a586-71dbb18f8485} is supported 07/26/18 03:59:09.779094 IsWriterSupported() Exit. 07/26/18 03:59:09.779094 IsWriterSupported() Enter. 07/26/18 03:59:09.779094 VSS Writer {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f} is supported 07/26/18 03:59:09.779094 IsWriterSupported() Exit. 07/26/18 03:59:09.779094 IsWriterSupported() Enter. 07/26/18 03:59:09.779094 VSS Writer {4969d978-be47-48b0-b100-f328f07ac1e0} is supported 07/26/18 03:59:09.779094 IsWriterSupported() Exit. 07/26/18 03:59:09.779094 IsWriterSupported() Enter. 07/26/18 03:59:09.779094 VSS Writer {a6ad56c2-b509-4e6c-bb19-49d8f43532f0} is supported 07/26/18 03:59:09.779094 IsWriterSupported() Exit. 07/26/18 03:59:09.779094 IsWriterSupported() Enter. 07/26/18 03:59:09.779094 VSS Writer {e8132975-6f93-4464-a53e-1050253ae220} is supported 07/26/18 03:59:09.779094 IsWriterSupported() Exit. 07/26/18 03:59:09.779094 IsWriterSupported() Enter. 07/26/18 03:59:09.779094 VSS Writer {0bada1de-01a9-4625-8278-69e735f39dd2} is supported 07/26/18 03:59:09.779094 IsWriterSupported() Exit. 07/26/18 03:59:09.779094 IsWriterSupported() Enter. 07/26/18 03:59:09.779094 VSS Writer {75dfb225-e2e4-4d39-9ac9-ffaff65ddf06} is supported 07/26/18 03:59:09.779094 IsWriterSupported() Exit. 07/26/18 03:59:09.780095 IsWriterSupported() Enter. 07/26/18 03:59:09.780095 VSS Writer {d61d61c8-d73a-4eee-8cdd-f6f9786b7124} is supported 07/26/18 03:59:09.780095 IsWriterSupported() Exit. 07/26/18 03:59:09.780095 VSS OTHER: No writers detected for this VSS saveset 07/26/18 03:59:09.780095 fullcanon(name=\\?\Volume{051a7d74-88d7-11e3-80b3-806e6f6e6963}\) 07/26/18 03:59:09.781094    exit fullcanon(fullname=\\?\Volume{051a7d74-88d7-11e3-80b3-806e6f6e6963}\) 07/26/18 03:59:09.781094 lgwstat: Failed to open file/directory: \\?\C:\Program Files\EMC NetWorker\nsr\tmp\dc8ca644-6be2-46f6-801a-cacfa2b2174507/26/18 03:59:09.781094 Found  ��  volume in the volume snapshot map. 07/26/18 03:59:09.781094 fullcanon(name=\\?\Volume{051a7d74-88d7-11e3-80b3-806e6f6e6963}\) 07/26/18 03:59:09.781094    exit fullcanon(fullname=\\?\Volume{051a7d74-88d7-11e3-80b3-806e6f6e6963}\) VSS OTHER: ERROR: VSS failed to process snapshot: 07/26/18 03:59:09.842097 VSS OTHER: sow_create_sysset_snapshot: VSS failed to process snapshot, error=0x0 90108:save: Unable to save the SYSTEM STATE save sets: cannot create the snapshot. 07/26/18 03:59:09.843096 WINDOWS ROLES AND FEATURES: sow_free_save called 07/26/18 03:59:09.843096 VSS USER DATA: sow_free_save called 07/26/18 03:59:09.843096 VSS OTHER: sow_free_save called 07/26/18 03:59:09.843096 WriteTracker: nsrwriter_stop(3460:3364) -->. 07/26/18 03:59:09.843096 WriteTracker: nsrwriter_stop(3460:3364) error: NULL hProcess detected.. 07/26/18 03:59:09.843096 WriteTracker: nsrwriter_stop(3460:3364) releasing hWriterMutex .... 07/26/18 03:59:09.843096 save workflow released nsrwriter lock. 07/26/18 03:59:09.843096 WriteTracker: nsrwriter_stop(3460:3364) <-- FALSE. 07/26/18 03:59:09.843096 lgwstat: Failed to open file/directory: \\?\C:\Program Files\EMC NetWorker\nsr\tmp\dc8ca644-6be2-46f6-801a-cacfa2b2174586024:save: Error occured while saving disaster recovery save sets.

Please anyone got any idea? Thanks!