We have an 8 node Hyper-V cluster on Windows 2008 R2 with a Clariion CX3-10 as storage for Cluster Shared Volumes and Microsoft DPM 2010 as the front-end requestor (don't laugh). Due to stability issues with this conbination since day one, we've ended up trying to mitigate for any possibility of the system having to do anything more difficult than simply create the snapshots. Each Hyper-V host has its own cluster shared volume and the virtual machines are hosted and aligned so that all machines hosted by a particular Hyper-V host is also physically stored on the CSV 'owned' by that host, to avoid ownership changes during snapshots. Each Hyper-V host is in its own storage group and has sight of all 8 CSVs, to allow migrations and fault tolerance and all the wonderful things that VMs are supposed to bring.

Our issue is that we are unable to get consistently robust backups via DPM, and the main problem appears to be that , for no obvious reason, the EMC Hardware VSS Provider decides that there are insufficient resources for the snapshot to be created. Some nights we can run snapshots on all 8 CSVs concurrently (we have had variously 8, 16 and 24 LUNs of a good size in the reserved LUN pool) and sometimes we even get what seems to be a 'not enough resources' issue when we are simply trying to re-snapshot one VM on one host.
The hwprov.log snippet below is from the latest failed attempt this morning. This is the only snapshot we are trying to create and there are 8 free LUNs in the RLP. I will consider donating internal organs or a case of something alcoholic to the person who can figure out why this fails.

Thank you.

Richard Davies

11:01:04 04/07/2014 CreateFriendlyName = TRUE.  Provider will create a friendly name for CLARIION snapview session and lun names.

11:01:04 04/07/2014 EnforceDefaultToClone = FALSE.  Provider will create Mirror(default) snapshots for plex.

11:01:04 04/07/2014 EnforceVPSnap not found.  Provider will create Timefinder/Snap differential snapshots.

11:01:04 04/07/2014 EnforceSymmMaskingFromList key not found.  Provider will use the standard rules for Tigon unmasking operations.

11:01:04 04/07/2014 GetRegistryDefaults: Leaving with SUCCESS.

11:01:04 04/07/2014 OnLoad: Entering.

11:01:04 04/07/2014 OnLoad: EMC VSS Provider (V4.7.1.0-9) is loading.

11:01:04 04/07/2014 OnLoad: Initializing a READ_WRITE local storapi session.

11:01:04 04/07/2014 OnLoad: Initializing a READ_MOD_NOWRITE local storapi session.

11:01:04 04/07/2014 StorUpdate: Entering.

11:01:04 04/07/2014 StorUpdate: Performing a StorDiscover to update the database.

11:01:18 04/07/2014 StorUpdate: Commiting database in memory to file.

11:01:18 04/07/2014 StorUpdate: Leaving with SUCCESS.

11:01:18 04/07/2014 IsLocalHostVM: Entering...

11:01:18 04/07/2014 IsLocalHostVM: Leaving with success.

11:01:18 04/07/2014 OnLoad: EMC VSS Provider (V4.7.1.0-9) has successfully loaded.

11:01:18 04/07/2014 OnLoad: Leaving with SUCCESS.

11:01:18 04/07/2014 AreLunsSupported: Entering.

11:01:18 04/07/2014 AreLunsSupported: lLunCount = 1, lContext = 0x0.

11:01:18 04/07/2014 AreLunsSupported: Iterating through 1 LUNs.

11:01:18 04/07/2014 VDS_LUN_INFORMATION for \\.\PHYSICALDRIVE4:

m_version = 1

m_DeviceType = 0

m_DeviceTypeModifier = 101

m_bCommandQueueing = 1

m_BusType = 6

m_szVendorId = DGC

m_szProductId = RAID 5

m_szProductRevision = 0326

m_szSerialNumber = CK200081900XXX

m_diskSignature = 09b021d7-0000-0000-0000-000000000000

There are 2 device descriptors:

[0] m_CodeSet = 1

m_Type = 3

m_cbIdentifier = 16

m_rgbIdentifier = 6006016069022100fa878db74b64e111

[1] m_CodeSet = 1

m_Type = 0

m_cbIdentifier = 16

m_rgbIdentifier = 0003000000000000001f000000000000

11:01:18 04/07/2014 AreLunsSupported: Calling IsLocalDevSupportable for \\.\PHYSICALDRIVE4

11:01:18 04/07/2014 IsLocalDevSupportable: Entering.

11:01:18 04/07/2014 IsLocalDevSupportable: Device = \\.\PHYSICALDRIVE4, lContext = 0x0.

11:01:18 04/07/2014 iPdevShow: Entering with \\.\PHYSICALDRIVE4.

11:01:18 04/07/2014 IsLocalHostVM: Entering...

11:01:18 04/07/2014 IsLocalHostVM: Leaving with success.

11:01:18 04/07/2014 IsLocalHostHyperVVM: Entering...

11:01:18 04/07/2014 IsLocalHostHyperVVM: Leaving with success.

11:01:18 04/07/2014 iPdevShow: Calling StorPdevShow for \\.\PHYSICALDRIVE4.

11:01:18 04/07/2014 iPdevShow: StorPdevShow successfully completed.

11:01:18 04/07/2014 iPdevShow: Leaving with SUCCESS.

11:01:18 04/07/2014 IsLocalDevSupportable: IsSupported: TRUE. Leaving with SUCCESS.

11:01:18 04/07/2014 AreLunsSupported: Device [\\.\PHYSICALDRIVE4] is supported.

11:01:18 04/07/2014 AreLunsSupported: Leaving with SUCCESS.

11:01:18 04/07/2014 BeginPrepareSnapshot: Entering.

11:01:18 04/07/2014 BeginPrepareSnapshot: SnapshotSet ID = {3FC5E911-182D-4DD9-BCBC-36ACAF8D54E9}, Snapshot ID = {7B8A993D-61EB-462F-8E9D-F0822112B9B8}, Context = 4194304, LunCount = 1.

11:01:18 04/07/2014 BeginPrepareSnapshot: Creating a new SnapshotSet.

11:01:18 04/07/2014 CreateSnapshotSet: Entering.

11:01:18 04/07/2014 CSnapshotSet: Entering constructor

11:01:18 04/07/2014 CSnapshotSet: Leaving constructor with success.

11:01:18 04/07/2014 CreateSnapshotSet: Creating SnapshotSet: 3fc5e911-182d-4dd9-bcbc-36acaf8d54e9.

11:01:18 04/07/2014 CreateSnapshotSet: Leaving with SUCCESS.

11:01:18 04/07/2014 BeginPrepareSnapshot: Calling DoSnapshotResourcesExist for \\.\PHYSICALDRIVE4

11:01:18 04/07/2014 DoSnapshotResourcesExist: Entering.

11:01:18 04/07/2014 DoSnapshotResourcesExist: lContext = 0x400000.

11:01:18 04/07/2014 iPdevShow: Entering with \\.\PHYSICALDRIVE4.

11:01:18 04/07/2014 IsLocalHostVM: Entering...

11:01:18 04/07/2014 IsLocalHostVM: Leaving with success.

11:01:18 04/07/2014 IsLocalHostHyperVVM: Entering...

11:01:18 04/07/2014 IsLocalHostHyperVVM: Leaving with success.

11:01:18 04/07/2014 iPdevShow: Calling StorPdevShow for \\.\PHYSICALDRIVE4.

11:01:18 04/07/2014 iPdevShow: StorPdevShow successfully completed.

11:01:18 04/07/2014 iPdevShow: Leaving with SUCCESS.

11:01:18 04/07/2014 IsVPSnapSupported: Entering.

11:01:18 04/07/2014 IsVPSnapSupported: Calling StorRepGroupRelCapGet to know the array capabilities.

11:01:18 04/07/2014 IsVPSnapSupported: VP Snaps are not supported for this array.

11:01:18 04/07/2014 IsVPSnapSupported: Leaving.

11:01:18 04/07/2014 DoSnapshotResourcesExist: VP Snaps are not supported on [ ] array.

11:01:18 04/07/2014 DoSnapshotResourcesExist: No context specified, will search for replica relationships and set the context accordingly

11:01:18 04/07/2014 DoSnapshotResourcesExist: Calling StorArraySync(-1) for .

11:01:18 04/07/2014 iArraySync: Entering.

11:01:18 04/07/2014 iArraySync: StorArraySync(-1) for .

11:01:30 04/07/2014 iArraySync: Leaving with SUCCESS.

11:01:30 04/07/2014 DoSnapshotResourcesExist: Replication is Plex and DefaultToClone is [0].device array is [3](2=symm, 3= Clar)

11:01:30 04/07/2014 DoSnapshotResourcesExist: Calling FindNextMirrorReplica...

11:01:30 04/07/2014 FindNextMirrorReplica: Entering.

11:01:30 04/07/2014 IsSnapshotRemote: Entering.

11:01:30 04/07/2014  IsSnapshotRemote: Could not retrieve value, Defaults will be used

11:01:30 04/07/2014 IsSnapshotRemote: Leaving with SUCCESS.

11:01:31 04/07/2014 FindNextMirrorReplica: No local mirror replicas were found for />, so now looking for remote mirror replicas.

11:01:31 04/07/2014 FindNextMirrorReplica: No remote mirror replicas were found for />.

11:01:31 04/07/2014 FindNextMirrorReplica: Leaving with FAILURE.

11:01:31 04/07/2014 DoSnapshotResourcesExist: ERROR = Device is not STOR_DEV_REP_IS_SNAPABLE.

11:01:31 04/07/2014 IsArrayVNXCapable: Array is not capable of VNX Snaps!!

11:01:31 04/07/2014 DoSnapshotResourcesExist: Leaving with SUCCESS.

11:01:31 04/07/2014 BeginPrepareSnapshot: Resources do not exist to support device = \\.\PHYSICALDRIVE4.

11:01:31 04/07/2014 BeginPrepareSnapshot: Leaving with FAILURE = VSS_E_VOLUME_NOT_SUPPORTED_BY_PROVIDER.

11:01:31 04/07/2014 BeginPrepareSnapshot: Leaving with FAILURE = VSS_E_VOLUME_NOT_SUPPORTED_BY_PROVIDER.

11:04:39 04/07/2014 OnUnload: Entering.

11:04:39 04/07/2014 CancelSnapshotSet: Entering.

11:04:39 04/07/2014 ~CSnapshotSet: Entering destructor.

11:04:39 04/07/2014 CSnapshotSet::FreeArrayData() entering.

11:04:39 04/07/2014 CSnapshotSet::FreeArrayData() exiting.

11:04:39 04/07/2014 CSnapshotSet::FreeLunData() entering.

11:04:39 04/07/2014 CSnapshotSet::FreeLunData() exiting.

11:04:39 04/07/2014 ~CSnapshotSet: Leaving destructor with success.

11:04:39 04/07/2014 CancelSnapshotSet: Leaving with SUCCESS.

11:04:39 04/07/2014 OnUnLoad: EMC VSS Provider has successfully unloaded.

11:04:39 04/07/2014 OnUnload: Leaving with SUCCESS.