Hardward VSS snapshot issue with CommVault on Hyper-V 2012

Hi,

Customer has a two node Hyper-V 2012 cluster. HIT 5.0 installed on both hosts.

Commvault is installed onto a VM.

Problem: Backups fail when VMs are on the 2nd host.

The snapshot is created and set online:
EqualLogic:MgmtExec:10-Oct-2017 09:01:53.669255:snapshotSetAdminStatus.cc:300:INFO::7.2.52:Snapshot FS01-VMFiles-2017-10-10-09:01:34.22054.1 was set online by the administrator.

EqualLogic:MgmtExec:10-Oct-2017 09:01:34.539251:SnapInfo.cc:82:INFO::8.2.4:Successfully created snapshot FS01-VMFiles-2017-10-10-09:01:34.22054.1

From there we can see it is logged into successfully from VM01:

EqualLogic:MgmtExec:10-Oct-2017 09:01:57.499257:targetAttr.cc:850:INFO::7.2.14:iSCSI login to target '10.101.0.4:3260, iqn.2001-05.com.equallogic:0-fe83b6-e5e7918cf-3460056884859dc7-fs01-vmfiles-2017-10-10-09:01:34.22054.1' from initiator '10.101.0.11:57467, iqn.1991-05.com.microsoft:hv1.domain.com' successful, using Jumbo Frame length.

Once it has logged in the connection does not last too long and the connection is closed and the snapshot is set offline:

EqualLogic:MgmtExec:10-Oct-2017 09:02:33.079261:targetAttr.cc:1455:INFO::7.2.15:iSCSI session to target '10.101.0.4:3260, iqn.2001-05.com.equallogic:0-fe83b6-e5e7918cf-3460056884859dc7-fs01-vmfiles-2017-10-10-09:01:34.22054.1' from initiator '10.101.0.11:57467, iqn.1991-05.com.microsoft:hv1.domain.com' was closed.

EqualLogic:MgmtExec:10-Oct-2017 09:02:37.729262:snapshotSetAdminStatus.cc:81:INFO::7.2.51:Snapshot FS01-VMFiles-2017-10-10-09:01:34.22054.1 was set offline by the administrator.

The snapshot is brought online again and this time it is logged into by VM02:

EqualLogic:MgmtExec:10-Oct-2017 09:02:58.159266:snapshotSetAdminStatus.cc:300:INFO::7.2.52:Snapshot FS01-VMFiles-2017-10-10-09:01:34.22054.1 was set online by the administrator.

EqualLogic:MgmtExec:10-Oct-2017 09:03:01.889268:targetAttr.cc:850:INFO::7.2.14:iSCSI login to target '10.101.0.5:3260, iqn.2001-05.com.equallogic:0-fe83b6-e5e7918cf-3460056884859dc7-fs01-vmfiles-2017-10-10-09:01:34.22054.1' from initiator '10.101.0.13:50505, iqn.1991-05.com.microsoft:hv2.domain.com' successful, using Jumbo Frame length.

This time there is no logout from VM02 and from there the snapshot is set offline again then back online with VM01 trying to log back into the snapshot but failing either due to it being offline or the connection from VM02 still existing:

EqualLogic:MgmtExec:10-Oct-2017 09:06:16.499289:snapshotSetAdminStatus.cc:81:INFO::7.2.51:Snapshot FS01-VMFiles-2017-10-10-09:01:34.22054.1 was set offline by the administrator.

EqualLogic:MgmtExec:10-Oct-2017 09:13:20.799311:snapshotSetAdminStatus.cc:300:INFO::7.2.52:Snapshot FS01-VMFiles-2017-10-10-09:01:34.22054.1 was set online by the administrator.

EqualLogic:MgmtExec:10-Oct-2017 09:13:58.849318:targetAttr.cc:593:ERROR::7.4.3:iSCSI login to target '10.101.0.4:3260, iqn.2001-05.com.equallogic:0-fe83b6-e5e7918cf-3460056884859dc7-fs01-vmfiles-2017-10-10-09:01:34.22054.1' from initiator '10.101.0.11:57722, iqn.1991-05.com.microsoft:hv1.domain.com' failed for the following reason:
Initiator cannot access this target because an iSCSI session from another initiator already exists and multihost access is not enabled for this target.

This behavior continues till the snapshot is deleted

EqualLogic:agent:10-Oct-2017 10:02:03.1000010:echoCli.c:10611:AUDIT:grpadmin:22.7.0:User action:volume select FS01-VMFiles snapshot delete FS01-VMFiles-2017-10-10-09:01:34.22054.1

The same issues are seen also with FS01-DATA

EqualLogic:MgmtExec:10-Oct-2017 09:01:34.169248:SnapInfo.cc:82:INFO::8.2.4:Successfully created snapshot FS01-DATA-2017-10-10-09:01:34.22053.1

EqualLogic:MgmtExec:10-Oct-2017 09:01:42.199252:snapshotSetAdminStatus.cc:300:INFO::7.2.52:Snapshot FS01-DATA-2017-10-10-09:01:34.22053.1 was set online by the administrator.

EqualLogic:MgmtExec:10-Oct-2017 09:01:46.539254:targetAttr.cc:850:INFO::7.2.14:iSCSI login to target '10.101.0.5:3260, iqn.2001-05.com.equallogic:0-fe83b6-e5e7918cf-8d80056884559dc7-fs01-data-2017-10-10-09:01:34.22053.1' from initiator '10.101.0.11:57464, iqn.1991-05.com.microsoft:hv1.domain.com' successful, using Jumbo Frame length.

EqualLogic:MgmtExec:10-Oct-2017 09:02:26.439259:targetAttr.cc:1455:INFO::7.2.15:iSCSI session to target '10.101.0.5:3260, iqn.2001-05.com.equallogic:0-fe83b6-e5e7918cf-8d80056884559dc7-fs01-data-2017-10-10-09:01:34.22053.1' from initiator '10.101.0.11:57464, iqn.1991-05.com.microsoft:hv1.domain.com' was closed.

EqualLogic:agent:10-Oct-2017 09:02:32.429913:echoCli.c:10611:AUDIT:grpadmin:22.7.0:User action:volume select FS01-DATA snapshot select FS01-DATA-2017-10-10-09:01:34.22053.1 offline

EqualLogic:agent:10-Oct-2017 09:02:48.149916:echoCli.c:10611:AUDIT:grpadmin:22.7.0:User action:volume select FS01-DATA snapshot select FS01-DATA-2017-10-10-09:01:34.22053.1 online

EqualLogic:MgmtExec:10-Oct-2017 09:02:51.819265:targetAttr.cc:850:INFO::7.2.14:iSCSI login to target '10.101.0.4:3260, iqn.2001-05.com.equallogic:0-fe83b6-e5e7918cf-8d80056884559dc7-fs01-data-2017-10-10-09:01:34.22053.1' from initiator '10.101.0.13:50501, iqn.1991-05.com.microsoft:hv2.domain.com' successful, using Jumbo Frame length.

EqualLogic:agent:10-Oct-2017 09:04:53.999919:echoCli.c:10611:AUDIT:grpadmin:22.7.0:User action:volume select FS01-DATA snapshot select FS01-DATA-2017-10-10-09:01:34.22053.1 offline

EqualLogic:MgmtExec:10-Oct-2017 09:10:59.539298:targetAttr.cc:593:ERROR::7.4.3:iSCSI login to target '10.101.0.5:3260, iqn.2001-05.com.equallogic:0-fe83b6-e5e7918cf-8d80056884559dc7-fs01-data-2017-10-10-09:01:34.22053.1' from initiator '10.101.0.11:57688, iqn.1991-05.com.microsoft:hv1.domain.com' failed for the following reason:
Initiator cannot access this target because an iSCSI session from another initiator already exists and multihost access is not enabled for this target.

These events are seen on the Hyper-V hosts event logs

 

10Oct17:09:02:02 Source: Microsoft-Windows-Ntfs Type: Information The description for Event ID '98' in Source 'Microsoft-Windows-Ntfs' cannot be found.  The local computer may not have the necessary registry information or message DLL files to display the message, or you may not have permission to access them.  The following information is part of the event:'FS01-DATA', '\Device\HarddiskVolume2427', '0'

 

10Oct17:09:02:02 Source: Microsoft-Windows-Ntfs Type: Information The description for Event ID '98' in Source 'Microsoft-Windows-Ntfs' cannot be found.  The local computer may not have the necessary registry information or message DLL files to display the message, or you may not have permission to access them.  The following information is part of the event:'\\?\Volume{42a71f3a-ad91-11e7-93fa-f4e9d4b7cdb2}', '\Device\HarddiskVolume2427', '0'

 

10Oct17:09:02:03 Source: Microsoft-Windows-Ntfs Type: Information The description for Event ID '98' in Source 'Microsoft-Windows-Ntfs' cannot be found.  The local computer may not have the necessary registry information or message DLL files to display the message, or you may not have permission to access them.  The following information is part of the event:'C:\Program Files\CommVault\Simpana\MediaAgent\SnapVolumeMounts\SnapMnt_1_2_461373_14_16634', '\Device\HarddiskVolume2427', '0'

Please can anyone share any thoughts, thanks.