Avamar: Clean up orphaned Hyper-V vmgroups manually

Invoke-Command -computername (get-clusternode).name -scriptblock { get-vmgroup | remove-vmgroup }

Invoke-Command -computername (get-clusternode).name -scriptblock { get-vmgroup | remove-vmgroup -Force }

Get-ExecutionPolicy

Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass

NOTE: This procedure is only applicable for Hyper-V clusters that do NOT have the "shared storage path" parameter configured. When it is not configured, the vmgroups are located in the local directory on each cluster node (C:\ProgramData\Microsoft\Windows\Hyper-V\Groups). 

 

Review the following KB for more details about configuring the "shared storage path" parameter.
Avamar: Hyper-V VSS plug-in - Configuration and installation of latest cumulative hotfixes for version 19.4 and higher 

For Hyper-V clusters where the "shared storage path" parameter is configured, these procedures are not applicable. Contact Customer Support for further troubleshooting 

Step# 1: Run the following command on each Hyper-V cluster node:

get-VMGroup

Alternatively, run the following command to get the vmgroups from each cluster node:

Invoke-Command -computername (get-clusternode).name -scriptblock { get-vmgroup }

Step#2a: If the VMMembers field is populated, then run the following command on each cluster node:

Get-VMGroup | % {Remove-VMGroupMember -VMGroup $_ -VM (Get-VM)}

Step#2b: If VMMembers is not populated:

1. Go to the C:\ProgramData\Microsoft\Windows\Hyper-V\Groups folder. Either type in the location manually in Windows Explorer or change the options to view hidden files and folders.
2. Edit the Virtual Machine Configuration file "*.vmcx" that matches the VM Group's InstanceID with a text editor such as Notepad or Notepad++. Click View in the Menu bar and then select 'Word wrap' option for better readability.
3. Copy the Virtual Machine's (VMs) Globally Unique Identifiers (GUIDs) into the text editor; they follow the 'GUID H' text in the *.vmcx file.   
4. In the text editor, click Edit and then Replace. Then in the 'Find what' field press spacebar on the keyboard and then click 'Replace All.' This cleanup provides a list of problematic VM GUIDs that must be Live Migrated back to the previous cluster node.
5. Run the following PowerShell command to find the VM tied to those GUIDs, along with the cluster nodes hosting it:

Invoke-Command -ComputerName (Get-ClusterNode).Name -ScriptBlock { Get-VM | ? VMID -eq "VM GUID Previously Collected" }

For example, use the below command syntax to find the Virtual Machine (VM) name that is assigned to the following GUID:

Invoke-Command -ComputerName (Get-ClusterNode).Name -ScriptBlock { Get-VM | ? VMID -eq "A6209559-F08F-440E-9AAE-21AEF4AF7E9D" }

The output from the command should include the VM name and the cluster node it is running on. If the output is blank, then the GUID no longer exists and can be safely ignored.

     6. Live migrate the VM to the host that has the stale VM Group using Failover Cluster Manager tool.

 
Repeat steps for each VM identified in the vmgroup's "*.vmcx" file. 

     7. Run the following PowerShell command to remove the VMs from the VMMembers field in each of the orphaned vmgroups:
    

Get-VMGroup | % {Remove-VMGroupMember -VMGroup $_ -VM (Get-VM)}

Confirm that the 'vmmembers' field for each of the vmgroups is blank by running the following command on each cluster node:

get-vmgroup

Alternatively, run the following command to confirm that the 'vmmembers' field is blank for all vmgroups on all nodes:

Invoke-Command -computername (get-clusternode).name -scriptblock { get-vmgroup }

 Once confirmed, then go to Step#3.

Step#3: Run the following PowerShell command to remove the orphaned vmgroups on the cluster node or standalone Hyper-V server:

get-VMGroup | Remove-VMGroup

Repeat the same procedures for other cluster nodes that have orphaned vmgroups.

Alternatively, use the following command to remove orphaned vmgroups on all cluster nodes simultaneously:

Invoke-Command -computername (get-clusternode).name -scriptblock { get-vmgroup | remove-vmgroup }

Once all nodes are completed, verify that all orphaned vmgroups have been removed:

Invoke-Command -computername (get-clusternode).name -scriptblock { get-vmgroup }

The "remove-vmgroupmember" and "remove-vmgroup" commands may fail on the first cluster node with an error similar to the following:

Remove-VMGroup : The operation cannot be performed while the object is in its current state

If that is the case, then try the commands on the remaining cluster nodes. If the procedures complete successfully, then go back to the first node and retry the PowerShell commands.

If the above commands fail on all cluster nodes, then restart the Hyper-V Virtual Machine Management service on the affected node. Alternatively, reboot each of the Hyper-V cluster nodes. Once done, then retry the PowerShell commands again to remove the VM members and vmgroups.

If the standard or manual procedures fail to remove the orphaned vmgroups, then the final option is to remove the vmgroup files themselves.