I am sorry my previous entry had information which is not correct. That was an extract from my first draft that I had created and we had the same issue as you have now.
Basically you want all the vHBAs zoned to every SP port, with the reason being that Hyper-V dynamically creates vHBAs on physical HBAs at time of live migration and there is no guarantee that a specific vHBA (Port Set A or Port Set B) will be created on a specific physical HBA.
I am so sorry for the wrong information before, I had to check my documentation to understand my mistake
Please let us know how things go.