VPLEX + ESXi Round-Robin Multipath

Perfect! Thanks a lot for your clear explanation. I'll keep on eye on the compatibility guide and host connectivity guide.

Any official news? From support.emc.com I can see that

"Host Connectivity Guide for VMWare ESX Server" has not been updated yet.

Thank you

I'd like to see the EMC docs updated too. But if you want something official, the VMware Compatibility Guide does list Round Robin. So I'm going to go ahead and recommend RR, except in the case of cross-connected Metro Cluster.

Here's the link.

It would be nice if we could define a preferred director or preferred ports for each volume in a storage view, and have VPLEX advertise those ports as ALUA optimized. That would be a way to eliminate the manual load balancing on the host side, and the issue with inter-director cache traffic.

Hi - the targeted release update to the ESX Host connectivity guide stating that RR is preferred for VPLEX is Q3/14.

Gary

Yes.  This applies to all supported (on VPLEX support matrix) versions of VPLEX code.

-Don

So, in my case I have ESXi 5.1 and 5.5 hosts and Geosynchrony 5.2.

Can you double-confirm the current recommended setting is Round-Robin?

Thanks again.

The VPLEX best practices guides have been recently updated and are the most accurate info available.  Here is a link to the latest guide, see page 11:

http://www.emc.com/collateral/technical-documentation/h13547-vplex-host-multipathing-best-practices.pdf

Hi garyo, can you please confirm that as I can see from here

https://support.emc.com/docu5265_Host-Connectivity-Guide-for-VMWare-ESX-Server.pdf?language=en_US

Round Robin multipathing is now the recommended setting? Does it apply for VPLEX 5.2 as well?

Thanks.

Hi there,

Yes, as Don says, Round-robin is preferred over Fixed. 

But I stress that there's technically nothing functionally incorrect with Fixed, it's just that round-robin is a much easier means to load balance.  Fixed policy has so much manual effort to get right (every host, every volume) and non-deterministic failover properties (no idea which path will get picked should the primary fail) that round-robin is simply "set it and forget it".

Gary

Garyo, the one you described is just my case. I've recently presented new luns with vplex and I've added new physical hosts to the environment. Keeping everything manually balanced with fixed path policies it's really really hard. I have to do thousands of clicks or copy-and-paste luns IDs in a script code to set it up.

That is really time consuming and prone to errors.

I'll go with the Round-Robin configuration on all my physical hosts, it's really easier to manage.

Thanks for your feedback.

Hi Burhan,

I think inefficient is too strong of a word.  Yes, you have the potential for more duplicated read cached pages among directors with a round-robin policy, but I believe that the probability of that happening is low.  (A host frequently reading the same page over and over again surely has it cached locally?)

Writes that have invalidate cache pages on other directors do increase the possibility of additional inter-director messages in round-robin that you otherwise don't get in a Fixed policy environment.

Do you get more inter-director cache-coherency chatter among directors in round-robin situation vs. Fixed? Sure, but I believe that the additional overhead of that is small.  In our performance lab, we test the differences between the two policies, and the latency differences are very small - near negligible in my opinion.  Inter-director messaging within a cluster is usually quite small - tens of microseconds usually, typically less than say 100usec.

If anything round-robin-ing across more directors has the potential to increase the total available cache for a given volume.  Each VS2 director offers roughly 32GB of usable cache space.  Now, how the cache gets used is entirely dynamic and naturally is highly an "it depends" kind of situation - number of volumes, size of volumes, total addressable space of those volumes, read vs. write ratio, etc...  So a four director storage-view just might increase the total available space for caching.

VPLEX cache stats reporting does leave a lot to be desired, since right now it doesn't easily show you things like percent cache hits, or duplicate cache pages.  (It'd be nice if I could quote some metrics.)  This is something we're working on improving.

Finally, an absolutely perfect caching model would be for each volume to only direct I/O to an individual director, but doing that manually is next to impossible and would be a forever losing battle.  Folks are certainly are welcome to do that, but as shown for any sizable VPLEX deployment - hundreds of hosts and thousands of volumes the manual effort required is substantial.  It's also not guaranteed to be right because different volumes and different hosts are active at different times - some frequently, others not at all...

Round-robin is like the "easy button" for balancing work across all the directors :-)

Gary

Interesting discussion. I have an aspect to add to the question "fixed or - new recommendation - round robin": it's an NDU. IHAC with a bunch of ESXi 5.x Hosts attached to a Metro Cluster running Geosynchrony 5.1.0.4 and we have an update to 5.3 SP2 being scheduled.

Now SolvE is apparently not (yet?) aware of the new recommendation or at least doesn't distinguish between whether or not the hosts are being cross-connected.

I recently generated the procedure for the upgrade to  be able to fulfill all the prereqs and in the document, under "Step 1 - Verify the Host environment" for VMware there is clearly stated:

"VMware: Set Native Multipathing to Fixed"

So since I have to take care of preventing the attached servers from crashing and in this environment, the hosts are actually on round robin, I got anxious and had to doublecheck with EMC and was told "you're fine with RR unless being cross-connected. If you are, reconfigure to fixed". I was keen on getting this piece of information in a written form, providing a statement that RR in my case is officially being supported to get support in the case of issues during the NDU.

As of now, I don't have this statement yet, but i'm assuming that "for NDUs with NMP in cross-connected topologies fixed only" is due to preventing the hosts from chosing paths across the link b/c NMP doesn't provide an auto-standby feature as PP/VE does. So it's to circumvent a mess in terms of local and remote path connectivity. Is it that simple? There's a couple of statements in the mentioned best practice WP with regards to this under "RR - New Policy Recommendation".

It's not explicitly stated, but I draw the obvious corollary that in a non-cross-connected topology, RR is still ok, even for an NDU if i'm on NMP. So the new recommendation holds true.

Thanks, the inconsistency in the Solve Desktop document has been raised and an internal defect has been filed to have it corrected.

> As of now, I don't have this statement yet, but i'm assuming that "for NDUs with NMP in cross-connected topologies fixed only" is due to preventing the hosts from chosing paths across the link b/c NMP doesn't provide an auto-standby feature as PP/VE does. So it's to circumvent a mess in terms of local and remote path connectivity. Is it that simple? There's a couple of statements in the mentioned best practice WP with regards to this under "RR - New Policy Recommendation".

In a cross-connected environment with NMP RR you wouldn't want the host to choose a remote path, but in the case of path failure you really have no idea which path it's going to pick (local vs. remote.)  But if your inter-cluster latency isn't that bad, then having a host choose a remote path during an NDU may not necessarily be the end of the world or result in any kind of failures.  And so if failback is working properly you might get back the local cluster as the primary path.  Again, all random though and requires manual balancing.  Use PP/VE with auto-standby for cross-connect to avoid all this.

Gary