1 Copper

disk.DiskMaxIOSize & VMAX

I was working with a customer who uses vSphere 5 connected to a VMAX array.  The customer is an internal service providor for many departments and the workload is both highly varied and totally out of the control of the VMware and storage administrators.  The system is being used at more than double its original intended capacity and they ran into performance issues.  Right-sizing the storage (spindles, engines, front-end ports) helped dramatically, but they are looking to get every ounce of performance out of the system they can find.

The latest inquiry from the storage team (and my VMAX expert colleage) was about the vSphere advanced option "disk.DiskMaxIOSize" and if it should be tweaked to a smaller number to take advantage of the VMAX's higher performance with smaller block IOs.  I suggested that while it may help, the default 32M setting was much higher than the expected IO size from most of the guest OSs, so it would have to be lowered dramatically to make any difference.  I was also concerned about the additional latency (and CPU use) of chunking the IOs on the hosts.  My colleagues Matt Cowger and Cody Hosterman shared some real-world and lab test experiences that backed that up:

1st (Matt): This option primarily exists to protect older arrays that cannot handle large IOs.  When certain arrays receive IOs they cannot handle, they respond with ABORT, which causes the ESX/ESXi host to fail the path the IO was sent on.  Thus, this option is not for performance - it should be used in the rare circumstance you are using an old array that breaks with large IOs.  It is better to let the VMAX break up the IOs and place the data optimally rather than try and break it up at the host level.

2nd (Cody): Testing shows that the process of breaking up the IOs on the host costs enough processing time to eliminate whatever performance benefit you may get on the back-end (AB: I assume in many cases this will work to the detriment of performance, taking longer on the general purpose host than on the optimized array).

So, the short answer: limiting DiskMaxIOSize when using a VMAX will not improve performance, will likely reduce performance, and will cost you host CPU resources.  Better to leave that at the default.