VMware

[Anonymous]

Hello,

Where did the difference between "rr" and "iops" as the policy come from?

Our customer got two type of policy as a command result of "esxcli storage nmp device list" after he set his IO Operation limits by the following two ways following "Using EMC VMAX Storage in VMware vSphere Environments" page 84.

It turn out the output both way show different result (highlighted in red).

1. Set by

esxcli storage nmp satp rule add -s "VMW_SATP_SYMM" -V "EMC" -M "SYMMETRIX" -P "VMW_PSP_RR" -O "iops=1"

And got from "esxcli storage nmp device list"

Path Selection Policy Device Config: {policy=rr,iops=1,bytes=10485760,useANO=0; lastPathIndex=0: NumIOsPending=0,numBytesPending=0}

2. Set by

esxcli storage nmp psp roundrobin deviceconfig set --device=<device NAA> --iops=1 --type iops

And got from "esxcli storage nmp device list"

Path Selection Policy Device Config: {policy=iops,iops=1,bytes=10485760,useANO=0; lastPathIndex=1: NumIOsPending=0,numBytesPending=0}

I found the similor question "XIO ESX 5.5 setting" in a private space so that it hasn't been answered yet.

Best regards,

Hiroki

[WorldPolice]

So this is simply because the first statement is changing the default rule (for type rr) for all new devices to use 1 iops instead of 1000 for the SATP.  The second statement is changing the behavior of an individual device, and that is done by iops or size (bytes).  There is no 'rr' type.  You'll notice that the columns you are not showing, stay the same on both statements for the device:

  Storage Array Type: VMW_SATP_SYMM

   Storage Array Type Device Config: {action_OnRetryErrors=off}

   Path Selection Policy: VMW_PSP_RR

So, not speaking for VMware of course, my assumption is the policy name differs so that the consumer knows when it is a rule at the system level, or was changed at the device level.  The behavior is the same, however.

[Anonymous]

Hello Drew,

So array level settings are displayed by "esxcli storage nmp device list" if only SATP setting has been done and no PSP setting hasn't been done yet? And the device level settings are displayed by the same list command "esxcli storage nmp device list" if the device level settings (PSP) for the devices ("IO Operation limit") has been changed for the specific devices once?

I'm sorry but I can't find the document for the detail information for the command "esxcli storage nmp device list".

Thank you for answering.

Best regards,

Hiroki

[DCsafe]

Hi Hiroki,

Found some information on things you are looking for. See if that helps

* Storage Array Type Plug-Ins (SATPs) run in conjunction with the VMware NMP and are responsible for array-specific operations.

https://pubs.vmware.com/vsphere-51/index.jsp?topic=%2Fcom.vmware.vsphere.storage.doc%2FGUID-DB5BC795...

* To check if SATP for particular storage array is loaded or not

~ # esxcli storage nmp satp list

eg output

Name                 Default PSP    Description

-------------------  -------------  ------------------------------------------

VMW_SATP_ALUA        VMW_PSP_MRU    Placeholder (plugin not loaded)

MW_SATP_ALUA_CX     VMW_PSP_FIXED  Placeholder (plugin not loaded)

VMW_SATP_SYMM        VMW_PSP_FIXED  Placeholder (plugin not loaded)

VMW_SATP_CX          VMW_PSP_MRU    Placeholder (plugin not loaded)

* Please go through

http://buildvirtual.net/configuring-multipathing-policies-on-esxi-5/ for more information on PSP, SATP, IOPs

[WorldPolice]

I don't want to confuse things more so let me see if I can expand in a simpler way.  VMAX has an SATP plug-in which is part of the ESXi code.  This plug-in ensures that all VMAX devices presented to an ESXi host will be assigned a path selection policy of RoundRobin when using NMP and a default IOPS=1000.  When you run this command "esxcli storage nmp device list" it is going to tell you about all the devices on the box.  Here is one of the devices on my VMAX/ESXi 6 environment:

naa.60000970000197200056533030303036

   Device Display Name: EMC Fibre Channel Disk (naa.60000970000197200056533030303036)

   Storage Array Type: VMW_SATP_SYMM

   Storage Array Type Device Config: {action_OnRetryErrors=off}

   Path Selection Policy: VMW_PSP_RR

   Path Selection Policy Device Config: {policy=rr,iops=1000,bytes=10485760,useANO=0; lastPathIndex=0: NumIOsPending=0,numBytesPending=0}

   Path Selection Policy Device Custom Config:

   Working Paths: vmhba2:C0:T0:L5, vmhba65:C0:T0:L5

   Is USB: false

That device was recognized as a VMAX device and was assigned the default rule associated with a VMAX device which says use PSP RoundRobin and IOPS=1000.  Now if I execute #1 in your question to change the rule and then reboot my box my device will look like this:

naa.60000970000197200056533030303036

   Device Display Name: EMC Fibre Channel Disk (naa.60000970000197200056533030303036)

   Storage Array Type: VMW_SATP_SYMM

   Storage Array Type Device Config: {action_OnRetryErrors=off}

   Path Selection Policy: VMW_PSP_RR

   Path Selection Policy Device Config: {policy=rr,iops=1,bytes=10485760,useANO=0; lastPathIndex=0: NumIOsPending=0,numBytesPending=0}

   Path Selection Policy Device Custom Config:

   Working Paths: vmhba2:C0:T0:L5, vmhba65:C0:T0:L5

   Is USB: false

Now if I don't want to reboot my box, I can execute #2 from your question  "esxcli storage nmp psp roundrobin deviceconfig set --device=<device NAA> --iops=1 --type iops".  After I run this command I now issue that same command which lists all devices "esxcli storage nmp device list".  Here is what my device looks like now:

naa.60000970000197200056533030303036

   Device Display Name: EMC Fibre Channel Disk (naa.60000970000197200056533030303036)

   Storage Array Type: VMW_SATP_SYMM

   Storage Array Type Device Config: {action_OnRetryErrors=off}

   Path Selection Policy: VMW_PSP_RR

   Path Selection Policy Device Config: {policy=iops,iops=1,bytes=10485760,useANO=0; lastPathIndex=0: NumIOsPending=0,numBytesPending=0}

   Path Selection Policy Device Custom Config:

   Working Paths: vmhba2:C0:T0:L5, vmhba65:C0:T0:L5

   Is USB: false

You can see my iops are now "1" and VMware has changed the policy to "iops", letting me know that this device has been altered from the rule for VMAX devices.  By default, VMware uses iops to decide when to change paths.  Note that you can also use 'bytes' which is why that is also shown above.  Here is what my device looks like if I alter the policy to change paths every 5000000 bytes:

naa.60000970000197200056533030303036

   Device Display Name: EMC Fibre Channel Disk (naa.60000970000197200056533030303036)

   Storage Array Type: VMW_SATP_SYMM

   Storage Array Type Device Config: {action_OnRetryErrors=off}

   Path Selection Policy: VMW_PSP_RR

   Path Selection Policy Device Config: {policy=bytes,iops=1,bytes=5000000,useANO=0; lastPathIndex=1: NumIOsPending=0,numBytesPending=0}

   Path Selection Policy Device Custom Config:

   Working Paths: vmhba2:C0:T0:L5, vmhba65:C0:T0:L5

   Is USB: false

So what all this means is that the command "esxcli storage nmp device list" will simply tell you about the current policies in place for NMP for a particular device.  If you see policy=iops or policy=bytes you know that individual device's configuration was changed.  If it says policy=rr for the VMAX device it means it was assigned by rule.  As my TechBook notes on page 84, however, the VMware default for iops is 1000 so you can either change the rule (#1 in your question) and reboot, or individually change each device (#2 in your question).

I hope this explanation was helpful.

[Anonymous]

Hello Drew,

The actual path policy, which is applied to the device EMC Fibre Channel Disk (naa.60000970000197200056533030303036), hasn't changed after you issued "esxcli storage nmp psp roundrobin deviceconfig set --device=<device NAA> --iops=1 --type iops"? And just the result of the command has chnaged? Otherwise the actual path policy has also changed?

So the difference of the result of the list command just comes from if the command #2 (PSP) has been issued to specific devices?

I konw the command #1 (SATP) requires rebooting where the command #2 (PSP) doesn't reqire it.

Thank you for replying.

Best regards,

Hiroki

[Anonymous]

Hello DCsafe,

I'll look at them.

Thank you for the information.

Best Regards,

Hiroki

[WorldPolice]

Yes if I understand your interpretation that is correct.  The only difference in the list command is that the first device list was set by the rule, the second by me issuing the iops type.  In the end, though, the path handling is the same for:

{policy=rr,iops=1

{policy=iops,iops=1,

[Anonymous]

Hello Drew,

I got that the first list shows it was set by the policy rule by SATP, and the second list shows it was set by device by iops. And the both have been set to policy=Round Robin and iops=1.

I really appreciate.

Best regards,

Hiroki