Start a Conversation

Unsolved

This post is more than 5 years old

5489

September 2nd, 2010 20:00

high queued -i/o's in powermt output

We had a issue with a node of a windows cluster. There are 6 devices mapped and masked to that node from DMX-4. When i see the powermt display dev=all out put i see that out of 6 devices 1 of them has High queued i/o's .The rest 5 dont have any queued i/o's . I did some analysis on that device and i see the below readings.


        DEVICE                   IO/sec        KB/sec     % Hits   %Seq     Num WP
19:48:11                       READ  WRITE    READ  WRITE  RD  WRT  RD  WRT  Tracks

19:48:11 DEV001       (0100)    124   2154   16776    5241 100  96   13  53    753

The FA port utilization also seems to be high

19:47:04    DIRECTOR    PORT      IO/sec     Kbytes/sec

19:47:04      FA-8C        1        2589          27628
19:47:04      FA-9C        1        5085          84102

can anyone tell me how to reduce the queued I/o's for that particular device (0100)

Pseudo

Symmetrix

Logical device

state=alive; policy=SymmOpt; priority=0; queued-IOs=28

==============================================================================

---------------- Host ---------------   - Stor -   -- I/O Path -  -- Stats ---

### HW Path                 I/O Paths    Interf.   Mode    State  Q-IOs Errors

==============================================================================

   2 port2\path0\tgt0\lun20    c2t0d20   FA  9cB   active  alive     14      0

   4 port4\path0\tgt0\lun20    c4t0d20   FA  8cB   active  alive     14      0

1.3K Posts

September 2nd, 2010 20:00

Sounds like you answered you own question.  The FA utilization is high, so adding more FA CPUs (not just ports) will most likely reduce the queues.

1 Rookie

 • 

137 Posts

September 3rd, 2010 02:00

Hello,

   Just to also add thats queued I/O's are historic as well. If they are stable at 14 and not incrementing they may indicate that there was an issue at one time, but not necessarily now.

The queued I/Os counter is only cleared at reboot when it is returned to zero.

Hope this helps,

   Nollaig

2 Intern

 • 

20.4K Posts

September 3rd, 2010 02:00

Quincy56,

wouldn't high FA utilization affect all devices in that case and not just one ?

2 Intern

 • 

20.4K Posts

September 3rd, 2010 02:00

Nollaig,

are you talking about PowerPath "Stats Q-IOs" counter ?

1.3K Posts

September 3rd, 2010 04:00

I am assuming all the devices do not have the same IO rate.  The devices with low IO rates won't show high queues.

152 Posts

September 3rd, 2010 04:00

That disk(0100) is a transaction log disk which is experiencing slow writes. The remaining disks have very less I/O or no I/O at all. The Queued I/O's on this Particular transaction log disk is varying from time to time and they are not able to process transactions fastly. So, can anyone tell me how to  minimize this effect. Will adding FA's help me to improve the perfromance or there any other ways for better perfromance for that particular drive (0100).

1.3K Posts

September 3rd, 2010 04:00

Of all the IO types the FA can process, writes to slots not yet write pending are the hardest for it to process.  They also tend to come in bursts.  I would bet good money that adding two more paths will significantly reduce the response times.

88 Posts

September 3rd, 2010 04:00

So it appears that a particular device is singled out and can focus there. What is different about this device than the others..? Is it a quorum device..? Did this just start happening and the only occurance..? Also, you can set "throttling-related" values per device, see: "PowerPath CLI and System Messages Reference Guide". In addtition, can also use powermt display with arguments (ex: powermt display dev=<> every=<>), to watch in intervals.. I know that powermt restore clears "historical errors" but unable to conclude this will help with queued I/O, due to being 'real time'. As Nolliag stated, need to see if these values change while being monitored...

1 Rookie

 • 

137 Posts

September 3rd, 2010 07:00

Hello,

  Sorry Dynamox, just saw query. Looks like possible issue has been nailed down now.

Thanks again!

   Nollaig

152 Posts

September 3rd, 2010 09:00

I decided to add two more FA's for the existing windows cluster. But I need some confirmation on the FA port settings. This is what we have at present on

on 8c:1 which is conencted to the cluster

SCSI Flags
            {
              Negotiate_Reset(N)           : Disabled
              Soft_Reset(S)                : Disabled
              Environ_Set(E)               : Disabled
              HP3000_Mode(B)               : Disabled
              Common_Serial_Number(C)      : Enabled
              Disable_Q_Reset_on_UA(D)     : Disabled
              Sunapee(SCL)                 : Disabled
              Siemens(S)                   : Disabled
              Sequent(SEQ)                 : Disabled
              Avoid_Reset_Broadcast(ARB)   : Disabled
              Server_On_AS400(A4S)         : Disabled
              SCSI_3(SC3)                  : Enabled
              SPC2_Protocol_Version(SPC2)  : Enabled
              SCSI_Support1(OS2007)        : Disabled
            }
The other FA (8a:1)which  we thought of adding to the cluster has the below settings
SCSI Flags
            {
              Negotiate_Reset(N)           : Disabled
              Soft_Reset(S)                : Disabled
              Environ_Set(E)               : Disabled
              HP3000_Mode(B)               : Disabled
              Common_Serial_Number(C)      : Enabled
              Disable_Q_Reset_on_UA(D)     : Disabled
              Sunapee(SCL)                 : Disabled
              Siemens(S)                   : Disabled
              Sequent(SEQ)                 : Disabled
              Avoid_Reset_Broadcast(ARB)   : Disabled
              Server_On_AS400(A4S)         : Disabled
              SCSI_3(SC3)                  : Enabled
              SPC2_Protocol_Version(SPC2)  : Disabled
              SCSI_Support1(OS2007)        : Disabled
            }

Can any one confirm if i can go ahead and use 8a:1 and 9a:1 as additional FA's for the windows cluster. SPC2 bit is different on both .

2 Intern

 • 

20.4K Posts

September 3rd, 2010 09:00

what version of windows ?

131 Posts

September 3rd, 2010 10:00

FA   Config Bits C D SCL ARB SC3 SPC-2 OS2007 V PP UWN VCM EAN
FA Bit   Description Common   Serial Number Disable   Queue Reset on Unit Attention Enable   Sunapee (Sun Cluster) Avoid   Reset Broadcast SCSI-3   Compliance SCSI-3   SPC-2 Compliance Host   SCSI
Compliance
2007
Enable   Volume Set Addressing Point-to-Point Unique   World Wide Name FC   VCM Enable   Auto Negotiation
Windows   2008 X X X O X X X X

152 Posts

September 3rd, 2010 10:00

If we set via hba_flags do we need to reboot the cluster in order to take effect the settings. 

2 Intern

 • 

20.4K Posts

September 3rd, 2010 10:00

or you can drop the zone from that host to the new FAs

Message was edited by: dynamox

152 Posts

September 3rd, 2010 10:00

Server Name CRPRCHMCS86N1
OS Name Win2008 64Bit - Ent
OS Version Service Pack 2

These are the Windows cluster details..

No Events found!

Top