UNSOLVED

dynamox

updated

16 years ago

D

dynamox

11 Legend

20419 Posts

87439 Points

2

3262

October 19th, 2010 13:00

Write times and Write pending correlation

Hello guys,

i am trying to make sense out of this situation. Why during the time when i have over 20000 write pendings (9:15am), average write times is good (less than 20ms), yet when write pendings are very low (around 10:15am), i have almost 90ms write times. Write pending threshold is around 55000 per devices (5% of total system write pending slots). This is 100% write workload.  Any clues ?

10-19-2010 4-34-56 PM.png

  • Quincy561

    1281 Posts

    799

    0

    Posted October 19th, 2010 14:00

    When the backend can't keep up with the frontend, the WP counts get high.

    If you are seeing low WP counts along with poor write response times, I would suspect a frontend bottleneck.

  • Boom1

    131 Posts

    799

    0

    Posted October 19th, 2010 14:00

    it could be possible..

    You are flushing your cache and you have low WP so your backend is good.

    but you have High Write response time so your Frontends are not pushing enough data in cache or backend.

  • dynamox

    11 Legend

    20419 Posts

    87439 Points

    799

    0

    Posted October 19th, 2010 14:00

    If you are seeing low WP counts along with poor write response times, I would suspect a frontend bottleneck.

    i don't think i am following you. If i have very low wp counts why would i have very high write times in symmetrix (as seen in that chart)  ?

  • Quincy561

    1281 Posts

    799

    0

    Posted October 19th, 2010 15:00

    I was suggesting that the writes can't get into the system fast enough because you don't have enough FA resources.

  • dynamox

    11 Legend

    20419 Posts

    87439 Points

    800

    0

    Posted October 20th, 2010 07:00

    Quincy,

    i could be looking at it all wrong, please correct me. When i look at chart with "Sampled average write times", this is from FA perspective ..how long it took for it to send request to the system and get it back. So if write pendings are low, there is nothing sitting in cache ..why would FA wait for so long.

    Thank you

  • Quincy561

    1281 Posts

    800

    0

    Posted October 20th, 2010 08:00

    I'm suggesting that you are writing to the FA faster than it can deal with the IOs, so it has to queue them.  I would suggest adding FA CPUs (not just ports) and see if the write response time gets better.

  • dynamox

    11 Legend

    20419 Posts

    87439 Points

    800

    0

    Posted October 20th, 2010 09:00

    this device is mapped to four FAs ..you tell me ..but they don't seem to be busy to me (these are CPU stats).

    10-20-2010 12-36-46 PM.png

  • dynamox

    11 Legend

    20419 Posts

    87439 Points

    800

    0

    Posted October 20th, 2010 10:00

    could be, but if you look at "sample average write time (ms)" ..it stays very high for almost an hour.  These devices are configured for SRDF/S, what metrics can i look in Performance Manager to see if SRDF could have anything to do with that ?

  • Quincy561

    1281 Posts

    800

    0

    Posted October 20th, 2010 10:00

    No, they don't look busy over that sample interval, but what about during a one or two second sample?

    We have seen many cases where writes burst for just a very short period of time, then stop.  During that short period of time they can overwhelm the front end causing high response times.

    It still could be something else such as a SAN problem or slot contention.

  • dynamox

    11 Legend

    20419 Posts

    87439 Points

    323

    0

    Posted October 20th, 2010 11:00

    we have a case opened but PSE kicked it back to local support and it takes local guys forever to setup STP and all that stuff. I am paying boat load of money for ECC/Performance Manager so trying to research myself as well. Thanks for suggestions so far.