UNSOLVED

bonnu06

updated

13 years ago

B

bonnu06

2 Intern

258 Posts

0

7350

April 10th, 2013 08:00

CIFS share running slow

Hello,

I have CIFS running on my VNX 5700 box. Since past week i have been seeing the CIFS share are running slower for a while and then comes back to normal. The thing is we are experiencing this slowness in the peak hours and users are complaining alot on this. I looked my backup jobs they are not running at all. Here are the stats i gathered from VNX and these look good. Any suggestions why it might be running slow..

[nasadmin@mpbwvnxacs1 ~]$ server_stats server_2 -monitor netDevices-std

server_2   device    Network     Network    Network     Network     Network     Network

Timestamp               In      In Errors      In         Out      Out Errors     Out

                      Pkts/s      diff       KiB/s       Pkts/s       diff       KiB/s

10:13:31   mge0              6          0           1           6           0           5

           mge1             16          0          21           9           0           1

           cge-1-0        2420          0         657        3892           0        2909

           cge-1-1           0          0           4           0           0           0

           cge-1-2        3675          0         827        3818           0        2351

           cge-1-3        4470          0         337        7611           0        8339

10:13:46   mge0              2          0           0           1           0           0

           mge1              5          0           7           3           0           0

           cge-1-0        3405          0        1176        5879           0        5620

           cge-1-1           0          0           4           0           0           0

           cge-1-2        4146          0        1021        5551           0        4675

           cge-1-3        4691          0         351        8045           0        8835

10:14:01   mge0              4          0           0           4           0           1

           mge1             16          0          23          11           0           1

           cge-1-0        3846          0        1806        5859           0        5317

           cge-1-1           0          0           5           0           0           0

           cge-1-2        5005          0        1291        6786           0        5479

           cge-1-3        4604          0         347        7902           0        8711

10:14:16   mge0              5          0           1           5           0           1

           mge1           1826          0        2617         988           0          83

           cge-1-0        7322          0        4584        7973           0        6351

           cge-1-1           0          0           5           0           0           0

           cge-1-2        6604          0        3326        8859           0        7038

           cge-1-3        4741          0         357        8050           0        8901

10:14:31   mge0              4          0           0           4           0           2

           mge1           2579          0        3702        1393           0         113

           cge-1-0        8323          0        2997       21106           0       24277

           cge-1-1           0          0           5           0           0           0

           cge-1-2        6078          0        2089        6825           0        6123

           cge-1-3        2249          0         168        3895           0        4311

10:14:46   mge0              1          0           0           1           0           0

           mge1              2          0           2           1           0           0

           cge-1-0        7434          0         873       25113           0       32363

           cge-1-1           0          0           5           0           0           0

           cge-1-2        2872          0         637        4428           0        4647

           cge-1-3        1558          0         117        2711           0        2997

  • dynamox

    11 Legend

    20419 Posts

    87439 Points

    3519

    0

    Posted April 10th, 2013 08:00

    could be anything, somebody running something against VNX, not even that share. Could be block connected hosts causing additional workload. Need to look at Analyzer and see what happens with array and then look at external items such such network switch load.

  • bonnu06

    2 Intern

    258 Posts

    3526

    0

    Posted April 10th, 2013 08:00


    I just turned on the celerra monitor. I am attaching a graph here can you please look into it and let me know if this helps

    Celerra_Monitor.jpg

  • dynamox

    11 Legend

    20419 Posts

    87439 Points

    3526

    0

    Posted April 10th, 2013 08:00

    Can you zoom in and see if those peaks correspond with the times users complain ?

  • bonnu06

    2 Intern

    258 Posts

    3519

    0

    Posted April 10th, 2013 08:00


    Yes i do have and i just started performance logging info...any reasons why it would be slow for a while and comes back to normal ..as the users are using those shares all day long but only for 1 hour its being really slow....

  • dynamox

    11 Legend

    20419 Posts

    87439 Points

    3519

    0

    Posted April 10th, 2013 08:00

    do you have Unisphere Analyzer to look at LUNs presented to datamovers ?

  • elitec8888

    11 Posts

    3526

    0

    Posted December 8th, 2015 07:00

    Hi Bonnu06,

    We are also experiencing simular slow preformance issues with CIFS Shares on the VNX5700 during the day.

    Did you manage to findout the root cause of the issue? Thanks

  • kelleg

    6 Operator

    4537 Posts

    3526

    1

    Posted December 10th, 2015 11:00

    This is a really old thread, you might want to start a new question thread.

    There are a number of things that can slow down an array - in this case, at one specific time during the day. Look at virus scanning - sometimes updates can cause all scanning to get reset to a specific time of day.

    Check for other jobs running on the array if you have hosts connected to the Block side.

    Check KB article 12289 - this is a master article listing all the KB's for performance on Block arrays - tips on using the Analyzer program, how to's and others.

    glen

  • elitec8888

    11 Posts

    3526

    0

    Posted December 10th, 2015 14:00

    Hi Glen, thanks for your reply. we have had EMC run a health check and no bottlenecks found. The VNX5700 is mainly just hosting users home directories and team CIFS Shares. All AV scanning is scheduled in the evenings and currently CAVA is disabled. The VNX is hardly pushed and CPU / Memory utilisation is low. During the day its slow and when all the users have left at the end of the day its fast again.

    The reason I did not create a new post is @Bonnu06 seem to have the same problem 2 years ago. I also noticed in his other posts he hosted home directories on the VNX5700 too and then later migrated the CIFS to Isilon..... so maybe he is aware of issues with home directories on VNX5700 with CIFS???  Bonnus if you are reading this then any insight much appreciate

    Regarding VNX CIFS file access problem

    Migrating CIFS share from VNX to Isilon

    p.s I can't seem to locate KB12289, is there a direct link? Thanks

  • kelleg

    6 Operator

    4537 Posts

    3526

    1

    Posted December 11th, 2015 07:00

    The link is below.  These KB's are mostly for Block storage, not File. But since the back-end block storage can impact the front-end FILE part, you'd need to take a look at the LUNs and Disks for performance issues.

    I'd recommend that you first stop Data Logging (if running), the Change the Archive Interval to 60 seconds (one minute polling) and Periodic Archiving enabled (to automatically collect each NAR file has it's created) then Set Log Period to non-stop and Start Data Logging. This will then collect a new NAR file every 2.6 hours on the array (the array can store 2GB per SP of NAR files, then the oldest is overwritten). This way you have the most granular polling interval (60 seconds) and you're capturing all the NARs. If you experience issues, then you can run the spcollects, gather up the NAR files that cover the time of the issue and open a case with EMC to take a look at the back-end performance during the time of the event.

    You can also take this data from the NAR files (dump the NAR file to a CSV file) and import that intoa database if you wanted to keep an historical look at what you array is doing over time (or you could get one the Analyzer packages like EMC Monitoring and Reporting - I think there's a free trial for that package).

    https://support.emc.com/kb/12289

    glen

  • elitec8888

    11 Posts

    530

    0

    Posted December 11th, 2015 11:00

    Hi Glen, when EMC carried out the health check they did enable logging and ran a heat map and found the disks where fine (not pushed at all). Also when the CIFS access was really slow, EMC performed a copy test within the Data Mover and that was fast. So issue is slow access to the CIFS when there are many users but the disks are fine (in the stats connection wise there were approx 10k but been told by EMC thats fine). We also ran a wireshark trace and saw requests to to the SAN but the response was slow coming back. So we have the storage blaming networks and networks blaming storage .

    Thanks for the link.