Celerra NX4 Poor Write Performance

Question

Hi Guys, I'm just looking for some non BS answers. I am a complete noob when it comes to SAN's in general. 12 months ago we implemented a virtual environment which included the NX4 implementation. EMC designed and installed the installation and we took all their reccomendations. The basic architecture goes like this. NX4 with dual datamovers and 10Gbe 12 x 600GB SAS 6 x 1TB SATA the 12 SAS drives were split into two raid 5 groups of five drives each with two global hotspares, the sata disks were put into 1 raid 5 group. The data movers are connected via 10Gb ethernet to HP6120 blade chassis switches inside an HP BL6000 blade chassis. All our hosts exist inside the blade chassis and therefore use the same 10GB switch, they include some Vmware ESXi servers and some Windows 2008R2 hosts. The ESXi hosts are accessing the datamovers via NFS and the windows machines are also accessing CIFS data. Our problem is generally very poor write performance. Our virtual environment for the most part works ok, anything write related appears to be slow but for the most part appears to work. On our physical windows boxes we are seeing some slow numbers for write performance. I must admit no one really complains accessing word docs etc but something doesn't feel right. I can read a 2GB iso from the cifs share to a windows host connected to the same blade switch as the NX4 at approx 190Mbps. If I copy the same file back to the share it copies at about 12MBps. I've tried other hosts in my environment connected to different switches etc and it is never any different, SMB1 or 2 makes no difference. It makes little difference whether I use the sata or sas drives share, it is always the same write performance. My suspicion although hard to measure is the virtual environment has similar performance because if i do the same copy test using the virtual machinesI get roughly the same performance as the physical hosts. I have contacted the support team and they checked it out remotely and basically said it is all fine. I am basically discounting the network as the problem as the read speed is pretty good, it seems like something inside the NX4 is the bottleneck. I am not using CAVA or deduplication. There are no errors reported on the switch ports. That write performance is utterly crap in my opinion, my laptop drive can write faster than that, a standard HP windows file server I installed recently could write at over 100MBps. I refuse to believe this NX4 should be that slow. One question I always had was EMC's decision to use clarsas_archive instead of clarsas_performance in the AVM profile, they said it made no difference, if that was the case why not use the one with the better name to put my concerns at ease, lol. Yes I know I might be missing something here but I'm just scratching my head wondering why it could be that slow. Could you please share your opinions and if possible the kind of performance numbers you have experienced. Thank You

mquayle · Answer

I just created a baby file system on the sas drives which won't have checkpointing on and the write speed is marginally better 18MBps but that is still pretty slow in my opinion. I don't know whether a file system's size has any bearing on performance.

dynamox · Answer

Can you test with a file system that does not have any checkpoints ?

Rainer_EMC · Answer

Just because the read speed is good doesnt mean the network is ok. There are Ethernet / IP problems that only show up in one direction.

Rainer_EMC · Answer

What OS release are you running ?

There were significant performance improvements for NFS write latency in the latest codes.

Also make sure you are using the uncached mount option for file systems hosting NFS data stores.

Rainer

mquayle · Answer

We are running Dart 6.0.40-5 and Flare 2.23.50.5.709,6.23.8 (0.13) Yes I am using uncached for the NFS mounts

mquayle · Answer

Fair point and I will continue to investigate but at this stage I wouldn't think the network is the most likely candidate, especially given we are only just maxing 100Mbps speed and the core fabric the celerra sits in is 10Gbps. I have checked all involved switch ports and the error count is 0, I have checked the tcp retransmission count on the celerra and it is so small you may as well say 0.

Open to any other suggestions for things to look at on the network.

Rainer_EMC · Answer

You want at least 6.0.41 to get the fix below.

If you cant upgrade immediately - the next 6.0.xx maintenance release is planned in a few weeks - beginning of December

Rainer

------

Impact Level

Severity 1

Symptom Desc.

User reported poor write response time with VMware ESX using Celerra through NFS with uncached file systems.

Fix

In this case, the Data Mover serialized the writes and sent the replies after the last write storage. Code

was changed to no longer serialize incoming NFS write requests.

Fixed in version

6.0.41.3

Rainer_EMC · Answer

You could run a ttcp from the client to the data mover and back Instructions should be here in forum Rainer

Rainer_EMC · Answer

BTW – 6.0.40.5 is over a year old now

96Chevyz71 · Answer

Check out this post. Others have the same issue. I removed my checkpoints and it helped but unfortunatley I can still to better on a standard Windows file server.

https://community.emc.com/message/560910.

Celerra

Was this post helpful?