Celerra Performance problems - slow write fast read

If you have to have checkpoints, there is nothing you can do. Otherwise, delete all of your checkpoints. Also, if you are running celerra replicator, it also creates some hidden checkpoints which will slowdown performance somewhat as well. We finally gave up trying to do anything about this and are looking forward to leaving the celerra platform in a couple of years.

Paul Shane | Systems Administrator | paul.shane@milliman.com

Milliman | 1550 Liberty Ridge Drive, Suite 200 | Wayne, PA 19087-5572 | USA

Tel +1 610 975 8012 | Fax +1 610 687 4236 | Mobile +1 610 389 5088 | milliman.com

Are saying that with the latency of a CIFS connection from the UK to the US you notice an impact of checkpoints ?

If you are copying from the UK to a CIFS share in the US than most likely its not the checkpoint that is limiting your performance

I suggest to try to do the same to a Windows server over the same distance or copy to a VNX file system without checkpoints

Because of the way Celerra’s currently use copy-on-write checkpoints, write performance will drop significantly when you have checkpoints. It was an unadvertised “feature” of these units. They have announced changes in their latest VNX code to change the way they do checkpoints.

Paul Shane | Systems Administrator | paul.shane@milliman.com

Milliman | 1550 Liberty Ridge Drive, Suite 200 | Wayne, PA 19087-5572 | USA

Tel +1 610 975 8012 | Fax +1 610 687 4236 | Mobile +1 610 389 5088 | milliman.com

Thanks for your response.

Yes, we do have checkpoints for file systems.

We have two NAS boxes in our environmet....UK & US

1) When i capoy the data from UK citrix server to US nas share....it takes time.

Thanks,

So what i have to do now to get the better perfomance. Kindly help me on this.

We are using Celerra NS80g backend storage with Clariion CX3-80 and another one Celerra NS80g backend storage with Symmterix V-max.

For anyone who's interested a similar story. We have been struggling with EMC support for 2 years. The main solution that emc gave us, was to delete all 60 checkpoints and create new schedules for the filesystems. Since then we have followed a lot of steps to gain filesystem/system performance, because even at this point, it still isn't what we expected of it.

- Before: All, filesystems 5 - 7 MB/s write performance

- Filesystems with 60 checkpoints, 1 schedule, every day.

- Filesystems without checkpoints,  20 - 50 MB/s write performance

The steps taken:

- Divided a 6 TB filesystem into 6 smaller 1 TB filesystems, EMC recommends a max. size of 2 TB.

- Went from 60 checkpoints to 14 checkpoints (EMC NAS Expert said that writespeed issues can occur when using more than 16 checkpoints for one filesystem) especially when looking at filesystems that contain a lot of files constantly changing, for instance, profiledata etc.

- Updated the Dart Code twice. The current code is 6.0.41-3 (we are planning to update to 6.0.41-4).

- We have installed a retransmission tool, to calculate retransmits, as it seems that these are questionable, we are investigating our network. But we still suspect the Celerra not performing because of the amount of checkpoints.

- We have 2 sites, site 1 is our primary CIFS/NFS nas, site 2 is used for backup purposes: Site 2 has 1 checkpoint per filesystem, and has write speeds between 25 - 50 MB/s

- There is no difference in write speeds using NFS or CIFS

- Bought 2 extra datamovers to divide the workload (Also because of the maximum TB supported per Datamover)

- Our network specialists and EMC have combined their teams to see what causes the retransmits, EMC says the retransmits might be caused on the network, the networkspecialists say the retransmits are caused within the datamovers.... in the end, nobody knew where to seek for a sollution and now we also have accepted the slow writes.

retransmission_stats]$ ./retransmission_stats.sh server_5

#########################################################
# Consider the following table for recommended values : #
# Above .1%                ->    Too high               #
# Between .01% and .1%     ->    Questionable           #
# Below .01%               ->    OK                     #
#########################################################

07/30/12 11:39:40 packet_sent=1100533848 packets_retransmitted=164825596 rate=0.049772%
07/30/12 11:40:15 packet_sent=1100707990 packets_retransmitted=164825632 rate=0.020673%
07/30/12 11:40:51 packet_sent=1100813751 packets_retransmitted=164825709 rate=0.072806%
07/30/12 11:41:23 packet_sent=1100919356 packets_retransmitted=164825755 rate=0.043559%
07/30/12 11:42:10 packet_sent=1100994037 packets_retransmitted=164825805 rate=0.066951%

- On short notice we are planning to replace all network cables connected to our datamovers, to see if that makes a difference.

All write tests are done with a the same 1GB iso file.

After:  filesystems  8 - 15 MB/s write performance

- Filesystems with 14 checkpoints: 2 schedules 1 daily, 1 weekly

- Filesystems without checkpoints,  20 - 50 MB/s write performance

Does anyone know if it is possible to reset the "server_netstat server_x -i" error counters?

If you are experiencing poor CIFS write performance then please have a look at this thread. We suffered for almost 3 years with this issue and the fix suggested in this thread transformed the write performance.

https://community.emc.com/message/657620#657620

Regards

Paul

I think the big speed increase you got with fastRTO was specific to your network setup