Slow Replication between 2 VNXE over WAN

Hello,

Did you ever get an answer to this question or figure it out on your own?  We also have 2 VNXe3100 replicating of a WAN.  We have a 100Mb connection from Cogent on both ends.  I remember at one time we had speeds of 80MB per minute, now I'm down to 40-55MB a minute depending on other network traffic.  Cogent says their circuits are fine so I am wondereing if something changed on the VNXe side.  My former manager set this up originally so I do not know if he did something special.  My email is jholley@bruneandrichard.com if you want to respond directly.  Thanks!

You could open a support ticket with EMC and one of their techs will capture the reports.  Granted the replication session has to be running in order for them to capture the detailed files.  The tech will then forward to some other team which reads and interprets the logs and tell you where or why your speeds have died down.  In my case, the techs ascertained that perhaps the bursting of speed was causing something unstable and causing retransmissions of the packets hence slowing down our transfer rate.  I dont know if applying traffic shaping on the router/switch would help out or not.

I did not have further time to troubleshoot this, so I ended up using two replication IPs on different subnets (one for each SP) and had them replicating at the same time.  Not sure how you have SANS setup.

Hope this is useful.

Hello myzyr,

Have you been able to eventually get a satisfactory result? I too am having issues where the speed of the WAN replication 'hangs' at about 1/3 of the max bandwidth provided by the ISP. However when I copy from box to box (VNXe3300 to VNXe3300) using scp for example I am able to do that at maximum bandwidth. Which is 85mbps. Normal replication gets stuck at about 30mbps.

Kind regards

I have the exact same problem - but only for 2 of 7 datastores.  These datastore have e-mail data in them.  Are you copying the data over a VPN tunnel across the internet, or is the connectivity between the two SANs a dedicated point to point?  Mine is a VPN tunnel over the WAN.  My link is 10 Mb on my DR side and 20 Mb going to my production site. I've opened at least 5 tickets with EMC and they say that I need to be patient with replication, but the replication with the other 5 datastores worked fine.  I'm not sure if it is related to the e-mail data, or something else. I am investigating WAN Optimizer appliances as a fix as I'm not sure if it is going to get resolved.

Hello Dave,

I'm copying over an IP VPN (MPLS). It's used for several other purposes as well, but for 99% of the time the replication of the VNXe's is the only data that goes over the WAN. I've had this same problem with a previous ISP where we had a layer 2 VPN, which was a 75Mbps link. That also resulted in a transfer rate of about half of the maximum bandwidth. I had tested with scp from box to box and was able to transfer at 69Mbps or so, so that was fine. The replication however would always stuck at 30/35Mbps.

The problem now seems to be at the ISP, but they've asked me to do a test with the laptop directly connected to their router. I'm still waiting for measurement results from them. Perhaps this time the ISP is the only problem, but I'll have to wait or their next steps.

Regards

Hi all,

This continues to be an issue for us and since I am the sole IT person at our company, I do not have the time to do all the testing that my ISP and EMC are asking of me. Nor can I shut down our WAN for more then a couple hours either very early in the morning or

We have 2 VNXe3100's replicating over the WAN. We have 100Mbps Internet connections from Cogent on both ends. We have SonicWall firewalls. One office is in NYC the other SF. Our typical replication speed is between 50 and 60 MB/minute. For a few days in late September of this year our replication speed jumped up to 580MB/minute, which is almost our entire bandwidth. Cogent could not explain it, but they say their circuits are fine. Hearing that others are having a similar problem leads me to believe it's more of an EMC issue, but when I spoke to support they insists it's an ISP problem or something on our LAN -- typical vendor finger pointing.

My email is jholley@bruneandrichard.com if you want to respond directly.  Thanks!

Hey Mackenzie,

What tests have you done to ensure that your connections meet their specifications? With our previous ISP we were able to copy files at the maximum data-rate as specified by the provider. The replication of the VNXe's however was at half that rate.

What I've done to test the connection was:

- Copy a large file from VNXe to VNXe using scp. Shure you have some additional overhead because of the security, but it does give you a good idea of the communication between the two boxes

- Test communication between Windows-boxes on the two different sites with iperf. This gives a bit more 'clean' result

- Try ping with a packet size of 1500 with the 'no fragment' bit on. If it says that fragmentation is needed you loose efficiency in your communication between the sites

Also, what's the SLA on your internet connections? Is the 100Mbps a guaranteed bandwidth or is it 'best efford' like multiple companies (up to ten) use the same 100Mbps? That might also be a problem if you're sharing the ISP's bandwidth with some heavy users.

Yesterday I was able to test the IP VPN and result is that the configuration on the ISP's end is incorrect. This is now the reason why I can't get the specified datarate. With this ISP however the replication is transferring at the maximum rate I get at the moment, while with the previous ISP I could get a higher transfer rate using iperf or scp but the replication stuck at half of that.

Another thing that might cause problems is your firewall. Do you by any chance have a service on your firewalls that was deactivated in September? We have had some problems with a Trend Micro module in our ASA because we used more connections then allowed per license. Disabling this module resolved some major performance issues with our internet access.

I hope this is helpfull for you. Please let me know if you have any further questions.

Kind regards,

Arnoud

In my earlier post, I was unable to replicate my e-mail data store over the WAN.  I had been trying for over a year.  The EMC SAN GUI would say it was going to take 2,200 hours to replicate the data (That's 90 days if you are wondering).  I knew after a year of unsuccessful trying to replicate this data, I needed another approach.  I siad I was looking at WAN Optimizers.  Well I'm here to say that I was able to get my replication issue resolved.  I am now seeing speeds over 350 Mb/min over a 20 Mb VPN tunnel (using an ISP at each end) between my two VNXe-3100 SANs by installing a Riverbed Steelhead appliance.  The sppeds will jump to over 700 Mb/min when I'm replicating the "change data" (re-syncing after the intial sync).  These WAN accelerators are amazing devices!!!  The installation of each appliance takes about 30 minutes (you need to install two to work) and the device compresses the data across the pipe about 15 to 50 times, depending on if this is a first time sync or a periodic replication.  The EMC SAN was reporting 2,200 hours (~90 days) to replicate my 1.7 Tb data store across a 20 Mb VPN tunnel, but with the WAN accelerators, the time took only 100 hours (about 4 days).  The great thing is that Riverbed will lend you two devices to put in your environment to prove their worth before you have to plunk down a dime.  If you want to see my environment in action with a webex session, I’d be happy to show it to you.  Good luck!

Dave