Letbetterp

updated

14 years ago

L

Letbetterp

2 Intern

•

196 Posts

0

3631

November 9th, 2012 13:00

Avamar Proxy 6.1 Datastore

Is it best to create a lun or luns for the proxies to live on, or does it matter?

We have 20 proxies 6.1 for image level backups.

Would it perform better with its own lun, instead of putting on luns with other vm's on it.

  • ionthegeek

    2 Intern

    •

    2049 Posts

    677

    0

    Posted November 9th, 2012 17:00

    If there are several VMs on the same LUN backing up at the same time, that could certainly cause drive contention. That is more likely to be the culprit than the I/O usage of the proxies themselves, in my mind. I won't rule out the proxies using a certain amount of I/O (as I said, I haven't tested it myself) but correlation (the proxy VMs are on the busy LUNs) doesn't imply causation (the proxy VMs are causing the LUNs to be busy).

    There is a point at which CBT reaches diminishing returns because the overhead from merging the CBT information outweighs the performance gain of backing up fewer sectors. This is generally a function of the change rate on the disks, rather than their size, however. It might be worthwihle to have somebody from the non-Windows client team take a look at the VM backup performance to see if you're reaching the point of diminishing returns.

    The snap itself will generally be very fast but the client software on the proxy has to read any sectors that have changed since the last backup out of the snap and that will generate quite a bit of I/O.

    I believe you can also set up stats counters for individual VMs in VMware. It would be interesting to see how much I/O the proxies themselves are actually using, though you might have to put the proxy VM on its own LUN to see whether the I/O is for the proxy's VMDK or the client's -- I'm not sure how granular the stat counters are. If you have the SAN space available, it certainly couldn't hurt to put the proxies on their own LUN(s).

  • Letbetterp

    2 Intern

    •

    196 Posts

    677

    0

    Posted November 9th, 2012 17:00

    That is my point if you have several vm's and proxies on the same lun trying to read (or snap) and write at the same time, doesn't that cause drive contention?

    So if you had lets say 2 dedicated luns for 20 proxies they would not try to read and write at the same time on same lun correct?

    I have acouple of large servers with cbt enabled that seem to take longer than should to finish.

    And in analizer on the array at the time of backups the luns that have proxies on them have higher utilization and latency.

    But the snap happens quick just throwing this out there, having to recreate proxies after SP1.

    Thanks for the reply

  • ionthegeek

    2 Intern

    •

    2049 Posts

    673

    1

    Posted November 9th, 2012 17:00

    In general, the performance of a client is going to be bottlenecked by the rate at which the Avamar client software can read the data from the client system's disks. In the case of VMware Image backups, the proxy mounts a snapshot of the target VM's disks to the proxy and the client software reads from that. This means that the bottleneck is likely to be the datastore where the client's VMDKs are stored rather than the datastore where the proxy's VMDKs are stored.

    I haven't tested this myself but based on the architecture design, I don't think you will get much of a performance boost (if any at all) from putting the proxy VMs on their own LUNs.

  • Letbetterp

    2 Intern

    •

    196 Posts

    677

    0

    Posted November 9th, 2012 18:00

    In vcenter the proxies at backup have red alerts of disk I/O usage, but I will setup stats counters and look at performance.

    With the vcenter plugins for the VNX5700 it has a EMC performance section i'll check that also.

    Thanks for the cbt diminishing returns lesson did not even think of that.

    The ESX luns are in the general pool with fast cache, fast vp, efd's, 15k's and some nl and data is going into the efd's.

    Thanks for quick responce

  • Letbetterp

    2 Intern

    •

    196 Posts

    677

    0

    Posted November 12th, 2012 18:00

    I have 10 of the redhat proxies upgraded to suse, out of 20.

    Not sure if it is the new os on proxy or moving it to its own lun. (I know should have done one at a time)

    But the problem is no longer there. And also utilization is down with no excessive I/O or alerts in vcenter.

    Wonder how many proxies I can get on a lun without degrading proxies performance.

    I was going to try two luns in the general pool for all 20 proxies’.

  • ionthegeek

    2 Intern

    •

    2049 Posts

    677

    1

    Posted November 16th, 2012 11:00

    Good to hear the problem is no longer happening. It would be interesting to find out how many proxies you are able to put on the LUN without degrading the performance.

  • Letbetterp

    2 Intern

    •

    196 Posts

    677

    0

    Posted November 16th, 2012 17:00

    So far there are 10 of 20 proxies on one lun and with a full load not doing much.

    avamarproxy10-20 are at the latest 6.1.

    Here is a example of analizer at full load.

    PerfOverview.gifLunPerfVcenter.gifVirtualDisk.gifPerformance Detail.gif

  • Letbetterp

    2 Intern

    •

    196 Posts

    677

    0

    Posted November 24th, 2012 19:00

    20ProxiesRunning.gifI setup 10 proxies on one lun and the same for the other 10.

    The backup jobs are completing faster than before.

    But I hit a big snag when recreating the proxies from 6.0.1.66 to 6.1.0.402. The proxy os has changed from rhel to suse, and it takes a reinstall. Before you could rename the proxies, but with suse you need the -proxy-1 at the end when you recreate them.

    I was in autopilot mode and followed the same way as always, some proxies would work and some would not. The ones that weren’t rebooted were working and the others were not, the application lost connection to the server and needed to be reregistered. The upgrade pdf shows to mount iso reboot and it will upgrade, but this is not so. I hope this saves someone from chasing their tail like I did. Some might need the same name or like a certain proxy name.

  • 677

    0

    Posted May 9th, 2013 05:00

    We are having this same issue with really high latency.  Did you also turn off CBT as well as put the proxies on their own LUN?

  • 94

    0

    Posted May 13th, 2013 05:00

    In my case, I put all proxies on a dedicated LUN and don't meet any problem.