UNSOLVED

deeppat

updated

14 years ago

D

deeppat

2 Intern

•

261 Posts

0

1583

October 31st, 2012 11:00

Rsync incremental copies

Hello Guys ! Need some Advise .

Can I  tune  my rsyncs so that I get the incremental runs to complete fast ? currently it is taking 14 hours to complete  for one of the NFS

Migrating  ~ 2 TB of data 'from Netapp NFS to VNX

At present this is what I am using =

rsync -av --delete --exclude=.snapshot --exclude=.etc --exclude=lost+found --exclude=.ckpt /netapp/syslog/ /vnx/syslog/ > syslog_rsync.log&

14 hours is too long , i want to know if i can reduce the duration by any means ?

thank you in anticipation .

  • dynamox

    11 Legend

    •

    20419 Posts

    •

    87439 Points

    427

    0

    Posted October 31st, 2012 12:00

    By default rsync will try to transmit only the changed “blocks” in files, it takes time to calculate that ..i have fast network so I just tell it to go ahead and transfer the whole file if has changed, I don’t want you to spend time trying to figure out what specific blocks have changed. I find it helpful when I transfer a lot of small files, your mileage may vary.

  • deeppat

    2 Intern

    •

    261 Posts

    427

    0

    Posted October 31st, 2012 12:00

    thanks Dynamox  i will try it now , but I am not clear what  W will do ?  

  • deeppat

    2 Intern

    •

    261 Posts

    427

    0

    Posted October 31st, 2012 12:00

    thank you so much ! Now its much  clear  what i am trying to do .. Will get back to you with the result tomorrow

  • dynamox

    11 Legend

    •

    20419 Posts

    •

    87439 Points

    427

    0

    Posted October 31st, 2012 12:00

    try to add W  (ie: -avW). Other option is to run multiple rsync at the same time for individual subdirectories ..of course that depends on the underlying directory structure.

  • Rainer_EMC

    6 Operator

    •

    8645 Posts

    427

    0

    Posted October 31st, 2012 13:00

    You get to a point with older filers though that the src filer simply cannot deliver faster

  • deeppat

    2 Intern

    •

    261 Posts

    427

    0

    Posted November 7th, 2012 05:00

    thank you dynamox - My inc run time is ~ 9 hours now  when running the script with your reccomendation. I saved 5 hours of precious time .

  • dynamox

    11 Legend

    •

    20419 Posts

    •

    87439 Points

    427

    0

    Posted November 7th, 2012 14:00

    Thats significant.. good luck with migration.