I was hoping other users would jump in on this one (you might try asking at the Temple.edu Listserv as well to get some everyday user comments). This practice is certainly feasible. Being successful at it can depend on the amount of data you're trying to push across vs. your network pipes, etc. You'll have to mind your network connection to make sure that can handle the copy of data across.
This use case has been fairly common for the EMC Disk Library customers who have NetWorker. A NetWorker Storage Node sits on the VTL itself and clones out the back of the unit across the wire to another box at the remote location. There you can even cut tapes by running another clone (provided you have a Storage Node propped up there.) See attached for info on this solution.
Soon, with the Data Domain integration, that "clone" can be of dedupe bits only and it will make it much more efficient.
How much data do you need to get over to the other site? What kind of network do you have at your disposal for doing so?
I' ve a production data zone were daily the data came from client and are written from backup server over a Data Domain old DD460 appliance ( without vtl functionality ).
The same data that are written onto DD are daily replicated with the DD replication functionality ( withou clone ) to the remote site far less about 500 Km ( DR site ).
Daily the amount of backup are 1 Tb of not deduplicated data that came across the network after the DD have deduplicated.
It is important to permit a faster DR that are replicate in the remote site the Networker Resource ( throught bootstrap, or so ) and Index backup.
Depending on your comms costs (Australia is just disgusting!) and your restore requirements, then some form of de-dupe is the best way to go. You'll usually find that the comms costs and lack of human intervention pay for the technology.
Ask EMC for help, they have a model to prove the point and help you justify the purchase.
I had Networker on Windows originally, but it runs faster better on Linux and also lets me do what you are seeking with the tools that come with Linux. I use CentOS, a RHEL clone, it's free and works great, but if you need OS support, by all means get RHEL. When I contact Networker support, I just tell them I have RHEL. Actually, one support person commented to me they run CentOS at EMC, so you are closer to what they develop under actually if you have CentOS, even though they say RHEL is required, officially speaking. http://www.centos.org for more info.
My backup group is called Daily, my local disk backup pool is called Daily, my remote disk backup pool is called Remote, my remote Tape pool is called RemoteTape, and my local tape pool is called LocalTape (as far as you know). That said, I do this at the root prompt:
So now there are 4 copies, on local disk, local tape, remote disk, remote tape. I dispose of the local disk copies. The device name for my local daily is /a/Daily, volume label Daily.001, and rd=remote-node:/a/Remote for the remote site, volume label Remote.001, so this clears out the disk backups:
The "set -e" makes the script exit if any error occurs, otherwise the nsrmm commands will run and blow away your recent backups whether they got out to tape or not. Also, if run from cron (in the background) the path to the Networker images isn't there, so I just specify where they are by adding "/usr/sbin/" to the Networker commands.
The above is all fine and dandy, until I got into the multi-TB range of data, and our WAN got in the habbit of croaking.
So, just to let you know, you can use rsync to make your copies. You set up a device at the remote node for the Daily pool. What you are going to do, but over your WAN, is send a copy as if you carried tape to the remote site, only things is, you carried a disk, but you didn't do that either, you copied the disk. Trust me, this actually works.
savegrp -l incr Daily
nsrmm -u Daily.001
mv /a/Daily /a/Daily-transfer # so Networker can't find it while copying
until rsync -az -e ssh -progress --delete /a/Daily-transfer remote-node:/a; do sleep 10m; done
To make the above work, you do need to have ssh set up between the nodes using public/private keys with no passphrase for username root. When logged in as root you need this to work without asking for a password, otherwise cron can't run the above automatically for you:
ssh remote-node
I am sure you can google how to do that.
Using the "until" statement above with rsync, if rsync bombs due to a network error, it waits 10 minutes, and then tries again. It won't exit until it succeeds in making a good remote copy.
For savesets larger than 500 GB, I use the Linux "split" command to split the save set files into 300 MB files, I rsync over the files, and then use "cat" to put them back together. This way the rsync doesn't need to restart over at the beginning, but within 300 MB of your network failing.
Actually (now the truth comes out) I make a clone locally, start a second process running that moves Daily out to tape while the first is splitting it, rsync it, cats it back together, and sends it out to tape remotely.
Some nodes I back up to tape first, clone a copy back to disk locally, split that, rsync it, cat it, mount, send to remote tape. You can go crazy with this stuff. I've got TBs of data, rsyncs of multiple drives running, scripts that check for single copies that made it to tape before getting cloned, and all sorts of stuff. Networker can get crazy slow and time out sometimes, so I've even added this:
The reason is because a volume wouldn't mount some times, but I could later mount it just fine, so I just made the script mount and unmount and mount again and that seems to work whatever kinks Networker has in getting a volume to mount up some times.
I've also needed to add "lock" files, just in case 24 hours go by and the first copy to remote tapes hasn't made it out to tape yet. You can google how to make a lock file in bash, check for it, and exit so only one copy of a script runs at a time.
Anyway, this is some of what I do. YMMV depending on how well you can program.
One more tip, the variable $? holds the error code from the last command. If it isn't zero, the last command failed. I could show you my backup scripts, but they are about 6000 lines now. THey check for errors, take action to correct for errors, look for single copy savesets that got missed earlier, recopy them. Like I said, you can go crazy, just not in the Networker console. You are really limited there.
Another thing to know, my clone disk pools, some only have one volume. The nsrclone command won't send to a particular volume, just to a particular pool. To fix that, if a pool only has one volume, well, that is the one place I know where it will go.
ecarter11
2 Intern
•
177 Posts
1
June 2nd, 2010 14:00
Hi Dale,
I was hoping other users would jump in on this one (you might try asking at the Temple.edu Listserv as well to get some everyday user comments). This practice is certainly feasible. Being successful at it can depend on the amount of data you're trying to push across vs. your network pipes, etc. You'll have to mind your network connection to make sure that can handle the copy of data across.
This use case has been fairly common for the EMC Disk Library customers who have NetWorker. A NetWorker Storage Node sits on the VTL itself and clones out the back of the unit across the wire to another box at the remote location. There you can even cut tapes by running another clone (provided you have a Storage Node propped up there.) See attached for info on this solution.
Soon, with the Data Domain integration, that "clone" can be of dedupe bits only and it will make it much more efficient.
How much data do you need to get over to the other site? What kind of network do you have at your disposal for doing so?
Thanks for the post!
-Eric
1 Attachment
H2316_emc_clariion_disk_lbry_networker_wp_ldv.pdf
dksteinhauser
6 Posts
0
June 3rd, 2010 10:00
Eric,
Thanks a bunch for the info. It will at least get me moving in the right direction. (Now all I have to do is find the $$ to do it)
To answer your questions, Our daily backups average between 300 and 350 gig. The weeklys are around 850 gig. And monthlys are just over 1Tb.
Right now, we only have a 100Mb link to the remote site, however we can boost that to a full gig if needed.
So my main challenges are bandwidth and funding. So if anyone has ideas on how to do this on a shoestring, i'm all ears.
dloparco
2 Posts
0
June 13th, 2010 15:00
Hy Dale
I' ve a production data zone were daily the data came from client and are written from backup server over a Data Domain old DD460 appliance ( without vtl functionality ).
The same data that are written onto DD are daily replicated with the DD replication functionality ( withou clone ) to the remote site far less about 500 Km ( DR site ).
Daily the amount of backup are 1 Tb of not deduplicated data that came across the network after the DD have deduplicated.
It is important to permit a faster DR that are replicate in the remote site the Networker Resource ( throught bootstrap, or so ) and Index backup.
Sorry form my english and best regards
Best regards
Siobhan1
49 Posts
0
June 14th, 2010 17:00
Depending on your comms costs (Australia is just disgusting!) and your restore requirements, then some form of de-dupe is the best way to go. You'll usually find that the comms costs and lack of human intervention pay for the technology.
Ask EMC for help, they have a model to prove the point and help you justify the purchase.
Siobhan
Dan9714
1 Message
0
December 10th, 2011 13:00
I had Networker on Windows originally, but it runs faster better on Linux and also lets me do what you are seeking with the tools that come with Linux. I use CentOS, a RHEL clone, it's free and works great, but if you need OS support, by all means get RHEL. When I contact Networker support, I just tell them I have RHEL. Actually, one support person commented to me they run CentOS at EMC, so you are closer to what they develop under actually if you have CentOS, even though they say RHEL is required, officially speaking. http://www.centos.org for more info.
My backup group is called Daily, my local disk backup pool is called Daily, my remote disk backup pool is called Remote, my remote Tape pool is called RemoteTape, and my local tape pool is called LocalTape (as far as you know). That said, I do this at the root prompt:
savegrp -l incr Daily
mminfo -q pool=Daily -r ssid,cloneid -xc/ | grep -v ssid | nsrclone -b Remote -S -f -
mminfo -q pool=Remote -r ssid,cloneid -xc/ | grep -v ssid | nsrclone -b RemoteTape -S -f -
mminfo -q pool=Daily -r ssid,cloneid -xc/ | grep -v ssid | nsrclone -b LocalTape -S -f -
So now there are 4 copies, on local disk, local tape, remote disk, remote tape. I dispose of the local disk copies. The device name for my local daily is /a/Daily, volume label Daily.001, and rd=remote-node:/a/Remote for the remote site, volume label Remote.001, so this clears out the disk backups:
nsrmm -u Daily.001
nsrmm -l -f /a/Daily -y -b Daily Daily.001
nsrmm -m -f /a/Daily
nsrmm -u Remote.001
nsrmm -l -f rd=remote-node:/a/Remote -y -b Remote Remote.001
nsrmm -m -f rd=remote-node:/a/Remote
You can, of course, put the above in a file, tell bash to run it, and have cron schedule it, under username root. I don't use the Networker scheduler:
#!/bin/bash
set -e
/usr/sbin/savegrp -l incr Daily
/usr/sbin/mminfo -q pool=Daily -r ssid,cloneid -xc/ | grep -v ssid | /usr/sbin/nsrclone -b Remote -S -f -
/usr/sbin/mminfo -q pool=Remote -r ssid,cloneid -xc/ | grep -v ssid | /usr/sbin/nsrclone -b RemoteTape -S -f -
/usr/sbin/mminfo -q pool=Daily -r ssid,cloneid -xc/ | grep -v ssid | /usr/sbin/nsrclone -b LocalTape -S -f -
/usr/sbin/nsrmm -u Daily.001
/usr/sbin/nsrmm -l -f /a/Daily -y -b Daily Daily.001
/usr/sbin/nsrmm -m -f /a/Daily
/usr/sbin/nsrmm -u Remote.001
/usr/sbin/nsrmm -l -f rd=remote-node:/a/Remote -y -b Remote Remote.001
/usr/sbin/nsrmm -m -f rd=remote-node:/a/Remote
The "set -e" makes the script exit if any error occurs, otherwise the nsrmm commands will run and blow away your recent backups whether they got out to tape or not. Also, if run from cron (in the background) the path to the Networker images isn't there, so I just specify where they are by adding "/usr/sbin/" to the Networker commands.
The above is all fine and dandy, until I got into the multi-TB range of data, and our WAN got in the habbit of croaking.
So, just to let you know, you can use rsync to make your copies. You set up a device at the remote node for the Daily pool. What you are going to do, but over your WAN, is send a copy as if you carried tape to the remote site, only things is, you carried a disk, but you didn't do that either, you copied the disk. Trust me, this actually works.
savegrp -l incr Daily
nsrmm -u Daily.001
mv /a/Daily /a/Daily-transfer # so Networker can't find it while copying
until rsync -az -e ssh -progress --delete /a/Daily-transfer remote-node:/a; do sleep 10m; done
ssh remote-node "mv /a/Daily-transfer /a/Daily"
nsrmm -m -f rd=remote-node/a/Daily
mminfo -q pool=Daily -r ssid,cloneid -xc/ | grep -v ssid | nsrclone -b RemoteTape -S -f -
To make the above work, you do need to have ssh set up between the nodes using public/private keys with no passphrase for username root. When logged in as root you need this to work without asking for a password, otherwise cron can't run the above automatically for you:
ssh remote-node
I am sure you can google how to do that.
Using the "until" statement above with rsync, if rsync bombs due to a network error, it waits 10 minutes, and then tries again. It won't exit until it succeeds in making a good remote copy.
For savesets larger than 500 GB, I use the Linux "split" command to split the save set files into 300 MB files, I rsync over the files, and then use "cat" to put them back together. This way the rsync doesn't need to restart over at the beginning, but within 300 MB of your network failing.
Actually (now the truth comes out) I make a clone locally, start a second process running that moves Daily out to tape while the first is splitting it, rsync it, cats it back together, and sends it out to tape remotely.
Some nodes I back up to tape first, clone a copy back to disk locally, split that, rsync it, cat it, mount, send to remote tape. You can go crazy with this stuff. I've got TBs of data, rsyncs of multiple drives running, scripts that check for single copies that made it to tape before getting cloned, and all sorts of stuff. Networker can get crazy slow and time out sometimes, so I've even added this:
nsrmm -m -f /a/Daily; sleep 15; nsrmm -u Daily.001; sleep 15; nsrmm -m -f /a/Daily
The reason is because a volume wouldn't mount some times, but I could later mount it just fine, so I just made the script mount and unmount and mount again and that seems to work whatever kinks Networker has in getting a volume to mount up some times.
I've also needed to add "lock" files, just in case 24 hours go by and the first copy to remote tapes hasn't made it out to tape yet. You can google how to make a lock file in bash, check for it, and exit so only one copy of a script runs at a time.
Anyway, this is some of what I do. YMMV depending on how well you can program.
One more tip, the variable $? holds the error code from the last command. If it isn't zero, the last command failed. I could show you my backup scripts, but they are about 6000 lines now. THey check for errors, take action to correct for errors, look for single copy savesets that got missed earlier, recopy them. Like I said, you can go crazy, just not in the Networker console. You are really limited there.
Another thing to know, my clone disk pools, some only have one volume. The nsrclone command won't send to a particular volume, just to a particular pool. To fix that, if a pool only has one volume, well, that is the one place I know where it will go.
Regards.