2dam
3 Silver

Large amount of small files takes huge amount of time to backup

Jump to solution
Hi Guys,

we have a FS in the TB that takes days to backup.

The FS is made up of millions of small files, I was wondering if there is a way of speeding it up What would you guys reccomend?

Thanks
2dam
0 Kudos
1 Solution

Accepted Solutions
Siobhan1
3 Silver

Re: Large amount of small files takes huge amount of time to backup

Jump to solution
Only problem with SnapImage is if you wish to do a full disk, or full directory restore. You really need a VTL or Advanced File Type device with SnapImage.

View solution in original post

0 Kudos
7 Replies
2dam
3 Silver

Re: Large amount of small files takes huge amount of time to backup

Jump to solution
Version of NW we are using is 7.3.4 on Win 2003sp2
0 Kudos
benzino1
3 Argentum

Re: Large amount of small files takes huge amount of time to backup

Jump to solution
There is always a problem with backing up millions of files. You can try to backup raw devices (but then you have to recover the whole volume to recover single file). You can also try to divide backup source to send more than one stream:
let's say you have to backup "/large" (or E:\large), try to define more than one saveset ie.:
/large/dir1
/large/dir2 etc.
than you can start more than one stream of backup at the time.
DavidHampson
4 Beryllium

Re: Large amount of small files takes huge amount of time to backup

Jump to solution
This is a common problem with no easy solution - it is a result of Networker generating more data to track the files than the files themselves. You could look at doing a raw backup of the disk but that can be painful to get back. You could look at splitting the files up so that you backup a different part each night which does not speed things up but spreads out the time of the backup.
tlemons1
3 Argentum

Re: Large amount of small files takes huge amount of time to backup

Jump to solution
You could also consider the NetWorker PowerSnap suite, which can leverage snapshot capabilities.

tl
0 Kudos
salvoemc
3 Silver

Re: Large amount of small files takes huge amount of time to backup

Jump to solution
You can consider use Snapimage with NDMP Client. Work in a lot of customer with this problem. Can backup a block level and recover is a file level.
0 Kudos
Siobhan1
3 Silver

Re: Large amount of small files takes huge amount of time to backup

Jump to solution
Only problem with SnapImage is if you wish to do a full disk, or full directory restore. You really need a VTL or Advanced File Type device with SnapImage.

View solution in original post

0 Kudos
dugans1
3 Argentum

Re: Large amount of small files takes huge amount of time to backup

Jump to solution
Ok so i think i know what your hinting at here and if so i have seen maybe the same problem.

IS tyour file server connected to shared storage "SAN" on an ethernet network or a Fabric solution?

is your storage node the same server as the management server?

I have had a situation where backing up across the network from a client has had poor or slow backup speeds due to offload of data at the storage node.

You might want to check and see what the Network cad is doing on the storage node and if it is offloading to a fabric solution to stream to the tape library this could be your bottle neck.

We had this problem and i put in a 4 port nic and teamed it with load balancing to let it offload to the fabric to then stream to the tape drives.

Another question is are you backing up from one shared storage in one data centre to a tape library in a second data centre? if so i would be looking to shapshot the SAN and present the LUN directly to the storage node at the second DC is the infrastructure is correct you should not hit copper anywhere across the 2 dc's and have a full fabric connection to the tape drives.

Let me know.
0 Kudos