Is there a backup API?

Did you try to use block based backup option?

No, we don't have a license for that.  It may be an option in the future.

PSS looks like the proper solution.  We're running 8.1 at the moment, but presumably we'll be moving to 8.2 in due course.  (Do you happen to know whether an 8.2 Windows client can do PSS with an 8.1 server?)

There are a lot of general guidelines you should follow:

  - Never use tapes for such a backup scenario. If the tape must reposition, it will cause extra delays.

  - Use a DD with DDBoost instead

  - Block based backups is an alternative                         but before you do anything else, may I suggest you

  - Give NW 8.2 and Parallel Saveset Streaming (PSS) a chance.

    With respect to this feature let me suggest that you also read this article:

       Parallel Save Streams for HDFS Performance Gains » NetWorker Information Hub

Do not forget that you can install NW on an extra host and it will run for 30 days in eval mode. The client can of course be backed up by 2 servers, assuming that the ..\nsr\res\servers file has been adjusted properly and the service has been restarted. Just be careful that you do not run the backups simultaneously.

ok - no idea when my response will be released as it is moderated (obviously due to a direct link). So i change it a little bit:

-----------

There are a lot of general guidelines you should follow:

  - Never use tapes for such a backup scenario. If the tape must reposition, it will cause extra delays.

  - Use a DD with DDBoost instead

  - Block based backups is an alternative                         but before you do anything else, may I suggest you

  - Give NW 8.2 and Parallel Saveset Streaming (PSS) a chance.

    With respect to this feature let me suggest that you go to this blog "nsrd.info/blog/" and read the blog dated 10/27.

Do not forget that you can install NW on an extra host and it will run for 30 days in eval mode. The client can of course be backed up by 2 servers, assuming that the ..\nsr\res\servers file has been adjusted properly and the service has been restarted. Just be careful that you do not run the backups simultaneously.

I am not sure. PSS is a client function and as each of the save sets are treated independently it might work. However I would not use it in production as compatibility is tested backwards, not forwards.

Thanks, but our Networker admin advises me that PSS probably won't help, because my server won't be able to keep the data flow high enough for Networker to decide to use multiple streams, and because we only have a limited number of streams available.

(Also: we're already using DDBoost.  The bottleneck is the speed with which the Networker client can read the contents of the disk, not on the network or the Networker servers.  We can't use image backup because we don't have a license for it.)

We do not know your system - not even the speed of your network adapter.

There can be so many issues that's why I suggested that you try it.

I recently 'played' with this feature on a test system with moderate file sizes (100kB).

This was not a serious benchmark test:

  - VMware workstation

  - All VMs running on a single disk

  - Backup device was a local AFTD disk

but my improvement was similar to Preston's (17 times faster).

Of course you could also try NW's good old rawasm for image backups.

So what do we argue at all?

Back to your original question "Is there a backup API?": I'm not aware of an API to NetWorker but save and recover command are able to read from and write to named pipes.

Just an idea...

hth.

The likely issue here is the amount of time it takes to traverse the filesystem and possibly the underlying disk storage for that filesystem.  If you're on RAID 5, read times will be horrible for the filesystem to read 2 or 3 seperate disks and checksum the file before it can fulfil the read request from the filesystem.  If you have RAID 0, it's just a straight read of the file and its metadata, and buffering can have the filesystem reading from both the original and the mirror disk depending on your raid firmware.

Block based backup would be your best option here.  Is a license really needed to use rawasm as part of your save command?  Setting up journalling on your filesystem should also help to provide NetWorker with just a list of changed files.

The filesystem lives on an iSCSI SAN.  RAID 6 at the SAN end, but that's handled in hardware and performance is more than adequate.  Seek times may be an issue, on my test server rearranging the data made a big difference.  On the other hand, I can pull the data off the production server (plain old robocopy, in parallel) at least three times faster than the backup, and I shouldn't be able to do that if the seek times were the bottleneck.

According to the documentation, rawasm is for UNIX only.  Journalling is already on, but it's the full backups that are a problem, and it can't help there.

Thanks for the advice everyone.  We've done some testing and concluded that moving the data to a new volume will produce enough of an improvement (2-3 times faster) to make the backups feasible again.  Presumably this is because it reorders the files on disk and/or in the MFT.  Presumably we'll need to repeat this process periodically, we'll have to wait and see how quickly the speed degrades.

We're also going to split the data into two separate volumes so they can back up in parallel.  Once we've upgraded to 8.2, we'll try PSS as well.