Avamar Backup/Restore parallel threads

Hello Dan,

AFAIK this tweaks could be performed using command line using pre-scripts to involve "avtar" directly on client side, or to use configuration in MCS for different data sets which will trigger different parameters and definitions ...

The thing is that you could involve avtar to handle separate filecache and hashcache files for your multiple Data sets ...

Please be sure that such things could make performance impact on your client machine ...
Also please keep on your mind that  the fractal values in "--filecachemax"  &  "--hashcachemax" must be tuned properly ...

hope this helps ...

.r

Hi,

thank you for your answer !

I have a client with ~ 850 GB (large files) and the restore takes ~ 8h.

Could be possible to create 2 or 3 sessions with different content for restore, sessions that runs in parallel ?

Could be any performance improvement in this case ?

Is there any documentation for this scenario ?

The client is a relative powerful UNIX machine.

Thankk you !

Dan

1. oracle database files

2. So you suggest to take 2 backup sessions with avtar, this will generate 2 different backup label numbers and also restore these 2 backups in parallel with avtar.
But can these 2 sessions runs in parallel, for example started from crontab at 5 min. interval ?

Or, can I restore from the same backup (same sequence number) but with 2 distinct avtar commands for different subfolders ?

3.We have Avamar with RAIN configuration (3 nodes).

4.Network is Gigabit and client is UNIX AIX, pSeries, 16 CPU power5 @2 GHz, 32 GB RAM with external storage (SAN)

5.Ulimit is unlimited

Thank you!

Dan,

> I have a client with ~ 850 GB (large files) and the restore takes ~ 8h.
We are talking about file system (File server)  or DB backup / restore ?  Maybe NDMP ... ????
What exactly ?  Please give us more input ?

> Could be possible to create 2 or 3 sessions with different content for restore, sessions that runs in parallel ?
Same as backup, you can involve 2 avtar processes simultaneity .
to list all backups with specific label
      # avtar --backups --verbose --label=<"Label_Name"> --id= --path=

to list all backups for specific client:
      # avtar --backups --verbose --list --id= --path=

to initiate restore you want ....
       # avtar --extract --label=<"Label_Name"> --id= --path= --target=<"DESTINATION_TARGET_NAME"> <"FILE_YOU_WANT_TO_RESTORE">

> Could be any performance improvement in this case ?
of course!  same as backup. This could increase perf.inpact on Avamar server in 30% too.  

Do you have AVE, Avamar standalone or maybe multi node ?

> Is there any documentation for this scenario ?
huh, not sure.  This is special scenario and some tips and tweak must be customized .

> The client is a relative powerful UNIX machine.
What CPU is used on this Unix.  Can you provide us hardware topology ... network throughput ... etc ..
SPARC, PA-RISC are not optimized for SHA1 operation ... this could be also the reason .
For this tweaks you must consider "ulimit" parameters on OS level . 

rgds,
.r

Oracle DB ... ok, this is something different now .

You do not need two processes, but just to tune hash cache on client side ...

let me calculate for you ...  will be back ASAP

Here we are:

1. oracle database files
you are performing dump with RMAN or what .... ??

3.We have Avamar with RAIN configuration (3 nodes).
ok, gr8

4.Network is Gigabit and client is UNIX AIX, pSeries, 16 CPU power5 @2 GHz, 32 GB RAM with external storage (SAN)
ok. perfect!
In this case we need to tune hash cache on client side ...

The formula, then, is for Y GB of database data, the hash cache must be at least Y MB.  for instant:
Using this rule, for a database of 500 GB of database data, the hash cache must be at least 500 MB. The hash cache must be allowed to grow to the next incremental size, 768 MB.

If the client has a few very large files, the default maximum hash cache size may be insufficient. The hash cache almost always needs to be tuned for large databases as the default cache sizes are set for file system backups. You can override the default limits on the size of the hash cache by using the hashcachemax option with the avtar command.
To adequately size the hash cache, set the hashcachemax value as follows: --hashcachemax = 2 x Y where Y is the size of the database to be backed up in gigabytes

in your scenario, we are talking about 850 GB dump ... that means we need to allow hash cache minimally 1536MB or even double (3072MB).
So let us give a try a test backup using "--hashcachemax=-8" value.

OK, please create the "avtar.cmd" flag file to involve tweaks ...
This must be touched in Avamar's "var" directory ... by default should be  "touch /opt/AVMRclnt/var"  directory

now place different parameters on it and re-run the backup .

===========

--filecachemax=-16
--hashcachemax=-8

--stats

--comstats

--verbose=1

===========

please let me know results after test backups ... or maybe log file will be perfect to analyze ... 

Cheers,

.r

Dan,

one more thing ...
maybe you could also rename existing "p_cache.dat" to the "orig_p_cache.dat" and allow Avamar client to create new "p_cache.dat" file with next backup.

please let me know status update ASAP.
.r

ok, I will try these settings on the next backup window.

Thank you !

Dan

Hi Rej,

Thank you for these info !

1. No RMAN for the moment, just a cold backup, so just some big files

4. We are talking about ~85 big files, ~ 15 files x 35 GB each and other small files.

Does tuning hash cache help only for backup or also for restore ?

Can we use these attributes (--filecachemax=-16 ... etc.) on a graphical interface (More Options ... -> More ...) ?

Thank you !

Dan

Does this mean Avamar client will consider this to be the first backup and this will take longer ?

The first backup on this client takes ~8h and the second backup ~4h. We just took these 2 backups.

If we delete the p_cache.dat the backup will take longer than 4h ?

>  Does this mean Avamar client will consider this to be the first backup and this will take longer ?
Yes, in this case Avamar have to build "p_cache.dat" structure from scratch ... and for this first backup (when p_hash.dat does not exist) will take longer ...  but I suspect not longer than 8 hours (100 GB/hour)

> The first backup on this client takes ~8h and the second backup ~4h. We just took these 2 backups.
> If we delete the p_cache.dat the backup will take longer than 4h ?
Yes, for the first backup only  ... but all further backups should decrease backup time .

If you want leave it as it is. You can every time try with this deletion ...

regards,

Hi Rej,

Replying on your previous restore speed calculation, I've got a similar issue.

We run a Avamar setup where we backup virtual machines using the VMware vStorage API. The maximum size of a virtual machine is currently approximately 1.3TB; and we need to be able to restore this in less than 6 hours.

We run an Avamar grid of 3.3TB nodes, 9 storage nodes in total (to be upgraded to the max of 16 later this year). Using your sum of 40GB/node/hr, I'd expect close to 360GB/hr to be restored (9*40MB/s). However, I'm currently performing a test restore and I'm seeing a restore speed of ~60GB/hr. In total, meaning we'd need ~22hours for the 1.3TB VM, which is not acceptable.

We've got gigabit connectivity between the Avamar and the VMware clusters, so not really a bottleneck there. Any idea's? Are we doing something wrong, or is this expected for VMware Image backups (I'd better hope not!)?

Tnx,

Jon