Start a Conversation

This post is more than 5 years old

Solved!

Go to Solution

13617

January 20th, 2016 08:00

Recovering rman data from Networker

Hello,

I need to recover dropped partition from rman backup (Oracle 10). The last level 0 backup that contains this partition was performed on 09/27/15. I'm using EMC networker for bakcup.

There was no catalog database and now backups from 09/27/15 are no longer visible in RMAN, because control_file_record_keep_time was set too low.

However some of those backups are still on tape - they are visible in mminfo results:

/usr/sbin/nsr/mminfo -s bdlbkp -o t -r "ssid,name,totalsize,savetime,volume,state" -q "client=sdbkp"

...

5342625141 RMAN:/home/scripts/sddb_backup.rcv 327 GB 09/27/15 SD72LT E

5264665370 RMAN:/home/scripts/sddb_backup.rcv 122 GB 09/27/15 SD72LT E

5315742731 RMAN:/home/scripts/sddb_backup.rcv 124 GB 09/27/15 SD72LT E

5512210329 RMAN:/home/scripts/sddb_backup.rcv 115 GB 09/27/15 SD72LT E

...

The saveset 5342625141 corresponds to backupiece containing tablespace with dropped partition (results of "list backup" recorderd at the time those backups were still visible in RMAN - now RMAN does not see them):

BS Key  Type LV Size       Device Type Elapsed Time Completion Time

------- ---- -- ---------- ----------- ------------ ---------------

124134  Incr 0  312.59G    SBT_TAPE    01:39:23     27-SEP-15     

        BP Key: 182503   Status: AVAILABLE  Compressed: NO  Tag: TAG20150927T0

        Handle: db_SDDB_214513_948961110   Media:

  List of Datafiles in backup set 121545

  File LV Type Ckp SCN    Ckp Time  Name

  ---- -- ---- ---------- --------- ----

...

When I tried to recover backup directly by SSID from tape to disk (in order to register copy on disk in RMAN catalog) I got the following error:

# /usr/sbin/nsr/recover -s bdlbkp -S 5242625441 -d /tmp

Total estimated disk space needed for recover is 327 GB.

Recovering 1 files within RMAN:\home\scripts\ into /tmp

Permission denied by server bdlbkp : Cross platform recovery not supported

Both backup server and destination machine have Solaris.

Then I tried to register backup in RMAN:

RMAN> catalog device type 'SBT_TAPE' backuppiece 'db_SDDB_214513_948961110';

I got the following error:

allocated channel: ORA_SBT_TAPE_1

channel ORA_SBT_TAPE_1: sid=163 devtype=SBT_TAPE

channel ORA_SBT_TAPE_1: NMO v4.2.0.0

allocated channel: ORA_SBT_TAPE_2

channel ORA_SBT_TAPE_2: sid=47 devtype=SBT_TAPE

channel ORA_SBT_TAPE_2: NMO v4.2.0.0

allocated channel: ORA_SBT_TAPE_3

channel ORA_SBT_TAPE_3: sid=45 devtype=SBT_TAPE

channel ORA_SBT_TAPE_3: NMO v4.2.0.0

allocated channel: ORA_SBT_TAPE_4

channel ORA_SBT_TAPE_4: sid=52 devtype=SBT_TAPE

channel ORA_SBT_TAPE_4: NMO v4.2.0.0

allocated channel: ORA_SBT_TAPE_5

channel ORA_SBT_TAPE_5: sid=182 devtype=SBT_TAPE

channel ORA_SBT_TAPE_5: NMO v4.2.0.0

allocated channel: ORA_SBT_TAPE_6

channel ORA_SBT_TAPE_6: sid=183 devtype=SBT_TAPE

channel ORA_SBT_TAPE_6: NMO v4.2.0.0

ORA-19870: error reading backup piece db_SDDB_214513_948961110

ORA-19507: failed to retrieve sequential file, handle="db_SDDB_214513_948961110", parms=""

ORA-27029: skgfrtrv: sbtrestore returned error

ORA-19511: Error received from media manager layer, error text:

   Could not locate the LNM save file 'db_SDDB_214513_948961110' on server 'bdlbkp

'. (2:9:0)

RMAN-00571: ===========================================================

RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============

RMAN-00571: ===========================================================

RMAN-03002: failure of catalog command at 01/20/2016 13:35:06

RMAN-06209: List of failed objects

RMAN-06211: ==========================

RMAN-06212:   Object Type   Filename/Handle

RMAN-06213: --------------- ---------------------------------------------------

RMAN-06214: Backup Piece    db_SDDB_214513_948961110

Another approach I tried was restoring older controlfile, that still contained records of those backups.

When I tried to restore controlfile that still contained records of backups I got:

run {

set until time "to_date('2015-11-27 00:00:00','yyyy-mm-dd hh24:mi:ss')";

allocate channel c0_db device type sbt

parms="ENV=(NSR_SERVER=bdlbkp, NSR_CLIENT=sdbkp)";

allocate channel c1_db device type sbt

parms="ENV=(NSR_SERVER=bdlbkp, NSR_CLIENT=sdbkp)";

restore controlfile to '/tmp/cfile';

}

executing command: SET until clause

allocated channel: c0_db

channel c0_db: sid=130 devtype=SBT_TAPE

channel c0_db: NMO v4.2.0.0

allocated channel: c1_db

channel c1_db: sid=55 devtype=SBT_TAPE

channel c1_db: NMO v4.2.0.0

Starting restore at 20-JAN-16

channel c0_db: starting datafile backupset restore

channel c0_db: restoring control file

output filename=/tmp/cfile

channel c0_db: reading from backup piece c-2315614121-20151102-02

ORA-19870: error reading backup piece c-2315614121-20151102-02

ORA-19507: failed to retrieve sequential file, handle="c-2315614121-20151102-02", parms=""

ORA-27029: skgfrtrv: sbtrestore returned error

ORA-19511: Error received from media manager layer, error text:

   Could not locate the LNM save file 'c-2315614121-20151102-02' on server 'bdlbkp

'. (2:9:0)

failover to previous backup

channel c0_db: starting datafile backupset restore

channel c0_db: restoring control file

output filename=/tmp/cfile

channel c0_db: reading from backup piece c-2315614121-20151102-01

ORA-19870: error reading backup piece c-2315614121-20151102-01

ORA-19507: failed to retrieve sequential file, handle="c-2315614121-20151102-01", parms=""

ORA-27029: skgfrtrv: sbtrestore returned error

ORA-19511: Error received from media manager layer, error text:

   A backup or restore session is already in progress. (100:17:0)

failover to previous backup

... // ("a backup or restore session is already in progress" error for all previous backups)

released channel: c0_db

released channel: c1_db

RMAN-00571: ===========================================================

RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============

RMAN-00571: ===========================================================

RMAN-03002: failure of restore command at 01/20/2016 14:45:10

RMAN-06026: some targets not found - aborting restore

RMAN-06101: no channel to restore a backup or copy of the control file

Even though controlfile backup corresponding to "c-2315614121-20151102-02" is visible in mminfo results:

3463255223 RMAN:/home/scripts/sddb_backup_lvl1.rcv 20 MB 11/02/15 SD61LT

Mminfo shows however that there are no savesets on tape for 03/11/15-11/12/15, even though they are visible in RMAN "list backup". Maybe that is the reason?

Restore for controlfile from 20/12/15 works.

I would be grateful for any hints or ideas what can I do at this point.

14.3K Posts

January 28th, 2016 01:00

Catalog can be populated via rman - your DBA should know that.  Data stream by RMAN is encapsulated and therefore only RMAN can read it (well, it can understand how to handle the stream).

If you want to restore something which is in catalog (or not), but expired on NW side, you must extend b/r policy, restore index and you are fine from there - that is correct.

I think it is essential to to have both catalog and NW retention synced.  In the past we used to use cleanup of catalog for stuff older than retention on backup server side, but we stopped with that since growth and impact are no so visible and it is good archive to keep audit data as well.

4 Operator

 • 

1.3K Posts

January 20th, 2016 21:00

Have you restored the index files for the backups that you are trying to recover ?

Since this is backed up using NMO/NMDA you will not be able to restore the savesets usinng saveset recovery.

I suggest you restore the data using index, ensure that the file that backup file that you are trying to access is present in the backups. You could confirm this using nsrorainfo - if you are using NMDA or incase you had backed this up using NMO use nsrinfo to get the information on the files that were backed up under this saveset.

14.3K Posts

January 21st, 2016 01:00

From what I see you have E state so I assume all backup pieces are expired.  You need to bring them back to be available for restore meaning extend browse and retention and then restore index before you can move on to RMAN restore.

14.3K Posts

January 26th, 2016 04:00

What is your-f261187dc7?  Do you use DHCP?  If not, check DNS/hosts to make sure name resolution is in place.

24 Posts

January 26th, 2016 04:00

Thanks for your responses.

As far as I know I have to mark saveset as notrecyclable before I can extend browse/retention times. When I tried to do that I got an error:

# /usr/sbin/nsr/nsrmm -S 1237518202/1704121005 -s bdlbackup -o notrecyclable

Mark save set copy 1237518202/1704121005 as not recyclable? y

Permission denied, user 'root' on 'sd1-bak' does not have 'Operate NetWorker' privilege to perform this operation.

The same error occurs for nsrinfo.

When trying to give administrator privileges to root@sd1-bak the following error occurs:

root@bdlbackup # nsraddadmin -u root@sd1-bak

39078:nsraddadmin: RAP error: Host name your-f261187dc7 is not valid

What does it mean?

24 Posts

January 27th, 2016 03:00

In meanwhile I wanted to try recovery procedure on more current saveset.

nsrinfo -vV -s bdlbkp sdbkp

scanning client `sdbkp' for all savetimes from the backup namespace on server bdlbkp

0 objects found

nsrinfo -vV -n oracle -s bdlbkp sdbkp

scanning client `sdbkp' for all savetimes from the backup namespace on server bdlbkp

UNIX file `c-2891241567-20160126-03' size=55838552, off=0, app=oracle(25), date=1453810756 Tue Jan 26 13:19:16 2016, (unknown fid), file size = 0

...

scanning client `sdbkp' for all savetimes from the backup namespace on server bdlbkp

UNIX file `db_SDDB_213567_802561214' size=262328, off=0, app=oracle(25), date=1453426280 Fri Jan 22 05:18:00 2016, (unknown fid), file size = 0

...

Is seems RMAN backups are in index of oracle namespace.

Now when I try to do:

/usr/sbin/nsr/recover -s bdlbkp -x oracle

I got:

39584:recover: No files have been backed up for this client. No index was found on the server for the path / on this client.

I tried also:

/usr/sbin/nsr/recover -a -s bdlbkp -x oracle -d /tmp db_SDDB_213567_802561214

Nothing to recover

When I do:

/usr/sbin/nsr/recover -s bdlbkp

It works:

Current working directory is /

recover> ls

(it prints contents of / directory)

How to extract this RMAN backup from tape?

14.3K Posts

January 27th, 2016 07:00

RMAN backup can only be restored by RMAN restore.  If you never did it, you are not DBA.  In that case I suggest you talk to DBAs - they can easily make RMAN restore script and run it.

24 Posts

January 27th, 2016 08:00

That's the problem - RMAN is no longer aware about those backups existence.

An Oracle guy in my company suggested restoring them from tape to disk and then registering their copies on disk in RMAN catalog. That would allow RMAN restore from data on disk.

As far as I understand now:

* There is no way to do that (extract RMAN data from tape to disk without using RMAN, i.e. with networker commands alone).

* I have to extend browse/retention date of those backups, change their state to notrecyclable, and then restore online file index for those backups (by nsrck -L7 -t ...) . Then I can try to register them in RMAN catalog and perform RMAN recovery from tape. If that doesn't work there is nothing more I can do.

Is that correct?

39 Posts

January 27th, 2016 23:00

Hi,

it is not possible to extend browse or retention policy of expired SSID. You have to load tape with needed SSID to drive and use command scanner for rebuild of media DB entries and indexes . See more details in man page of scanner command...

You will need to restore control file, which contains data about your needed backups. I hope that you have enabled autobackup of control file in your backup RMAN script.

173 Posts

January 28th, 2016 02:00

Hello this is not true of course, you can do it always… Not sure with exact steps, but first of all you have to extend retention time, than set SS to recoverable state, than extend browse time??, than run nsrck –L7 which will reset all extended SS to browsable state. You always has to work with ssid/cloneid format when working with SS.

nsrmm –e 05302016 –S SSID/CLONEID – extend retention time

nsrmm –o notrecyclable –S SSID/cloneid

nsrmm –w 05302016 –S SSID/CLONEID – extend browse time – not sure if this step is possible

nsrck –t 30112015 –L7 client name – this has to be date close after index backup for backup you want to restore it will add old backup to you index databes – it is save…

regards

Lukas Slabihoudek

+420773748000

24 Posts

January 28th, 2016 02:00

One more question then: is level 7 nsrck 'safe'? I.e. if I run it the current content of index will not be erased, modified or overwritten? (That's how I understand the documentation: 'Note that it will not overwrite existing files in the client file index', 'Note that recovering the index to a specific time adds the entire contents of the index as of that time to the current index contents'.)

Thanks for your help anyway.

14.3K Posts

January 28th, 2016 05:00

tolejniczak wrote:

One more question then: is level 7 nsrck 'safe'? I.e. if I run it the current content of index will not be erased, modified or overwritten? (That's how I understand the documentation: 'Note that it will not overwrite existing files in the client file index', 'Note that recovering the index to a specific time adds the entire contents of the index as of that time to the current index contents'.)

Thanks for your help anyway.

It is safe.  Records are merged with existing ones.  If you check structure under index, you will notice under db6 structure that there is folder for each each day and is each folder then contains specific data for backup sets of that day.

24 Posts

January 28th, 2016 06:00

OK, I managed to restore index, but I don't know how to make savesets browsable.

For each saveset I want to recover I did:

nsrmm -S 4513442715/1441523008 -s bdlbkp -e "03/03/2016"

nsrmm -S 4513442715/1441523008 -s bdlbkp -o notrecyclable

Then I restored index backup the same way.

Then I did:

nsrck -L7 -t 1441537800 sdbkp

It seems the index was successfully restored:

nsrinfo -vV -n all -s bdlbkp sdbkp

...

UNIX file `c-2254432831-20150906-01', size=21496640, off=0, app=oracle(25), date=1441523007 Sun Sep 06 09:03:27 2015, (unknown fid), file size=0

...

But the savesets from 6th september are still not browsable:

...

RS56LT        sdbkp        09/06/15 09:03:27  20 MB 4513442715 cr full RMAN:/home/scripts/sddb_backup.rcv

...

When I tried:

nsrmm -S 4513442715 -s bdlbkp -w "03/02/2016"

I got:

Cannot exceed the browse time of a recoverable save set 4513442715.

How to extend the browse time?

14.3K Posts

January 28th, 2016 07:00

You can always check status of ssid via

mminfo -avot -q ssid= -r name,group,client,savetime,ssbrowse,ssretent,ssflags,sumflag

in your case I do not see what mminfo shows for ssretent value now.  I see that second value is not the same.  So, so far you did something similar to what I did here:

# mminfo -q family=tape -q ssrecycle -r ssid

569194131

3659836964

4072470046

# mminfo -q ssid=569194131 -r savetime,ssbrowse,ssretent,ssflags,sumflags,cloneid

  date    browse   retent  ssflags fl   clone id

09/07/15 12/07/15 12/07/15 vrEF  cE  1441609363

# nsrmm -o notrecyclable -y -S 569194131/1441609363

# nsrmm -e 01/29/16 -S 569194131/1441609363

If this saveset has dependents with longer retention periods, the dependent savesets will not be browsable after the retention policy expiration date `Fri Jan 29 00:00:00 2016'.

Are you sure you want to change the retention policy expiration of this saveset to `Fri Jan 29 00:00:00 2016'? (yes/no)  [no] yes

# mminfo -q ssid=569194131 -r savetime,ssbrowse,ssretent,ssflags,sumflags

  date    browse   retent  ssflags fl

09/07/15 12/07/15 01/29/16 vrF   cr

Now, nsrck -L7 will apply original policy to current date so I would suggest to set ssretent to some higher value (you can always get that adjusted later).  In my case I can't do nsrck -L7 because I do not have indices from this ssid and the only reason why I have it in mdb is because volume hasn't been relabeled yet.

if this won't get you anyway, and you are in rush and have no time to have someone take closer look, you can always removed ssid from mdb if this is on tape (take mminfo for it before including position on tape) and then run scanner on it. scanner will ask you about position and ssid and will bring back record as if policy has been applied today.  But in general, this will take much longer.

24 Posts

January 28th, 2016 07:00

I'm not sure what you mean by 'nsrck -L7 will apply original policy to current date so I would suggest to set ssretent to some higher value'.

ssretent is set to 3rd March (it was set before nsrck):

name                          group          client       date    browse   retent  ssflags fl

RMAN:/home/scripts/sddb_backup.rcv sdora_sddb sdbkp 09/06/15 10/18/15 03/03/16 vrF cr

nsrck did not change browse date - originally this ssid looked like that:

ssid         clone id name                                total  date   volume         state   retent   browse

4513442715  1441523008 RMAN:/home/scripts/sddb_backup.rcv 20 MB 09/06/15 RS56LT E  11/29/15 10/18/15

No Events found!

Top