This post is more than 5 years old
24 Posts
0
13836
Recovering rman data from Networker
Hello,
I need to recover dropped partition from rman backup (Oracle 10). The last level 0 backup that contains this partition was performed on 09/27/15. I'm using EMC networker for bakcup.
There was no catalog database and now backups from 09/27/15 are no longer visible in RMAN, because control_file_record_keep_time was set too low.
However some of those backups are still on tape - they are visible in mminfo results:
/usr/sbin/nsr/mminfo -s bdlbkp -o t -r "ssid,name,totalsize,savetime,volume,state" -q "client=sdbkp"
...
5342625141 RMAN:/home/scripts/sddb_backup.rcv 327 GB 09/27/15 SD72LT E
5264665370 RMAN:/home/scripts/sddb_backup.rcv 122 GB 09/27/15 SD72LT E
5315742731 RMAN:/home/scripts/sddb_backup.rcv 124 GB 09/27/15 SD72LT E
5512210329 RMAN:/home/scripts/sddb_backup.rcv 115 GB 09/27/15 SD72LT E
...
The saveset 5342625141 corresponds to backupiece containing tablespace with dropped partition (results of "list backup" recorderd at the time those backups were still visible in RMAN - now RMAN does not see them):
BS Key Type LV Size Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ ---------------
124134 Incr 0 312.59G SBT_TAPE 01:39:23 27-SEP-15
BP Key: 182503 Status: AVAILABLE Compressed: NO Tag: TAG20150927T0
Handle: db_SDDB_214513_948961110 Media:
List of Datafiles in backup set 121545
File LV Type Ckp SCN Ckp Time Name
---- -- ---- ---------- --------- ----
...
When I tried to recover backup directly by SSID from tape to disk (in order to register copy on disk in RMAN catalog) I got the following error:
# /usr/sbin/nsr/recover -s bdlbkp -S 5242625441 -d /tmp
Total estimated disk space needed for recover is 327 GB.
Recovering 1 files within RMAN:\home\scripts\ into /tmp
Permission denied by server bdlbkp : Cross platform recovery not supported
Both backup server and destination machine have Solaris.
Then I tried to register backup in RMAN:
RMAN> catalog device type 'SBT_TAPE' backuppiece 'db_SDDB_214513_948961110';
I got the following error:
allocated channel: ORA_SBT_TAPE_1
channel ORA_SBT_TAPE_1: sid=163 devtype=SBT_TAPE
channel ORA_SBT_TAPE_1: NMO v4.2.0.0
allocated channel: ORA_SBT_TAPE_2
channel ORA_SBT_TAPE_2: sid=47 devtype=SBT_TAPE
channel ORA_SBT_TAPE_2: NMO v4.2.0.0
allocated channel: ORA_SBT_TAPE_3
channel ORA_SBT_TAPE_3: sid=45 devtype=SBT_TAPE
channel ORA_SBT_TAPE_3: NMO v4.2.0.0
allocated channel: ORA_SBT_TAPE_4
channel ORA_SBT_TAPE_4: sid=52 devtype=SBT_TAPE
channel ORA_SBT_TAPE_4: NMO v4.2.0.0
allocated channel: ORA_SBT_TAPE_5
channel ORA_SBT_TAPE_5: sid=182 devtype=SBT_TAPE
channel ORA_SBT_TAPE_5: NMO v4.2.0.0
allocated channel: ORA_SBT_TAPE_6
channel ORA_SBT_TAPE_6: sid=183 devtype=SBT_TAPE
channel ORA_SBT_TAPE_6: NMO v4.2.0.0
ORA-19870: error reading backup piece db_SDDB_214513_948961110
ORA-19507: failed to retrieve sequential file, handle="db_SDDB_214513_948961110", parms=""
ORA-27029: skgfrtrv: sbtrestore returned error
ORA-19511: Error received from media manager layer, error text:
Could not locate the LNM save file 'db_SDDB_214513_948961110' on server 'bdlbkp
'. (2:9:0)
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of catalog command at 01/20/2016 13:35:06
RMAN-06209: List of failed objects
RMAN-06211: ==========================
RMAN-06212: Object Type Filename/Handle
RMAN-06213: --------------- ---------------------------------------------------
RMAN-06214: Backup Piece db_SDDB_214513_948961110
Another approach I tried was restoring older controlfile, that still contained records of those backups.
When I tried to restore controlfile that still contained records of backups I got:
run {
set until time "to_date('2015-11-27 00:00:00','yyyy-mm-dd hh24:mi:ss')";
allocate channel c0_db device type sbt
parms="ENV=(NSR_SERVER=bdlbkp, NSR_CLIENT=sdbkp)";
allocate channel c1_db device type sbt
parms="ENV=(NSR_SERVER=bdlbkp, NSR_CLIENT=sdbkp)";
restore controlfile to '/tmp/cfile';
}
executing command: SET until clause
allocated channel: c0_db
channel c0_db: sid=130 devtype=SBT_TAPE
channel c0_db: NMO v4.2.0.0
allocated channel: c1_db
channel c1_db: sid=55 devtype=SBT_TAPE
channel c1_db: NMO v4.2.0.0
Starting restore at 20-JAN-16
channel c0_db: starting datafile backupset restore
channel c0_db: restoring control file
output filename=/tmp/cfile
channel c0_db: reading from backup piece c-2315614121-20151102-02
ORA-19870: error reading backup piece c-2315614121-20151102-02
ORA-19507: failed to retrieve sequential file, handle="c-2315614121-20151102-02", parms=""
ORA-27029: skgfrtrv: sbtrestore returned error
ORA-19511: Error received from media manager layer, error text:
Could not locate the LNM save file 'c-2315614121-20151102-02' on server 'bdlbkp
'. (2:9:0)
failover to previous backup
channel c0_db: starting datafile backupset restore
channel c0_db: restoring control file
output filename=/tmp/cfile
channel c0_db: reading from backup piece c-2315614121-20151102-01
ORA-19870: error reading backup piece c-2315614121-20151102-01
ORA-19507: failed to retrieve sequential file, handle="c-2315614121-20151102-01", parms=""
ORA-27029: skgfrtrv: sbtrestore returned error
ORA-19511: Error received from media manager layer, error text:
A backup or restore session is already in progress. (100:17:0)
failover to previous backup
... // ("a backup or restore session is already in progress" error for all previous backups)
released channel: c0_db
released channel: c1_db
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of restore command at 01/20/2016 14:45:10
RMAN-06026: some targets not found - aborting restore
RMAN-06101: no channel to restore a backup or copy of the control file
Even though controlfile backup corresponding to "c-2315614121-20151102-02" is visible in mminfo results:
3463255223 RMAN:/home/scripts/sddb_backup_lvl1.rcv 20 MB 11/02/15 SD61LT
Mminfo shows however that there are no savesets on tape for 03/11/15-11/12/15, even though they are visible in RMAN "list backup". Maybe that is the reason?
Restore for controlfile from 20/12/15 works.
I would be grateful for any hints or ideas what can I do at this point.
ble1
2 Intern
2 Intern
•
14.3K Posts
0
January 28th, 2016 01:00
Catalog can be populated via rman - your DBA should know that. Data stream by RMAN is encapsulated and therefore only RMAN can read it (well, it can understand how to handle the stream).
If you want to restore something which is in catalog (or not), but expired on NW side, you must extend b/r policy, restore index and you are fine from there - that is correct.
I think it is essential to to have both catalog and NW retention synced. In the past we used to use cleanup of catalog for stuff older than retention on backup server side, but we stopped with that since growth and impact are no so visible and it is good archive to keep audit data as well.
crazyrov
4 Operator
4 Operator
•
1.3K Posts
1
January 20th, 2016 21:00
Have you restored the index files for the backups that you are trying to recover ?
Since this is backed up using NMO/NMDA you will not be able to restore the savesets usinng saveset recovery.
I suggest you restore the data using index, ensure that the file that backup file that you are trying to access is present in the backups. You could confirm this using nsrorainfo - if you are using NMDA or incase you had backed this up using NMO use nsrinfo to get the information on the files that were backed up under this saveset.
ble1
2 Intern
2 Intern
•
14.3K Posts
0
January 21st, 2016 01:00
From what I see you have E state so I assume all backup pieces are expired. You need to bring them back to be available for restore meaning extend browse and retention and then restore index before you can move on to RMAN restore.
ble1
2 Intern
2 Intern
•
14.3K Posts
0
January 26th, 2016 04:00
What is your-f261187dc7? Do you use DHCP? If not, check DNS/hosts to make sure name resolution is in place.
tolejniczak
24 Posts
0
January 26th, 2016 04:00
Thanks for your responses.
As far as I know I have to mark saveset as notrecyclable before I can extend browse/retention times. When I tried to do that I got an error:
# /usr/sbin/nsr/nsrmm -S 1237518202/1704121005 -s bdlbackup -o notrecyclable
Mark save set copy 1237518202/1704121005 as not recyclable? y
Permission denied, user 'root' on 'sd1-bak' does not have 'Operate NetWorker' privilege to perform this operation.
The same error occurs for nsrinfo.
When trying to give administrator privileges to root@sd1-bak the following error occurs:
root@bdlbackup # nsraddadmin -u root@sd1-bak
39078:nsraddadmin: RAP error: Host name your-f261187dc7 is not valid
What does it mean?
tolejniczak
24 Posts
0
January 27th, 2016 03:00
In meanwhile I wanted to try recovery procedure on more current saveset.
nsrinfo -vV -s bdlbkp sdbkp
scanning client `sdbkp' for all savetimes from the backup namespace on server bdlbkp
0 objects found
nsrinfo -vV -n oracle -s bdlbkp sdbkp
scanning client `sdbkp' for all savetimes from the backup namespace on server bdlbkp
UNIX file `c-2891241567-20160126-03' size=55838552, off=0, app=oracle(25), date=1453810756 Tue Jan 26 13:19:16 2016, (unknown fid), file size = 0
...
scanning client `sdbkp' for all savetimes from the backup namespace on server bdlbkp
UNIX file `db_SDDB_213567_802561214' size=262328, off=0, app=oracle(25), date=1453426280 Fri Jan 22 05:18:00 2016, (unknown fid), file size = 0
...
Is seems RMAN backups are in index of oracle namespace.
Now when I try to do:
/usr/sbin/nsr/recover -s bdlbkp -x oracle
I got:
39584:recover: No files have been backed up for this client. No index was found on the server for the path / on this client.
I tried also:
/usr/sbin/nsr/recover -a -s bdlbkp -x oracle -d /tmp db_SDDB_213567_802561214
Nothing to recover
When I do:
/usr/sbin/nsr/recover -s bdlbkp
It works:
Current working directory is /
recover> ls
(it prints contents of / directory)
How to extract this RMAN backup from tape?
ble1
2 Intern
2 Intern
•
14.3K Posts
1
January 27th, 2016 07:00
RMAN backup can only be restored by RMAN restore. If you never did it, you are not DBA. In that case I suggest you talk to DBAs - they can easily make RMAN restore script and run it.
tolejniczak
24 Posts
0
January 27th, 2016 08:00
That's the problem - RMAN is no longer aware about those backups existence.
An Oracle guy in my company suggested restoring them from tape to disk and then registering their copies on disk in RMAN catalog. That would allow RMAN restore from data on disk.
As far as I understand now:
* There is no way to do that (extract RMAN data from tape to disk without using RMAN, i.e. with networker commands alone).
* I have to extend browse/retention date of those backups, change their state to notrecyclable, and then restore online file index for those backups (by nsrck -L7 -t ...) . Then I can try to register them in RMAN catalog and perform RMAN recovery from tape. If that doesn't work there is nothing more I can do.
Is that correct?
edino1
39 Posts
0
January 27th, 2016 23:00
Hi,
it is not possible to extend browse or retention policy of expired SSID. You have to load tape with needed SSID to drive and use command scanner for rebuild of media DB entries and indexes . See more details in man page of scanner command...
You will need to restore control file, which contains data about your needed backups. I hope that you have enabled autobackup of control file in your backup RMAN script.
Lucky85
173 Posts
0
January 28th, 2016 02:00
Hello this is not true of course, you can do it always… Not sure with exact steps, but first of all you have to extend retention time, than set SS to recoverable state, than extend browse time??, than run nsrck –L7 which will reset all extended SS to browsable state. You always has to work with ssid/cloneid format when working with SS.
nsrmm –e 05302016 –S SSID/CLONEID – extend retention time
nsrmm –o notrecyclable –S SSID/cloneid
nsrmm –w 05302016 –S SSID/CLONEID – extend browse time – not sure if this step is possible
nsrck –t 30112015 –L7 client name – this has to be date close after index backup for backup you want to restore it will add old backup to you index databes – it is save…
regards
Lukas Slabihoudek
+420773748000
tolejniczak
24 Posts
0
January 28th, 2016 02:00
One more question then: is level 7 nsrck 'safe'? I.e. if I run it the current content of index will not be erased, modified or overwritten? (That's how I understand the documentation: 'Note that it will not overwrite existing files in the client file index', 'Note that recovering the index to a specific time adds the entire contents of the index as of that time to the current index contents'.)
Thanks for your help anyway.
ble1
2 Intern
2 Intern
•
14.3K Posts
0
January 28th, 2016 05:00
It is safe. Records are merged with existing ones. If you check structure under index, you will notice under db6 structure that there is folder for each each day and is each folder then contains specific data for backup sets of that day.
tolejniczak
24 Posts
0
January 28th, 2016 06:00
OK, I managed to restore index, but I don't know how to make savesets browsable.
For each saveset I want to recover I did:
nsrmm -S 4513442715/1441523008 -s bdlbkp -e "03/03/2016"
nsrmm -S 4513442715/1441523008 -s bdlbkp -o notrecyclable
Then I restored index backup the same way.
Then I did:
nsrck -L7 -t 1441537800 sdbkp
It seems the index was successfully restored:
nsrinfo -vV -n all -s bdlbkp sdbkp
...
UNIX file `c-2254432831-20150906-01', size=21496640, off=0, app=oracle(25), date=1441523007 Sun Sep 06 09:03:27 2015, (unknown fid), file size=0
...
But the savesets from 6th september are still not browsable:
...
RS56LT sdbkp 09/06/15 09:03:27 20 MB 4513442715 cr full RMAN:/home/scripts/sddb_backup.rcv
...
When I tried:
nsrmm -S 4513442715 -s bdlbkp -w "03/02/2016"
I got:
Cannot exceed the browse time of a recoverable save set 4513442715.
How to extend the browse time?
ble1
2 Intern
2 Intern
•
14.3K Posts
0
January 28th, 2016 07:00
You can always check status of ssid via
mminfo -avot -q ssid= -r name,group,client,savetime,ssbrowse,ssretent,ssflags,sumflag
in your case I do not see what mminfo shows for ssretent value now. I see that second value is not the same. So, so far you did something similar to what I did here:
# mminfo -q family=tape -q ssrecycle -r ssid
569194131
3659836964
4072470046
# mminfo -q ssid=569194131 -r savetime,ssbrowse,ssretent,ssflags,sumflags,cloneid
date browse retent ssflags fl clone id
09/07/15 12/07/15 12/07/15 vrEF cE 1441609363
# nsrmm -o notrecyclable -y -S 569194131/1441609363
# nsrmm -e 01/29/16 -S 569194131/1441609363
If this saveset has dependents with longer retention periods, the dependent savesets will not be browsable after the retention policy expiration date `Fri Jan 29 00:00:00 2016'.
Are you sure you want to change the retention policy expiration of this saveset to `Fri Jan 29 00:00:00 2016'? (yes/no) [no] yes
# mminfo -q ssid=569194131 -r savetime,ssbrowse,ssretent,ssflags,sumflags
date browse retent ssflags fl
09/07/15 12/07/15 01/29/16 vrF cr
Now, nsrck -L7 will apply original policy to current date so I would suggest to set ssretent to some higher value (you can always get that adjusted later). In my case I can't do nsrck -L7 because I do not have indices from this ssid and the only reason why I have it in mdb is because volume hasn't been relabeled yet.
if this won't get you anyway, and you are in rush and have no time to have someone take closer look, you can always removed ssid from mdb if this is on tape (take mminfo for it before including position on tape) and then run scanner on it. scanner will ask you about position and ssid and will bring back record as if policy has been applied today. But in general, this will take much longer.
tolejniczak
24 Posts
0
January 28th, 2016 07:00
I'm not sure what you mean by 'nsrck -L7 will apply original policy to current date so I would suggest to set ssretent to some higher value'.
ssretent is set to 3rd March (it was set before nsrck):
name group client date browse retent ssflags fl
RMAN:/home/scripts/sddb_backup.rcv sdora_sddb sdbkp 09/06/15 10/18/15 03/03/16 vrF cr
nsrck did not change browse date - originally this ssid looked like that:
ssid clone id name total date volume state retent browse
4513442715 1441523008 RMAN:/home/scripts/sddb_backup.rcv 20 MB 09/06/15 RS56LT E 11/29/15 10/18/15