Unsolved
This post is more than 5 years old
22 Posts
0
6488
Help with recover
Hi Gurus,
I am new to legato networker and I was assigned a task to recover some files. The client is using Solaris and the server is a Linux machine.
It is returning the following error as below when I am trying to do a recover for a client as one of the users has corrupted the data in databases for them.
We are running networker 7.3
When tried from server:
[root@server01 ~]# recover -c oracle02 -s server01
6550:recover: No browsable index found for client oracle02.
When tried from client side
bash-3.00# recover -c oracle02 -s server01
39584:recover: No files have been backed up for this client. No index was found on the server for the path / on this client.
Currently we are backing these three loation /, /backup1, /opt/oracle for client server
/backup1 is a mounted from netapp storage.
When I try below commands you specified for above partitions I am able to change to the required time for only / and rest of the two partition it throws the error.
From client machine (oracle02)
bash-3.00# pwd
/
bash-3.00# recover -s server01
Current working directory is /
recover> changetime 14-july-2012 10:00
6497:recover: time changed to Sat Jul 14 10:00:00 2012
recover> exit
bash-3.00# cd /backup1
bash-3.00# recover -s server01
4181:recover: Path /backup1 is within hmdnas02b:/vol/asbkup002T
53362:recover: Cannot start session with server server01: client `hmdnas02b' is not properly configured on the NetWorker Server or `hmdnas02b' is not in the aliases list for client `server01'
bash-3.00# cd /opt/oracle/
bash-3.00# recover -s server01
Current working directory is /opt/oracle/
recover> changetime 14-july-2012 10:00
6496:recover: cannot browse /opt/oracle/ as of `Sat Jul 14 10:00:00 2012'
I have check for nsrck for this specific client and it shows no indexes available as below
Server side:
[root@server01 ~]# nsrck -c server
nsrck: checking index for 'server01'
nsrck: /nsr/index/server01 contains 0 records occupying 0 KB
nsrck: Completed checking 1 client(s)
But every day in the save group completion I am able to see that this server is been backed up and got index files on server. Below is the backed up data from yesterday
[root@server01 ~]# mminfo |grep -i oracle02
A00024L4 oracle02 15/07/12 230 MB incr /
A00024L4 oracle02 15/07/12 40 GB incr /backup1
A00024L4 oracle02 15/07/12 7333 MB incr /opt/oracle
A00024L4 oracle02 16/07/12 7350 MB incr /opt/oracle
A00024L4 oracle02 16/07/12 40 GB incr /backup1
A00024L4 oracle02 16/07/12 11 MB incr /
Any idea or information regarding this would be really helpful.
Thanks,
Phani
masonb
445 Posts
0
July 16th, 2012 08:00
phani6039,
From the mminfo output the only backups in the Media database reported are from 15 and 16/7. You change time to 14/7?
What is the browse/retention period of the client backups?
Also which user are you logged in as on the client - I can see from the NetWorker server output you are in as root but not on the client?
Under the /nsr/index/oracle02 directory - if you do a long listing of this directory are there any files present?
Check your alias name resolution for oracle02 also to ensure all names and IP address can be resolved.
Hope this helps,
Regards,
Bill Mason
phani6039
22 Posts
0
July 16th, 2012 09:00
Thanks Bill for your response,
What is the browse/retention period of the client backups?
The browse and retention policy is 3 months and i have also got the tape in the Robot for 14th July.
Also which user are you logged in as on the client - I can see from the NetWorker server output you are in as root but not on the client?
I have logged in as root on the client as well.
Under the /nsr/index/oracle02 directory - if you do a long listing of this directory are there any files present?
On the server side i am able to see below output
[root@server01 ~]# cd /nsr/
[root@server01 nsr]# cd index/
[root@server01 index]# cd oracle02/
[root@server01 oracle02]# ll
total 8
drwx------ 3 root root 4096 Jul 16 16:13 db6
-rw-r--r-- 1 root root 970 Aug 4 2011 README
-rw------- 1 root root 0 Aug 4 2011 v6ck.lck
[root@server01 oracle02]# cd db6/
[root@server01 db6]# ll
total 12
drwxr-xr-x 2 root root 4096 Jul 16 16:13 tmprecov
-rw------- 1 root root 48 Jul 16 12:28 v6hdr
-rw------- 1 root root 0 Aug 4 2011 v6hdr.lck
-rw------- 1 root root 0 Jul 16 12:28 v6journal
-rw------- 1 root root 54 Jul 16 16:13 v6tmp.ptr
And on the client side i cant find the index folder anywhere.
Check your alias name resolution for oracle02 also to ensure all names and IP address can be resolved.
hmdnas02b:/vol/asbkup002T is a storage partition that is been mounted on to the client (oracle02) as /backup1 which gets backed up eveyday at mid night.
hmdnas02b:/vol/asbkup002T 100GB 55GB 45GB /backup1
I have check the alias names and found i have got only 2 names up there oracle02 oracle02.gas.com
No do we need to specify the storage ie hmdnas02b up there as well?
Now I have mounted this partition locally on to the server01 and tried this as below
[root@server01 ~]# cd /
[root@server01 /]# mkdir /backup1
[root@server01 /]# mount hmdnas02b:/vol/asbkup002T /backup1/
[root@server01 /]# cd backup1/
[root@server01 backup1]# cd prd/
Started the recovery as below after mounting the location
[root@server01 ecisgprd]# recover -c server01 -s server01
Current working directory is /backup1/prd/
recover> changetime 14-july-2012 10:00
6496:recover: cannot browse /backup1/prd/ as of `Sat 14 Jul 2012 10:00:00 CET'
recover> changetime 13-july-2012 10:00
6496:recover: cannot browse /backup1/prd/ as of `Fri 13 Jul 2012 10:00:00 CET'
Any idea if i am missing anything?
Thanks,
Phani
ble1
2 Intern
2 Intern
•
14.3K Posts
0
July 16th, 2012 17:00
Is that NFS mount? Of course it is. In that case you must be aware that there are some special rules when it comes to backup of NFS shares. Not that is is not possible, but by default it is not done unless you use switch where you say all mnt points will be seen as local file systems. You might use that since you did backup something already (unless under that mounting point you had some other data at the time of backup which would make nested mounting point and things more difficult). Now, your data is in mdb under name of client oracle2. One thing to do is following:
- go to oracle2
- do recover -s server01 (do this from root directory)
- do ls
- make sure you see folder you wish to restore, like /backup1
- run versions backup1
- do changetime
- cd into backup1
- cd into substructure and select file to restore
- restore it
If it complains about destination, set target destination to local folder and restore it there.
Obviously, you can do also ssid restore, but perhaps we shouldn't go there at this point.
You may find above strange, but with filers or anything else, you should use NDMP protocol to make backup. Some people like to avoid certain license fees so they mount they whole datacenter to backup server and backup everything under one client. Surely you can try to do that, but it comes with price tag unless you know what you are doing. Anyway, try above and if it doesn't work you should try to indetify ssid in which data is which you wish to recover and do ssid recover instead.
phani6039
22 Posts
0
July 16th, 2012 23:00
Hi Hrvoje,
You were right that it is a NFS mount. I have tried above steps
- logged into oracle2
- recover -s server01 (do this from root directory)
- ls
- cd /backup1
recover> ls
prd
recover> cd prd
recover> ls
archive export hot
recover> cd export
recover> ll
total 285096
284764 -rw-r--r-- oracle dba 91017625 Jul 16 22:58 x_20120716_2020.dmp.gz
108 -rw-r--r-- oracle dba 5100 Jul 16 22:58 x_20120716_2020.log
112 -rw-r--r-- oracle dba 6652 Jul 16 22:58 y_20120716.log
112 -rw-r--r-- oracle dba 6652 Jul 16 22:58 y_20120716_2020.log
recover> cd
At this point we need /backup1/prd/export directory that needs to be recovered on to a different location from last week on to Friday (initial request) I have checked the tapes and the Friday tapes that is used in is still in robot but when I run the versions command it is not showing up.
recover> pwd
/
recover> versions /backup1
Versions of `/backup1/':
4 drwxr-xr-x oracle dba 4096 Aug 4 2011 backup1/
save time: Tue Jul 17 01:31:46 2012
location: Unknown
4 drwxr-xr-x root root 4096 Jul 16 16:53 backup1/
save time: Mon Jul 16 21:00:02 2012
location: A00024L4 at /dev/nst0
recover> changetime 16-Jul-2012
6497:recover: time changed to Mon Jul 16 23:59:59 2012
recover> ll
recover> ls
recover> changetime 16-Jul-2012 21:00
6496:recover: cannot browse /backup1 as of `Mon Jul 16 21:00:00 2012'
recover> changetime 16-Jul-2012 21:00:02
6497:recover: time changed to Mon Jul 16 21:00:02 2012
recover> ll
recover> ls
Here I was able to see the backup1 on 17th July but I actually need the data from 13th July (Friday) which is not showing up here. I tried for 16th to see if I can get anything but I can’t find any files inside it at all.
If you check the mminfo output for below command which shows the full backup of oracle02 server on all the tapes.
[root@server01 ~]# mminfo -s server01 -v -q 'client=oracle02,name=/backup1,level=full' | more
volume client date time size fl lvl name
A00001L4 oracle02 27/04/12 01:32:28 75 GB tb full /backup1
A00024L4 oracle02 13/07/12 01:31:46 49 GB tb full /backup1
A00077L4 oracle02 13/01/12 01:33:22 69 GB 3 cE full /backup1
Removed rest of the tapes details as not needed.
My initial task is to get the data of /backup1/ecisgprd/export on to different locations for the below tapes. Got tape 24 in the robot itself
Once the above is done then need to get the data of same location from other date
2. A00001L4 oracle02 27/04/12 01:32:28 75 GB tb full /backup1
Then the third one for below which is expired
3. A00077L4 oracle02 13/01/12 01:33:22 69 GB cE full /backup1
Any idea if i am missing anything as not able to recover the data for above tapes
Thanks,
Phani
ble1
2 Intern
2 Intern
•
14.3K Posts
0
July 17th, 2012 11:00
Hi Phani,
The data from 17th says location is unknown - this may indicate that tape is either out of library of that index and mdb records are not connected. Later one can happen with NFS backups if you make saves without -L and -c options (top of my head, I would need to check manual to be 100% sure). Now, I think we can debate for days as to what happened and what needs to be changed, but I will skip that part as you need data back and get to plan B.
Plan B is - ssid restore. We do not read index but we read tape and we get data back - period. Now, you need data from Friday 13th (critics will say that is your problem). That backup is spanning two volumes:
A00024L4 oracle02 13/07/12 01:31:46 49 GB 1140812989 tb full /backup1
A00025L4 oracle02 13/07/12 01:31:46 32 GB 1140812989 hb full /backup1
Start or (h)ead is on 25 and end or (t)ail is on 24. In total some 81GB. Not sure how familiar you are with ssid restore, but it's simplest form is following:
recover -s -S 1140812989
Do not run this yet, but read on. Above command says talk to and start restoring ALL data from unique save set id . In above example I used ssid which represents data you want from Friday. What NW would do is to read whole thing and restore it back - some 81GB data. Now, you do not wish to get 81GB data back and perhaps even not at the same location. Location is easy - just used -d switch and say where it should go, for example:
recover -s -S 1140812989 -d /var/adm/crash
In example above, those 81GB will go instead of original location to /var/adm/crash. Of course, in your case, that might be something else like /tmp or whatever place you want with enough space. Which brings us to final topic - you want specific data - /backup1/ecisgrpd/export
To get that, at the end you just add path you wish to restore, for example:
recover -s -S 1140812989 -d /var/adm/crash /backup1/ecisgrpd/export
Do not run it yet, but read on Above will read ssid, dump everything matching file/folder specified into -d location. Now, to understand what is going on you must know that ssid restore reads data from tape - it does not care about index database at all. This means that ssid restore does not know where your data exactly is withing those 81GB so whole ssid will be read. What is restored is only what you requested (the best way to understand it is that this is sort of grep filter which is, as grep, case sensitive so make sure to type pattern correctly). This is your fast way out and method to restore it or simply plan B. To make sure plan A works next time, have someone with more experience check your config and make necessary changes and tests to make sure plan B does not happen again.
ble1
2 Intern
2 Intern
•
14.3K Posts
0
July 19th, 2012 11:00
Group is not important here - client is. More precisely, to what client does data belong to. By default NW knows this and does backup of local data only. You can make it backup NFS data of course, but then you need to make sure that data is saved under index name you want and that NW does traverse mnt point. So, in your case that would mean backup command in client resource should be save -L -c . To make is clean, use clientname of the machine where NFS share is mounted on. For vast majority of filers, you should consider NDMP based backup.
phani6039
22 Posts
0
July 19th, 2012 11:00
Hi Hrvoje,
Thanks for your detailed explanation. I am able to recover the data as such of now.
Just a quick question as why for this client alone the networker is failing to store the indexes when it is storing for all other clients which belong to the same group.
What i have noticed is that everyday it is backing up the client but it doesnt have the CFI stored any information what can cause these issues?
Once again thanks a lot for the explanation.
Thanks,
Phani
phani6039
22 Posts
0
July 19th, 2012 12:00
Thanks for quick reply,
Data belongs to oracle02 client for which NW backs up three locations / , /opt/oracle (both local) and /backup1 which is a NFS mount used by this client.
Indexes are not getting stored for any backed up locations for this client (oracle02).
[root@server01 ~]# nsrck -c oracle02
nsrck: checking index for 'oracle02'
nsrck: /nsr/index/oracle02 contains 0 records occupying 0 KB
nsrck: Completed checking 1 client(s)
I had a look at media > properties > configurations as well there store indexes are checked for this pool.
how could i find what command is it using for backing up every time as you were specifying save -L -c .
I havent configured this and it was configured before i joined here.
In order to mount the clientname (hmdnas02b) of the machine where NFS share is mount we havent got enough client licenses available at the moment.
I had a look at the properties of the client (oracle02) using GUI and they seems to be normal as that of other clients which are getting backed up.
May be i am being dumb up here, could explain it plz.
Thanks,
Phani
ble1
2 Intern
2 Intern
•
14.3K Posts
0
July 19th, 2012 12:00
Nope, it does not belong to oracle2 client. It belongs to client from where that NFS share is coming from. And NetWorker knows that so by default this data is not local to oracle2 and thus it does not get saved under oracle2 index.
Your nsrck shows that data gets saved somewhere else - actually it shows index db for oracle2 is empty which indicates there is some other trouble going on (since / and /opt/oracle are local). Now, there is a question why and there are several possible reason there (David has similar problem too, though I doubt you have same cause).
First, you use oracle2 as name for client. I will assume this is correct and I will also assume your DNS or/and /etc/hosts are correct (but you better double check). Go to backup server and check what do you see under /nsr/index/oracle2. More precisely, check if you have something which called nested index structure.
Second, make sure box really does resolve back to oracle2. Assumtion is mother of all so you better check that twice. There bunch of stories where and if name authetification have caused havoc (and many times it was not even DNS's fault).
Quick workaround is that for all backups you use in backup command save -L -c oracle2. So, if you go to client resouce and then one of those tabs (I believe second one - I'm talking now blindly) there is a field called backup command. Inside type "save -L -c oracle2" (without quotes) and see if there is difference this time. I'm quite sure there is going to be and I'm quite sure your index does not get deleted immediatelly (as seen by recover versions output in one of post above), but it gets removed by 24h nsrim check. nsrim does check for both index and media database and does so called cross-check. I believe this is where your data gets removed from index due wong name used, nested index structure, wrong clientid or any other potential cause that may cause this behavior. So, to make sure that your index this time is left there, after backup is done, do nsrim -X -c oracle2. This will do same thing as periodic check, but only for oracle2 client. Obviously, there are several possible things here they may have wrong and you should find what it is - either by trying to nail it down yourself or by letting support do it - in hands of people who know where to look this could be fixed in hald an hour over webex so I would suggest to talk to your support representative. Over the forum this may take some time and on top of that it may require you to send some data which some people in your company might not want to share (database entries of backups, client names, etc... in short your head might be rolling on the floor if someone wants to).