Highlighted
phani6039
1 Nickel

Help with recover

Hi Gurus,

I am new to legato networker and I was assigned a task to recover some files. The client is using Solaris and the server is a Linux machine.

It is returning the following error as below when I am trying to do a recover for a client as one of the users has corrupted the data in databases for them.

We are running networker 7.3

When tried from server:

[root@server01 ~]# recover -c oracle02 -s server01

6550:recover: No browsable index found for client oracle02.

When tried from client side

bash-3.00# recover -c oracle02 -s server01

39584:recover: No files have been backed up for this client. No index was found on the server for the path / on this client.

Currently we are backing these three loation /, /backup1, /opt/oracle for client server

/backup1 is a mounted from netapp storage.

When I try below commands you specified for above partitions I am able to change to the required time for only / and rest of the two partition it throws the error.

From client machine (oracle02)

bash-3.00# pwd

/

bash-3.00# recover -s server01

Current working directory is /

recover> changetime 14-july-2012 10:00

6497:recover: time changed to Sat Jul 14 10:00:00 2012

recover> exit

bash-3.00# cd /backup1

bash-3.00# recover -s server01

4181:recover: Path /backup1 is within hmdnas02b:/vol/asbkup002T

53362:recover: Cannot start session with server server01: client `hmdnas02b' is not properly configured on the NetWorker Server or `hmdnas02b' is not in the aliases list for client `server01'

bash-3.00# cd /opt/oracle/

bash-3.00# recover -s server01

Current working directory is /opt/oracle/

recover> changetime 14-july-2012 10:00

6496:recover: cannot browse /opt/oracle/ as of `Sat Jul 14 10:00:00 2012'

I have check for nsrck for this specific client and it shows no indexes available as below

Server side:

[root@server01 ~]# nsrck -c server

nsrck: checking index for 'server01'

nsrck: /nsr/index/server01 contains 0 records occupying 0 KB

nsrck: Completed checking 1 client(s)

But every day in the save group completion I am able to see that this server is been backed up and got index files on server. Below is the backed up data from yesterday

[root@server01 ~]# mminfo |grep -i oracle02

A00024L4       oracle02 15/07/12 230 MB    incr  /

A00024L4       oracle02 15/07/12  40 GB    incr  /backup1

A00024L4       oracle02 15/07/12 7333 MB   incr  /opt/oracle

A00024L4       oracle02 16/07/12 7350 MB   incr  /opt/oracle

A00024L4       oracle02 16/07/12  40 GB    incr  /backup1

A00024L4       oracle02 16/07/12  11 MB    incr  /

Any idea or information regarding this would be really helpful.

Thanks,

Phani

0 Kudos
9 Replies
masonb
2 Iron

Re: Help with recover

phani6039,

From the mminfo output the only backups in the Media database reported are from 15 and 16/7. You change time to 14/7?

What is the browse/retention period of the client backups?

Also which user are you logged in as on the client - I can see from the NetWorker server output you are in as root but not on the client?

Under the /nsr/index/oracle02 directory - if you do a long listing of this directory are there any files present?

Check your alias name resolution for oracle02 also to ensure all names and IP address can be resolved.

Hope this helps,

Regards,

Bill Mason

0 Kudos
phani6039
1 Nickel

Re: Help with recover

Thanks Bill for your response,

What is the browse/retention period of the client backups?

The browse and retention policy is 3 months and i have also got the tape in the Robot  for 14th July.

Also which user are you logged in as on the client - I can see from the NetWorker server output you are in as root but not on the client?

I have logged in as root on the client as well.

Under the /nsr/index/oracle02 directory - if you do a long listing of this directory are there any files present?

On the server side i am able to see below output

[root@server01 ~]# cd /nsr/

[root@server01 nsr]# cd index/

[root@server01 index]# cd oracle02/

[root@server01 oracle02]# ll

total 8

drwx------ 3 root root 4096 Jul 16 16:13 db6

-rw-r--r-- 1 root root  970 Aug  4  2011 README

-rw------- 1 root root    0 Aug  4  2011 v6ck.lck

[root@server01 oracle02]# cd db6/

[root@server01 db6]# ll

total 12

drwxr-xr-x 2 root root 4096 Jul 16 16:13 tmprecov

-rw------- 1 root root   48 Jul 16 12:28 v6hdr

-rw------- 1 root root    0 Aug  4  2011 v6hdr.lck

-rw------- 1 root root    0 Jul 16 12:28 v6journal

-rw------- 1 root root   54 Jul 16 16:13 v6tmp.ptr

And on the client side i cant find the index folder anywhere.

Check your alias name resolution for oracle02 also to ensure all names and IP address can be resolved.

hmdnas02b:/vol/asbkup002T is a storage partition that is been mounted on to the client (oracle02) as /backup1 which gets backed up eveyday at mid night.


hmdnas02b:/vol/asbkup002T       100GB 55GB 45GB /backup1


I have check the alias names and found i have got only 2 names up there oracle02 oracle02.gas.com

No do we need to specify the storage ie hmdnas02b up there as well?

Now I have mounted this partition locally on to the server01 and tried this  as below

[root@server01 ~]# cd /

[root@server01 /]# mkdir /backup1

[root@server01 /]# mount hmdnas02b:/vol/asbkup002T /backup1/

[root@server01 /]# cd backup1/

[root@server01 backup1]# cd prd/

Started the recovery as below after mounting the location

[root@server01 ecisgprd]# recover -c server01 -s server01

Current working directory is /backup1/prd/

recover> changetime 14-july-2012 10:00

6496:recover: cannot browse /backup1/prd/ as of `Sat 14 Jul 2012 10:00:00 CET'

recover> changetime 13-july-2012 10:00

6496:recover: cannot browse /backup1/prd/ as of `Fri 13 Jul 2012 10:00:00 CET'

Any idea if i am missing anything?

Thanks,

Phani

0 Kudos
ble1
5 Iridium

Re: Help with recover

Is that NFS mount? Of course it is.  In that case you must be aware that there are some special rules when it comes to backup of NFS shares. Not that is is not possible, but by default it is not done unless you use switch where you say all mnt points will be seen as local file systems.  You might use that since you did backup something already (unless under that mounting point you had some other data at the time of backup which would make nested mounting point and things more difficult). Now, your data is in mdb under name of client oracle2.  One thing to do is following:

- go to oracle2

- do recover -s server01 (do this from root directory)

- do ls

- make sure you see folder you wish to restore, like /backup1

- run versions backup1

- do changetime <time as seen in outout of previous command>

- cd into backup1

- cd into substructure and select file to restore

- restore it

If it complains about destination, set target destination to local folder and restore it there.

Obviously, you can do also ssid restore, but perhaps we shouldn't go there at this point.

You may find above strange, but with filers or anything else, you should use NDMP protocol to make backup. Some people like to avoid certain license fees so they mount they whole datacenter to backup server and backup everything under one client. Surely you can try to do that, but it comes with price tag unless you know what you are doing. Anyway, try above and if it doesn't work you should try to indetify ssid in which data is which you wish to recover and do ssid recover instead.

0 Kudos
phani6039
1 Nickel

Re: Help with recover

Hi Hrvoje,

You were right that it is a NFS mount. I have tried above steps

- logged into oracle2

- recover -s server01 (do this from root directory)

- ls

- cd /backup1

recover> ls

prd     

recover> cd prd  

recover> ls

archive export    hot

recover> cd export

recover> ll

total 285096

284764 -rw-r--r-- oracle   dba      91017625 Jul 16 22:58 x_20120716_2020.dmp.gz

108 -rw-r--r-- oracle   dba 5100 Jul 16 22:58 x_20120716_2020.log

112 -rw-r--r-- oracle   dba 6652 Jul 16 22:58 y_20120716.log

112 -rw-r--r-- oracle   dba 6652 Jul 16 22:58 y_20120716_2020.log

recover> cd

At this point we need /backup1/prd/export directory that needs to be recovered on to a different location  from last week on to Friday (initial request) I have checked the tapes and the Friday tapes that is used in is still in robot but when I run the versions command it is not showing up.

recover> pwd

/

recover> versions /backup1

Versions of `/backup1/':

   4 drwxr-xr-x oracle   dba 4096 Aug 4 2011 backup1/

     save time: Tue Jul 17 01:31:46 2012                         

      location: Unknown

   4 drwxr-xr-x root     root         4096 Jul 16 16:53 backup1/

     save time: Mon Jul 16 21:00:02 2012                         

      location: A00024L4 at /dev/nst0

recover> changetime 16-Jul-2012

6497:recover: time changed to Mon Jul 16 23:59:59 2012

recover> ll 

recover> ls

recover> changetime 16-Jul-2012 21:00

6496:recover: cannot browse /backup1 as of `Mon Jul 16 21:00:00 2012'

recover> changetime 16-Jul-2012 21:00:02

6497:recover: time changed to Mon Jul 16 21:00:02 2012

recover> ll

recover> ls

Here I was able to see the backup1 on 17th July but I actually need the data from 13th July (Friday) which is not showing up here. I tried for 16th to see if I can get anything but I can’t find any files inside it at all.

If you check the mminfo output for below command which shows the full backup of oracle02 server on all the tapes.

[root@server01 ~]#  mminfo -s server01 -v -q 'client=oracle02,name=/backup1,level=full' | more

volume client           date     time         size       fl lvl name

A00001L4       oracle02       27/04/12 01:32:28   75 GB  tb full /backup1

A00024L4       oracle02       13/07/12 01:31:46   49 GB  tb full /backup1

A00077L4       oracle02       13/01/12 01:33:22   69 GB 3 cE full /backup1

Removed rest of the tapes details as not needed.

My initial task is to get the data of /backup1/ecisgprd/export on to different locations for the below tapes. Got tape 24 in the robot itself

  1. A00024L4       oracle02       13/07/12 01:31:46   49 GB  tb full /backup1

Once the above is done then need to get the data of same location from other date

   2. A00001L4       oracle02       27/04/12 01:32:28   75 GB  tb full /backup1

Then the third one for below which is expired

   3. A00077L4       oracle02       13/01/12 01:33:22   69 GB  cE full /backup1 

Any idea if i am missing anything as not able to recover the data for above tapes

Thanks,

Phani

0 Kudos
ble1
5 Iridium

Re: Help with recover

Hi Phani,

The data from 17th says location is unknown - this may indicate that tape is either out of library of that index and mdb records are not connected. Later one can happen with NFS backups if you make saves without -L and -c options (top of my head, I would need to check manual to be 100% sure).  Now, I think we can debate for days as to what happened and what needs to be changed, but I will skip that part as you need data back and get to plan B.

Plan B is - ssid restore. We do not read index but we read tape and we get data back - period. Now, you need data from Friday 13th (critics will say that is your problem). That backup is spanning two volumes:

A00024L4       oracle02       13/07/12 01:31:46   49 GB 1140812989 tb full /backup1

A00025L4       oracle02       13/07/12 01:31:46   32 GB 1140812989 hb full /backup1

Start or (h)ead is on 25 and end or (t)ail is on 24. In total some 81GB. Not sure how familiar you are with ssid restore, but it's simplest form is following:

recover -s <server> -S 1140812989

Do not run this yet, but read on. Above command says talk to <server> and start restoring ALL data from unique save set id <ssid>.  In above example I used ssid which represents data you want from Friday. What NW would do is to read whole thing and restore it back - some 81GB data.  Now, you do not wish to get 81GB data back and perhaps even not at the same location.  Location is easy - just used -d switch and say where it should go, for example:

recover -s <server> -S 1140812989 -d /var/adm/crash

In example above, those 81GB will go instead of original location to /var/adm/crash.  Of course, in your case, that might be something else like /tmp or whatever place you want with enough space.  Which brings us to final topic - you want specific data - /backup1/ecisgrpd/export

To get that, at the end you just add path you wish to restore, for example:

recover -s <server> -S 1140812989 -d /var/adm/crash /backup1/ecisgrpd/export

Do not run it yet, but read on   Above will read ssid, dump everything matching file/folder specified into -d location. Now, to understand what is going on you must know that ssid restore reads data from tape - it does not care about index database at all. This means that ssid restore does not know where your data exactly is withing those 81GB so whole ssid will be read. What is restored is only what you requested (the best way to understand it is that this is sort of grep filter which is, as grep, case sensitive so make sure to type pattern correctly). This is your fast way out and method to restore it or simply plan B. To make sure plan A works next time, have someone with more experience check your config and make necessary changes and tests to make sure plan B does not happen again.

phani6039
1 Nickel

Re: Help with recover

Hi Hrvoje,

Thanks for your detailed explanation. I am able to recover the data as such of now.

Just a quick question as why for this client alone the networker is failing to store the indexes when it is storing for all other clients which belong to the same group.

What i have noticed is that everyday it is backing up the client but it doesnt have the CFI stored any information what can cause these issues?

Once again thanks a lot for the explanation.

Thanks,

Phani

0 Kudos
ble1
5 Iridium

Re: Help with recover

Group is not important here - client is. More precisely, to what client does data belong to.  By default NW knows this and does backup of local data only.  You can make it backup NFS data of course, but then you need to make sure that data is saved under index name you want and that NW does traverse mnt point. So, in your case that would mean backup command in client resource should be save -L -c <client name>. To make is clean, use clientname of the machine where NFS share is mounted on. For vast majority of filers, you should consider NDMP based backup.

0 Kudos
phani6039
1 Nickel

Re: Help with recover

Thanks for quick reply,

Data belongs to oracle02 client for which NW backs up three locations / , /opt/oracle (both local) and /backup1 which is a NFS mount used by this client.

Indexes are not getting stored for any backed up locations for this client (oracle02).

[root@server01 ~]# nsrck -c oracle02

nsrck: checking index for 'oracle02'

nsrck: /nsr/index/oracle02 contains 0 records occupying 0 KB

nsrck: Completed checking 1 client(s)

I had a look at media > properties > configurations as well there store indexes are checked for this pool.

how could i find what command is it using for backing up every time as you were specifying save -L -c <client name>.

I havent configured this and it was configured before i joined here.

In order to mount the clientname (hmdnas02b) of the machine where NFS share is mount we havent got enough client licenses available at the moment.

I had a look at the properties of the client (oracle02) using GUI and they seems to be normal as that of other clients which are getting backed up.

May be i am being dumb up here, could explain it plz.

Thanks,

Phani

0 Kudos
ble1
5 Iridium

Re: Help with recover

Nope, it does not belong to oracle2 client.  It belongs to client from where that NFS share is coming from.  And NetWorker knows that so by default this data is not local to oracle2 and thus it does not get saved under oracle2 index.

Your nsrck shows that data gets saved somewhere else - actually it shows index db for oracle2 is empty which indicates there is some other trouble going on (since / and /opt/oracle are local). Now, there is a question why and there are several possible reason there (David has similar problem too, though I doubt you have same cause).

First, you use oracle2 as name for client.  I will assume this is correct and I will also assume your DNS or/and /etc/hosts are correct (but you better double check). Go to backup server and check what do you see under /nsr/index/oracle2. More precisely, check if you have something which called nested index structure.

Second, make sure box really does resolve back to oracle2. Assumtion is mother of all <censorship> so you better check that twice. There bunch of stories where and if name authetification have caused havoc (and many times it was not even DNS's fault).

Quick workaround is that for all backups you use in backup command save -L -c oracle2. So, if you go to client resouce and then one of those tabs (I believe second one - I'm talking now blindly) there is a field called backup command.  Inside type "save -L -c oracle2" (without quotes) and see if there is difference this time. I'm quite sure there is going to be and I'm quite sure your index does not get deleted immediatelly (as seen by recover versions output in one of post above), but it gets removed by 24h nsrim check.  nsrim does check for both index and media database and does so called cross-check.  I believe this is where your data gets removed from index due wong name used, nested index structure, wrong clientid or any other potential cause that may cause this behavior.  So, to make sure that your index this time is left there, after backup is done, do nsrim -X -c oracle2.  This will do same thing as periodic check, but only for oracle2 client. Obviously, there are several possible things here they may have wrong and you should find what it is - either by trying to nail it down yourself or by letting support do it - in hands of people who know where to look this could be fixed in hald an hour over webex so I would suggest to talk to your support representative. Over the forum this may take some time and on top of that it may require you to send some data which some people in your company might not want to share (database entries of backups, client names, etc... in short your head might be rolling on the floor if someone wants to).

0 Kudos