ndmp recovery

Hi,

    You cannot user scanner to build the indexes for NDMP savesets. If the indexes are present then you can use nsrck command to rebuild the indexes.

-c option is for source client

you can do the following

Use either NetWorker recover wizard or NetWorker User GUI and select old Celerra as the Source Client and New VNX as the Destination CLient and perform a Saveset Recovery of the file system you are looking for.

Gautam, thanks for coming to my help.

However, after running the 'nsrck' then retried the nsrndmp_recover command, the result remains the same.

Also, nsrck out put shows  NAS-Filer client contains 12 records occupying 35KB.

I used scanner -I because that's what most people in other forums recommends for NDMP recovery.

NetWorker GUI is not showing anything.

Note that there is no existing Index as this is a brand new Networker server setup just for this recovery operation. The original Networker server is gone.

Please what should I do next.

Was Celerra to VNX supported?  I know VNX in first gen was pretty much Celerra, but I'm not sure about difference they made in respect to ndmpd.  Also, if NAS-filer01 is original client, why is that mentioned in -m too (or did you call new VNX box after original Celerra so it was kind of HW upgrade?).  The log itself on NW side doesn't say much so you must check ndmpd log on VNX side.

Actually, the old NAS were Celerra NS480 and VNX7500 while the new NAS is VNX3200 --just for recovery purpose.

Basically, the 2 old NAS (ns480 & vnx7500) were decommissioned and data moved to a different platform.  VNX3200 was retained for recovery purposes.

We have recreated dummy instances of the old NAS and repopulate Networker client requirement. But I read somewhere that in order for it to work, the original clientID from the SaveSet must be used. I'm not sure how to do that.

yes that is correct. you would need the original Client ID of the old client. Was it not backed up to this NetWorker Server ? if backed up to another server, is that alive ?

It's Good that you found the Old Client ID. You can do the following to replace the client ID,

                          1. Copy the Client ID from Globals (1of1) tab of the Old NAS client you created.

                          2. Delete the Old NAS Client from NetWorker GUI

                          3. Create another client with name dummy1 and before you click ok, put the Client ID that you noted in step 1. It will give you a warning, accept it and you old NAS client would be free and without any client ID.

                          4. Create client with the OLD NAS filer name and put the "old client ID" in the "Globals (1of1) tab. and click ok.

                          5. Run scanner and nsrck to rebuild the indexes

                          6. Try to perform the recovery.

My 2 Cents.

I have deleted the Old NAS client from networker,

Created another client with the name "dummy1" and assigned the original Client-ID obtained from tape, but system rejected the dummy1 name as an invalid hostname.

I attempted to use other valid client names from the Old NAS, but all were produced this error: "Found NAS-Filer01, client ID: fe3f12******00004. Specify this client ID or leave the client ID blank. Otherwise the client name must first be renamed."

Basically, I can't use any valid client name as those are all present in the media index.

To complete the steps you suggested, I went ahead and setup a new valid host name as an alias to the original NAS-Filer01 client-name pointing it to the same IP as NAS-Filer01, put in the old Client-ID, did nsrck, tried recovery but still get the same problem.

**I did not do scanner again since the media info had already been done. Is my thinking right on this or do I have to repeat the scanning process?

Is there a way to move or hide the Index from Networker so as to recreate the Old client with the original client ID before reintroducing the media index into the mix?

I have gone through over a week scanning dozens of LT05 tapes into the media index and will hate to have to repeat this process.

Note: the old NAS filers and the old Networker servers are all gone for good. We are trying to recreate/restore the data from a new Networker-server to a new NAS filer.

Please any help will be highly appreciated.

Hi,

  You need to assign the new Client ID (for the old NAS Client) generated by NetWorker to dummy1. it will give you a warning that it is assigned to another host and would you like to rename it, accept it.

Then create the old NAS client with the Client ID that you found from the Tape. Run Scanner and try to perform the recovery.

You will also have to make sure, NetWorker server is able to resolve the old NAS client, you can put a host entry for any dummy IP.

To use dummy name, add dummy to local hosts table with some fake IP.

I have done all that - that's

deleted both mm and index folders from nsr\ folder.

I recreated all four instances of the old NAS servers using the original names and Client-ID (from tape).

Added each instance to the local Host file on Networker-Server, (I had 2 instances sharing a single IP)

Updated DNS records (all tested successfully to their assigned IP)

Ran scanner -iv on all 20 tapes.

Ran nsrck

Retried nsrndmp_recover and still have errors.
(I'm having trouble pasting the full text of the nsrndmp_recover output, not sure why the system is not allowing copy/paste  of unformatted regular text).

Here's just what stood out from the output:

1: nsrndmp_recover: Failed to query the address type of the NDMP data connection

2: nsrndmp_recover: Error during File NDMP Extraction

3: nsrndmp_recover: Error during NDMP recover

Note: Authentication to the NAS was successful.

C:\>nsrndmp_recover -D5 -s Networker-Server -c NAS-Filer01 -S 2968749678 -v off -m NAS-Filer01::Shared02/Shared02 /root_vdm_4/Shared02

05/25/16 13:16:19.046741 mmdbd server host name: Networker-Server
05/25/16 13:16:19.093612 RPC Authentication: error in LookupAccountSid: No mapping between account names and security IDs was done. (Win32 error 0x534)
05/25/16 13:16:19.124865 lgto_auth: redirected to Networker-Server prog 390103 vers 2
05/25/16 13:16:19.156118 lgto_auth for `nsrmmdbd' succeeded
05/25/16 13:16:19.187371 Destructive Restore is specified.
05/25/16 13:16:19.187371 SSID = 2968749678, Mount point =Shared02/Shared02, Raw device =(null)
05/25/16 13:16:19.202997 save set name: /root_vdm_4/Shared02
05/25/16 13:16:19.202997 Not a Celestra retrieval.
05/25/16 13:16:19.218623 save set name: /root_vdm_4/Shared02
05/25/16 13:16:19.234249 lgto_auth: redirected to Networker-Server prog 390103 vers 2
05/25/16 13:16:19.265502 lgto_auth for `nsrd' succeeded
05/25/16 13:16:19.281128 ndmp_create_connect: host(NAS-Filer01),user(ndmp),data(0000000000000000),timeout(60)
05/25/16 13:16:19.296755 socket (800) is opened for NDMP connection.
05/25/16 13:16:19.312381 ndmp_wait_connect: ndp=0000000002970130,timeout=60
05/25/16 13:16:19.312381 ndmp_skiprecord: 1
05/25/16 13:16:19.328007 Error in message header: No such file or directory.
05/25/16 13:16:19.328007 ndmp_skiprecord: 1
05/25/16 13:16:19.343718 dispatch NDMP_NOTIFY_CONNECTION_STATUS seqno 1
05/25/16 13:16:19.343718 ndmp_notify_connect: ndp(0000000002970130)
05/25/16 13:16:19.359349 Receive successful NDMP connection.
05/25/16 13:16:19.359349 ndmp_connect_open: ndp(0000000002970130)
05/25/16 13:16:19.359349 call NDMP_CONNECT_OPEN seqno 1
05/25/16 13:16:19.374975 ndmp_skiprecord: 0
05/25/16 13:16:19.374975 NDMP version 4 is used.
05/25/16 13:16:19.374975 ndmp_connect_auth= ndp(0000000002970130), host(NAS-Filer01), user (ndmp)
05/25/16 13:16:19.390601 call NDMP_CONFIG_GET_SERVER_INFO seqno 2
05/25/16 13:16:19.390601 ndmp_skiprecord: 1
05/25/16 13:16:19.390601 dispatch NDMP_LOG_MESSAGE seqno 3
05/25/16 13:16:19.406228 NDMP Service Debug: The process id for NDMP service is 0x99c3870
05/25/16 13:16:19.406228 ndmp_skiprecord: 1
05/25/16 13:16:19.406228 type supported : 1
05/25/16 13:16:19.406228 type supported : 2
05/25/16 13:16:19.421854 The NDMP user 'ndmp' has a password for host 'NAS-Filer01'.
05/25/16 13:16:19.421854 authentication method=2, host=NAS-Filer01, user=ndmp, pas=00000000028C4710
05/25/16 13:16:19.437480 CreateMD5Digest: ndp(0000000002970130)
05/25/16 13:16:19.437480 call NDMP_CONFIG_GET_AUTH_ATTR seqno 3
05/25/16 13:16:19.453106 ndmp_skiprecord: 1
05/25/16 13:16:19.453106 call NDMP_CONNECT_CLIENT_AUTH seqno 4
05/25/16 13:16:19.453106 ndmp_skiprecord: 1
05/25/16 13:16:19.468734 AUTH_MD5 connection succeeded!
05/25/16 13:16:19.468734 ndmp_connect_close
05/25/16 13:16:19.468734 call NDMP_CONNECT_CLOSE seqno 5
05/25/16 13:16:19.484359 ndmp_shutdown
42795:nsrndmp_recover: Performing recover from Non-NDMP type of device
05/25/16 13:16:19.484359 In fetchcvr_id func: fetching the details of saveset 2968749678
05/25/16 13:16:19.499986 get_vol_place:
05/25/16 13:16:19.515612 lgto_auth: redirected to Networker-Server prog 390103 vers 2
05/25/16 13:16:19.546866 lgto_auth for `nsrd' succeeded
05/25/16 13:16:19.562492 jukebox_device_host(), client:'', server:'', jukebox:'NW01_Lib', device_type: ''
05/25/16 13:16:19.578118 device_host(), devfull: \\.\Tape4
05/25/16 13:16:19.593744 jukebox_device_host(), device node: ''
05/25/16 13:16:19.593744 save set name: /root_vdm_4/Shared02
05/25/16 13:16:19.625002 nsrdsa_recover started, pid 5596
05/25/16 13:16:19.625002 Reading savetime from nsrdsa_save
05/25/16 13:16:19.640623 reset the ndmp_clnt_tbl to use the ndmp_bt_clnt_tbl.
05/25/16 13:16:19.640623 task = 2
05/25/16 13:16:19.656253 The block size used as in NSR_ATTR_NDMP_BLOCK_SIZE : 61440
05/25/16 13:16:19.656253 ndmp_create_connect: host(NAS-Filer01),user(ndmp),data(00000000007FF390),timeout(60)
05/25/16 13:16:19.671876 socket (828) is opened for NDMP connection.
05/25/16 13:16:19.687502 ndmp_wait_connect: ndp=0000000002989450,timeout=60
05/25/16 13:16:19.703129 ndmp_skiprecord: 1
05/25/16 13:16:19.703129 Error in message header: No such file or directory.
05/25/16 13:16:19.718755 ndmp_skiprecord: 1
05/25/16 13:16:19.718755 dispatch NDMP_NOTIFY_CONNECTION_STATUS seqno 1
05/25/16 13:16:19.718755 ndmp_notify_connect: ndp(0000000002989450)
05/25/16 13:16:19.734382 Receive successful NDMP connection.
05/25/16 13:16:19.734382 ndmp_connect_open: ndp(0000000002989450)
05/25/16 13:16:19.750008 call NDMP_CONNECT_OPEN seqno 1
05/25/16 13:16:19.750008 ndmp_skiprecord: 0
05/25/16 13:16:19.750008 NDMP version 4 is used.
05/25/16 13:16:19.765634 ndmp_connect_auth= ndp(0000000002989450), host(NAS-Filer01), user (ndmp)
05/25/16 13:16:19.765634 call NDMP_CONFIG_GET_SERVER_INFO seqno 2
05/25/16 13:16:19.781262 ndmp_skiprecord: 1
05/25/16 13:16:19.781262 dispatch NDMP_LOG_MESSAGE seqno 3
05/25/16 13:16:19.781262 NDMP Service Debug: The process id for NDMP service is 0x99c3870
05/25/16 13:16:19.796887 ndmp_skiprecord: 1
05/25/16 13:16:19.796887 type supported : 1
05/25/16 13:16:19.812514 type supported : 2
05/25/16 13:16:19.812514 The NDMP user 'ndmp' has a password for host 'NAS-Filer01'.
05/25/16 13:16:19.812514 authentication method=2, host=NAS-Filer01, user=ndmp, pas=0000000002970920
05/25/16 13:16:19.828139 CreateMD5Digest: ndp(0000000002989450)
05/25/16 13:16:19.843767 call NDMP_CONFIG_GET_AUTH_ATTR seqno 3
05/25/16 13:16:19.843767 ndmp_skiprecord: 1
05/25/16 13:16:19.843767 call NDMP_CONNECT_CLIENT_AUTH seqno 4
05/25/16 13:16:19.859392 ndmp_skiprecord: 1
05/25/16 13:16:19.859392 AUTH_MD5 connection succeeded!
05/25/16 13:16:19.859392 Connected to NDMP data server NAS-Filer01
05/25/16 13:16:19.875019 ndmp_negotiate_extensions(): try_bre 0, try_cae 1, try_cab 1
05/25/16 13:16:19.875019 call NDMP_CONFIG_GET_SERVER_INFO seqno 5
05/25/16 13:16:19.890645 ndmp_skiprecord: 1
05/25/16 13:16:19.890645 call NDMP_CONFIG_GET_EXT_LIST seqno 6
05/25/16 13:16:19.890645 ndmp_skiprecord: 1
05/25/16 13:16:19.906271 NDMP CAE is available
05/25/16 13:16:19.906271 call NDMP_CONFIG_SET_EXT_LIST seqno 7
05/25/16 13:16:19.906271 ndmp_skiprecord: 1
05/25/16 13:16:19.921898 NDMP data server extensions supported: cae 1, cab 0
05/25/16 13:16:19.921898 waitchild reaped, pid 5596
05/25/16 13:16:19.921898 call NDMP_CONFIG_GET_CONNECTION_TYPE seqno 8
05/25/16 13:16:19.937524 ndmp_skiprecord: 1
05/25/16 13:16:19.937524 Data server uses IPV4 for data connection
05/25/16 13:16:19.953151 Data server uses IPV6 for data connection
42799:nsrndmp_recover: Failed to query the address type of the NDMP data connection.
05/25/16 13:16:19.953151 Unexpected mover version 0. Mover listen address not freed.
05/25/16 13:16:19.968776 dsa mover abort dsapid = 5596
05/25/16 13:16:19.968776 Kill failed on 5 with error: The system cannot find the file specified. (Win32 error 0x2)
05/25/16
13:16:19.984403 ndmp_data_get_state: 0000000002989450,00000000007FF288
05/25/16 13:16:20.000029 call NDMP_DATA_GET_STATE seqno 9
05/25/16 13:16:20.000029 ndmp_skiprecord: 1
05/25/16 13:16:20.015656 The data process is in progress (0).
42871:nsrndmp_recover: Error during File NDMP Extraction.
05/25/16 13:16:20.015656 ndmp_connect_close
05/25/16 13:16:20.031282 call NDMP_CONNECT_CLOSE seqno 10
05/25/16 13:16:20.031282 ndmp_shutdown
05/25/16 13:16:20.031282 Connections closed sucessfully
42840:nsrndmp_recover: NDMP recover failed.
42880:nsrndmp_recover: Error during NDMP recover

I suspect this is a port issue between your data mover and backup server. I get this from:

- Failed to query the address type of the NDMP data connection

- Mover listen address not freed

- Kill failed on 5 (likely the file descriptor rather than an actual file)

Please get a network capture of communications between the backup server and VNX, and also the storage node and VNX (if applicable) and mail these to me privately - email address sent to your Community Inbox.... or it would have been if I could do this without us having a connection - please "follow me" so that I can send you this.

Get this network capture by downloading Wireshark from www.wireshark.org and installing this on your backup server (including winpcap).  Start a capture using the filter "host NAS-Filer01", run the recover attempt as above, stop the network capture and save it, then mail me both the recover log and the network trace file.

Hi StuartWhitby:

I have the captured network communication output - NAS-networker. Could you send me your email for me to send it over ASAP.

Thanks a lot. 

Stuwhitby:

I have emailed the network communication captured data to your email as instructed.

for some reason I couldn't get thru this forum too.

Thank you very much for your help.