NDMP Backup fails when changing Tape

Yes after the upgrade.

Did the error started after upgrade?

I recently went through specs of S3 and it looks same as oldish CS was. Going from S2 to S3 should not cause this, but it seems as it is somehow related. I would suspect that something in emulation done by S3 is not as it should be if you get this. At the end of the day, Celerra is the same and so is the backup software. I assume with S2 you also used emulation of i2000?  Is there a way to compare old and new config settings of i2000 on CS appliance?

Yes, I think thats there's something diferent with the emulation, but it work's well with the Networker server an two dedicated storage nodes (all are windows 2003 server). I can't compare with the S2 version, we upgrade was made in place and I don't have a test envirorment to check it.

Try to do:

They may be 3 problems:

1 - the CS800 has changed something at FC and has only affected to the NAS.

        - If it is a bug, should perform the following steps to verify that it works, if not work would be confirmed the bug.

2 that the NAS have connected wrong by FC

        - solution: try to restart the FC or restart the Celerra.

3 - in the NAS devices have misconfigured for some strange reason

        - solution: Rescan tape drives in Networker and reconfigure the tape devices of the NAS.

And take a look of that:http://support.microsoft.com/kb/2005265

http://support.microsoft.com/kb/2005265

Output of ndmp log:

2012-07-19 14:43:32: NDMP: 4: Session 805 (thread ndmp805) Name: PREFIX Value: /root_vdm_12/GR_D

2012-07-19 14:43:32: NDMP: 4: Session 805 (thread ndmp805) Name: FILESYSTEM Value: /root_vdm_12/GR_D

2012-07-19 14:43:35: NDMP: 3: Thread ndmp806 DartTapeInterface: malloc error

2012-07-19 14:43:35: NDMP: 3: Session 807 (thread ndmp807) TAPE_WRITE, size(262144) > ndmpMaxBufSize (131072)

2012-07-19 14:43:36: NDMP: 3: Session 809 (thread ndmp809) TAPE_WRITE, size(262144) > ndmpMaxBufSize (131072)

2012-07-19 14:43:40: NDMP: 4: Session 805 (thread ndmp805) tape drive: c32t0l3, CALLER_ADDRESS 0x00015f595b.

2012-07-19 14:43:40: NDMP: 6: Session 805 (thread ndmp805) < receive DATA_ABORT request >

2012-07-19 14:43:40: NDMP: 6: Session 805 (thread ndmp805) < sent DATA_ABORT reply, error: NDMP_NO_ERR >

2012-07-19 14:43:41: NDMP: 3: Session 810 (thread ndmp810) TAPE_WRITE, size(262144) > ndmpMaxBufSize (131072)

2012-07-19 14:43:54: NDMP: 3: Session 812 (thread ndmp812) TAPE_WRITE, size(262144) > ndmpMaxBufSize (131072)

2012-07-19 14:43:54: NDMP: 4: Session 813 (thread ndmp813) < Backup type: vbb >

2012-07-19 14:43:54: NDMP: 4: Session 813 (thread ndmp813) Name: HIST Value: y

2012-07-19 14:43:54: NDMP: 4: Session 813 (thread ndmp813) Name: DIRECT Value: y

2012-07-19 14:43:54: NDMP: 4: Session 813 (thread ndmp813) Name: UPDATE Value: y

2012-07-19 14:43:54: NDMP: 4: Session 813 (thread ndmp813) Name: OPTIONS Value: nt

2012-07-19 14:43:54: NDMP: 4: Session 813 (thread ndmp813) Name: SNAPSURE Value: Y

2012-07-19 14:43:54: NDMP: 4: Session 813 (thread ndmp813) Name: HOST Value: celerrandmp-cei

2012-07-19 14:43:54: NDMP: 4: Session 813 (thread ndmp813) Name: NDMP_BUTYPE Value: vbb

2012-07-19 14:43:54: NDMP: 4: Session 813 (thread ndmp813) Name: TYPE Value: vbb

2012-07-19 14:43:54: NDMP: 4: Session 813 (thread ndmp813) Name: LEVEL Value: 0

2012-07-19 14:43:54: NDMP: 4: Session 813 (thread ndmp813) Name: PREFIX Value: /root_vdm_14/ARXIU_B

2012-07-19 14:43:54: NDMP: 4: Session 813 (thread ndmp813) Name: FILESYSTEM Value: /root_vdm_14/ARXIU_B

2012-07-19 14:44:02: NDMP: 4: Session 813 (thread ndmp813) tape drive: c32t0l2, CALLER_ADDRESS 0x00015f595b.

2012-07-19 15:14:57: NDMP: 3: Thread ndmp817 DartTapeInterface: malloc error

2012-07-19 15:15:02: NDMP: 3: Thread ndmp818 DartTapeInterface: malloc error

2012-07-19 15:15:03: NDMP: 3: Session 819 (thread ndmp819) TAPE_WRITE, size(262144) > ndmpMaxBufSize (131072)

2012-07-19 15:24:20: NDMP: 3: Thread ndmp822 DartTapeInterface: malloc error

2012-07-19 15:24:26: NDMP: 3: Thread ndmp823 DartTapeInterface: malloc error

2012-07-19 15:24:26: NDMP: 3: Session 824 (thread ndmp824) TAPE_WRITE, size(262144) > ndmpMaxBufSize (131072)

2012-07-19 15:55:19: NDMP: 3: Session 813 (thread vbbproject04) NdmpdSession::ndmpdApiWrite fails in local or remote wirte, moverAddressType=0, dataBuf=0x00c060a000, length=8192

2012-07-19 15:55:19: NDMP: 6: Session 813 (thread ndmp813) < receive DATA_ABORT request >

2012-07-19 15:55:19: NDMP: 6: Session 813 (thread ndmp813) < sent DATA_ABORT reply, error: NDMP_NO_ERR >

2012-07-19 15:55:19: NDMP: 3: Session 826 (thread ndmp826) TAPE_WRITE, size(262144) > ndmpMaxBufSize (131072)

2012-07-19 15:55:26: NDMP: 3: Session 828 (thread ndmp828) TAPE_WRITE, size(262144) > ndmpMaxBufSize (131072)

2012-07-19 15:55:26: NDMP: 4: Session 829 (thread ndmp829) < Backup type: vbb >

2012-07-19 15:55:26: NDMP: 4: Session 829 (thread ndmp829) Name: HIST Value: y

2012-07-19 15:55:26: NDMP: 4: Session 829 (thread ndmp829) Name: DIRECT Value: y

2012-07-19 15:55:26: NDMP: 4: Session 829 (thread ndmp829) Name: UPDATE Value: y

2012-07-19 15:55:26: NDMP: 4: Session 829 (thread ndmp829) Name: OPTIONS Value: nt

2012-07-19 15:55:26: NDMP: 4: Session 829 (thread ndmp829) Name: SNAPSURE Value: Y

2012-07-19 15:55:26: NDMP: 4: Session 829 (thread ndmp829) Name: HOST Value: celerrandmp-cei

2012-07-19 15:55:26: NDMP: 4: Session 829 (thread ndmp829) Name: NDMP_BUTYPE Value: vbb

2012-07-19 15:55:26: NDMP: 4: Session 829 (thread ndmp829) Name: TYPE Value: vbb

2012-07-19 15:55:26: NDMP: 4: Session 829 (thread ndmp829) Name: LEVEL Value: 0

2012-07-19 15:55:26: NDMP: 4: Session 829 (thread ndmp829) Name: PREFIX Value: /root_vdm_14/ARXIU_B

2012-07-19 15:55:26: NDMP: 4: Session 829 (thread ndmp829) Name: FILESYSTEM Value: /root_vdm_14/ARXIU_B

2012-07-19 15:56:01: NDMP: 4: Session 829 (thread ndmp829) tape drive: c32t0l3, CALLER_ADDRESS 0x00015f595b.

2012-07-19 16:11:38: NDMP: 3: Thread ndmp831 DartTapeInterface: malloc error

2012-07-19 16:11:44: NDMP: 3: Thread ndmp832 DartTapeInterface: malloc error

2012-07-19 16:11:44: NDMP: 3: Session 833 (thread ndmp833) TAPE_WRITE, size(262144) > ndmpMaxBufSize (131072)

2012-07-19 16:23:09: NDMP: 3: Thread ndmp836 DartTapeInterface: malloc error

2012-07-19 16:23:15: NDMP: 3: Thread ndmp837 DartTapeInterface: malloc error

2012-07-19 16:37:25: NDMP: 3: Session 840 (thread ndmp840) md5 auth failed

2012-07-19 16:37:43: NDMP: 3: Session 841 (thread ndmp841) md5 auth failed

2012-07-19 16:41:47: NDMP: 3: Session 842 (thread ndmp842) md5 auth failed

2012-07-19 16:42:04: NDMP: 3: Session 843 (thread ndmp843) md5 auth failed

2012-07-19 16:42:18: NDMP: 3: Session 844 (thread ndmp844) md5 auth failed

2012-07-19 16:42:31: NDMP: 3: Session 845 (thread ndmp845) md5 auth failed

2012-07-19 16:49:58: NDMP: 3: Session 846 (thread ndmp846) md5 auth failed

2012-07-19 16:50:34: NDMP: 3: Session 847 (thread ndmp847) md5 auth failed

2012-07-19 16:50:45: NDMP: 3: Session 848 (thread ndmp848) md5 auth failed

2012-07-19 16:51:08: NDMP: 3: Session 849 (thread ndmp849) Error opening SCSI device: c32t0l1 NDMP_DEVICE_BUSY_ERR.

2012-07-19 17:03:16: NDMP: 3: Session 853 (thread ndmp853) Error opening SCSI device: c32t0l1 NDMP_DEVICE_BUSY_ERR.

According to http://www.scribd.com/doc/49845701/61/Table-2-35-NDMP-error-messages on page 142, you should try to reduce number of sessions due to overflowed buffer. With ndmp, I belive golden rule was one session per tape and no more than 4 sessions per data mover (though one could change that, but had to do some other tuning too).

In above, I assume that malloc error is the cause of your trouble.

Thanks Hrvoje,

Now we don't have de malloc error, we have changed the parameter: bufsz (Tape buffer size for I/O (kilobytes)) from 128 to 256, but the backup stops. We are now seeing a new error:

2012-07-20 18:13:50: NDMP: 6: Session 023 (thread ndmp023) < receive DATA_ABORT request >

2012-07-20 18:13:53: NDMP: 3: Session 023 (thread vbbproject00) NdmpdSession::ndmpdApiWrite fails in local or remote wirte, moverAddressType=0, dataBuf=0x0062b98000, length=8192.

Just out out curiosity, can you try backup without snapsure? I know that sysvol at capacity would cause all kinds of strange errors with NDMP backups (which is the reason why I do not use it since we never really built sysvol with that in mind). It is also funny to see that programmers still have that typo in error message...

Hi Hrvoje,

What kind of backup I can use, we are using vbb, maybe dump o tar backup. Without snapsure the volume is consistency state and granular recovery of files is possible.

Now we have more informatio the error:

1343196568: NDMP: 4: Session 175 (thread ndmp175) tape drive: c64t0l3, CALLER_ADDRESS 0x00015f595b.

1343196568: VBB: 6: Src: /root_vdm_14/ARXIU_B Ckpt: /root_vdm_14/automaticNDMPCkpts/automaticTempNDMPCkpt57-556-1343196545 BD /root_vdm_14/automaticNDMPCkpts/automaticTempNDMPCkpt57-556-1343196545

1343196770: NDMP: 3: Thread vbbproject05 DartTapeInterface: write error

1343196770: NDMP: 3: Session 128 (thread vbbproject05) Tape write error: No error reported  on drive c64t0l1.

1343196770: NDMP: 3: Session 128 (thread vbbproject05 fails in tapeWrite or tapeWriteMsg for drive c64t0l1, numMsg=8, maxMsg=64, n=-1, bcnt=61440, length=61

440, fileName=../NdmpdMover.cxx, lineNo=693.

1343196770: NDMP: 3: Session 128 (thread vbbproject05) NdmpdSession::ndmpdApiWrite fails in local or remote wirte, moverAddressType=0, dataBuf=0x007f914000,

length=8192

The drive activity stops and we get this error.

Any Ideas or alternatives to the ndmp backup for Celerra.

Thanks for your help.

I don't think your problem is NDMP... the message suggests that path to device has been gone. You may wish restart ports on switch or even restart ES3.  You can also engage Celerra support to check status of device paths (or you can do it yourself if you are familiar with commands and expected output).

We changed the Fc port of the S3 (and restarted once or twice) and connected it to a new port on the fc switch. We get the same result. Do you know how to check the devices paths, I only know howt to check if the devices are configured.

Thanks,

Ivan.

Top of my head it something like server_devconfig -list -scsi -nondisks which will give you device names and then you can test it with server_mt status (but you better check manual before as I'm currently not at the desk so I can't check it against my Celerra).

Thanks, that are the commands. I see the tapes and i can read it, rewind. I'm goint to do a test and see the tape status when the backup stops.