Looking for some NDMP help. NetBackup error 99.

Please consider moving this question as-is (no need to recreate) to the proper forum for maximum visibility.  Questions written to the users' own "Discussions" space don't get the same amount of attention and questions can go unanswered for a long time.

You can do so by selecting "Move" under ACTIONS along the upper-right.  Then search for and select: "VNX Support Forum" which would be the most relevant for this question.

TimK wrote: 2013-07-17 13:30:36: NDMP: 4: Got stat buffer with data pointer=NULL at 1576, abort=1 2013-07-17 13:30:36: NDMP: 4: Thread nasa00 cifs_ea new count: 1810975, delete count: 1810975, freeListCount: 142 2013-07-17 13:30:36: NDMP: 3: Thread nasa00 Backup is aborted.

In the meantime, can you see if the following KB article is related:

emc300279: "NDMP Backups Failing in NetBackup"

Are you also getting the following error in the NetBackup logs?

ERR - Cannot write TIR data to media, NDMP return code 20.

Suggests an issue with the choice of link aggregation.  Does this pertain to your active data mover that it is failing on? 

Hi Christopher,

I checked out emc300279. I do not think it applies, but I am going to look into it further. I will let you know what I find.

Tim

Hi Karl,

Thank you for the advice. I do have an EMC support case open, but support gets busy and I wanted to see what input the user community might have.

All I have so far is it looks like ndmp is aborting, and I can not figure out exactly why.

Tim

If you haven't done so, you'll want to search the Symantec forums for the instructions to elevate the logging level in Netbackup.  Open a case with EMC Support, and be sure to tell them which version of NBU you're running.  Support can provide you with a set of commands (.server_config) to elevate the NDMP logging level on the datamover.  You can then grab the consolidated NBU logfile and compare it to the NDMP logs from the datamover, which will probably shed light on the problem.

Let us know if this helps!

TimK wrote: 2013-07-17 13:28:40: NDMP: 6: Session 014 (thread nasa00) backup dir /root_vdm_2/symlive is not on read-only checkpoint. 2013-07-17 13:28:40: PAX: 6: Thread nasa00 dedup backup ratio threshold of /root_vdm_2/symlive is 90% 2013-07-17 13:30:36: NDMP: 6: Session 014 (thread ndmp014) < receive DATA_ABORT request >

In the meantime, while you wait on support (as Mark stated, they will likely ask you to increase the debugging log level), I will make one more comment that I don't think it is able to create a SnapSure checkpoint first prior to taking the (NDMP) backup.  This is suggested by the error message, and I believe the filesystem has reached the fs_dedupe (default) 90% savvol_threshold.  Do you also see events within Unisphere that deduplication for that filesystem has aborted as the SavVol usage capacity had reached at least 90% of the filesystem's SavVol High Water Mark itself (which is also by default 90%). This doesn't in itself suggest that snapshots can not be created but limited by the information you provided, it is possible.

I am assuming of course that you had even set the option to create and backup a SnapSure checkpoint as part of the backup policy for that filesystem.  Generally speaking assuming it is supported by the Data Management Application (need to check the "NAS NDMP Interoperability Matrix", "Configuring NDMP Backups on VNX", and cross-reference also with Symantec support documents), the backup policy is created with the following setting: (SET) SNAPSURE=Y, or only if unsupported by the DMA, then can be set globally on the data mover via the following parameter: NDMP.snapsure

Can you verify that you did set your environment to take checkpoints first as part of the NDMP backup?  If so, then can you run the following commands:

1) fs_ckpt -list -all

2) fs_dedupe -info

Finally, can you try to manually create a checkpoint to see if this may be the culprit?

Hi Christopher,

I did not specifically setup check points, yet.

As you can see the out put from the fs_ckpt does not show any checkpoints.

$ fs_ckpt symlive -list -all

id ckpt_name creation_time inuse fullmark total_savvol_used ckpt_usage_on_savvol

Info 26306752329: The value of ckpt_usage_on_savvol for read-only checkpoints may not be consistent with the total_savvol_used.

id wckpt_name inuse fullmark total_savvol_used base ckpt_usage_on_savvol

$

$ fs_dedupe -info symlive

Id = 34

Name = symlive

Deduplication = On

Status = Idle

As of the last file system scan (Fri Jul 12 18:16:46 CDT 2013):

Files scanned = 2472278

Files deduped = 220134 (9% of total files)

File system capacity = 1613399 MB

Original data size = 1462223 MB (91% of current file system capacity)

Space saved = 229193 MB (16% of original data size)

File system parameters:

Case Sensitive = no

Duplicate Detection Method = sha1

Access Time = 15

Modification Time = 15

Minimum Size = 24 KB

Maximum Size = 8388608 MB

File Extension Exclude List =

Minimum Scan Interval = 7

SavVol Threshold = 90

Backup Data Threshold = 90

Cifs Compression Enabled = yes

Pathname Exclude List =

I was able to manually create a check point.

$ fs_ckpt symlive -Create

operation in progress (not interruptible)...id = 34

name = symlive

acl = 0

in_use = True

type = uxfs

worm = off

volume = v189

pool = Gold File Pool 45

member_of = root_avm_fs_group_42

rw_servers= server_3

ro_servers=

rw_vdms = odcvnx2107dm3_vdm

ro_vdms =

auto_ext = no,thin=no

deduplication = On

thin_storage = False

tiering_policy = Auto-Tier/Optimize Pool

compressed= False

mirrored = False

ckpts = symlive_ckpt1

stor_devs = APM00114602107-11B2,APM00114602107-11B3,APM00114602107-11B4,APM00114602107-11B5,APM00114602107-11B6

disks = d19,d32,d20,d33,d21

disk=d19 stor_dev=APM00114602107-11B2 addr=c0t2l6 server=server_3

disk=d19 stor_dev=APM00114602107-11B2 addr=c16t2l6 server=server_3

disk=d32 stor_dev=APM00114602107-11B3 addr=c16t2l7 server=server_3

disk=d32 stor_dev=APM00114602107-11B3 addr=c0t2l7 server=server_3

disk=d20 stor_dev=APM00114602107-11B4 addr=c0t2l8 server=server_3

disk=d20 stor_dev=APM00114602107-11B4 addr=c16t2l8 server=server_3

disk=d33 stor_dev=APM00114602107-11B5 addr=c16t2l9 server=server_3

disk=d33 stor_dev=APM00114602107-11B5 addr=c0t2l9 server=server_3

disk=d21 stor_dev=APM00114602107-11B6 addr=c0t2l10 server=server_3

disk=d21 stor_dev=APM00114602107-11B6 addr=c16t2l10 server=server_3

id = 36

name = symlive_ckpt1

acl = 0

in_use = True

type = ckpt

worm = off

volume = vp193

pool = Gold File Pool 45

member_of =

rw_servers=

ro_servers= server_3

rw_vdms =

ro_vdms = odcvnx2107dm3_vdm

checkpt_of= symlive Fri Jul 19 10:26:09 CDT 2013

deduplication = Off

thin_storage = False

tiering_policy = Auto-Tier/Optimize Pool

compressed= False

mirrored = False

used = 1%

full(mark)= 90%

stor_devs = APM00114602107-119E,APM00114602107-119F,APM00114602107-11A0,APM00114602107-11A1,APM00114602107-11A2

disks = d9,d22,d10,d23,d11

disk=d9 stor_dev=APM00114602107-119E addr=c0t1l2 server=server_3

disk=d9 stor_dev=APM00114602107-119E addr=c16t1l2 server=server_3

disk=d22 stor_dev=APM00114602107-119F addr=c16t1l3 server=server_3

disk=d22 stor_dev=APM00114602107-119F addr=c0t1l3 server=server_3

disk=d10 stor_dev=APM00114602107-11A0 addr=c0t1l4 server=server_3

disk=d10 stor_dev=APM00114602107-11A0 addr=c16t1l4 server=server_3

disk=d23 stor_dev=APM00114602107-11A1 addr=c16t1l5 server=server_3

disk=d23 stor_dev=APM00114602107-11A1 addr=c0t1l5 server=server_3

disk=d11 stor_dev=APM00114602107-11A2 addr=c0t1l6 server=server_3

disk=d11 stor_dev=APM00114602107-11A2 addr=c16t1l6 server=server_3

$ fs_ckpt symlive -list -all

id ckpt_name creation_time inuse fullmark total_savvol_used ckpt_usage_on_savvol

36 symlive_ckpt1 07/19/2013-10:26:09-CDT y 90% 1% 1%

Info 26306752329: The value of ckpt_usage_on_savvol for read-only checkpoints may not be consistent with the total_savvol_used.

id wckpt_name inuse fullmark total_savvol_used base ckpt_usage_on_savvol

$

Tim

Hi Karl,

I did get the commands to enable debug on the datamover from support.  My assumption might be wrong, but since ndmp backups work just fine on datamover 2 and do not work on datamover 3.  I have been focused on the datamover side and have not dug into the netbackup logs.  That might be the next step based on what support has to say about the datamover logs with debug turned on.

Tim

Thanks for the info.  Ignore me, I misinterpreted the logs.

Christopher,

No worries.  I appreciate all the in put I can get at this point.

Hi Karl,

The passwords should be the same, but I am not using Global NDMP passwords. Only one of my netbackup media servers has the ndmp license.

The below test works from my media server with out any issue so I know it is not a password issue.

odcmedia1:/tmp # /usr/openv/volmgr/bin/tpautoconf -verify ndmp-vnx2107-dm3

Connecting to host "ndmp-vnx2107-dm3" as user "ndmp"...

Waiting for connect notification message...

Opening session--attempting with NDMP protocol version 4...

Opening session--successful with NDMP protocol version 4

host supports TEXT authentication

host supports MD5 authentication

Getting MD5 challenge from host...

Logging in using MD5 method...

Host info is:

host name "server_3"

os type "DartOS"

os version "EMC Celerra File Server.T.7.0.54.3"

host id "abc1997"

Login was successful

Host supports LOCAL backup/restore

Host supports 3-way backup/restore

Tim

Sounds good to me.  Out of curiosity, do you have different NDMP passwords on DM2 and DM3?  Or are you using a Global NDMP password in Netbackup?

Yup - the verify eliminates the password issue.  So much for my shot in the dark!

Hi Karl,

No problem. I am tired of reloading, I have taken so many shots in the dark on this problem.

EMC support is escalating to a senior engineer, and I just sent off some very large logs to Symantec Support.

Tim

All,

The below Symantec technote resloved my ndmp backup issues.

http://www.symantec.com/business/support/index?page=content&id=TECH192541

Note the technote does not reference VNX but changing my ndmp backup type from dump to tar

worked around the issue.  I was also getting the exact errors in my netbackup debug log as referenced in TECH192541.

Note: The version of NetBackup I am running is 7.5.0.4

I would like to extend a thank you to everyone who commented on my post.

-Tim

The below is a cut and paste out of TECH192541.

Solution

Review the NetBackup Hardware Compatibility list and confirm that the filer and backup method are supported.

For this specific filer and NetBackup version, NDMP backups of TYPE=dump are not supported due to the unexpected values in the file history.  Change the NDMP backup to TYPE=tar to perform a successful backup.

Support for this new type of file history is currently planned for NetBackup 7.6

http://www.symantec.com/business/support/index?page=content&id=TECH192541

All,

The work around I provided might not be as good as I thought.

See the limitation of using ndmp with type=tar.

See technote with the below title.

Why it is not recommended to use “set type=tar” in an NDMP backup policy protecting an EMC networked storage device

http://www.symantec.com/business/support/index?page=content&id=TECH202412