Start a Conversation

This post is more than 5 years old

Solved!

Go to Solution

5916

October 3rd, 2016 10:00

transport mode ignoring whats in the client config

Hello,

We have recently upgraded to networker 8.2 sp3. we utilize the legacy VADP solution currently, as we need to expand our storage to accommodate the 500gb minimum image size for the appliance.

We were then  having VADP backups that were taking 6+ hours for under 100gb of total space. after digging through the logs i found that SOME (not all) of them were defaulting to NBD instead of hotadd. i dont know if this is the cause of the slowness, but its somethign that is not working correctly, so i want to troubleshoot.

so for the sake of troubleshooting i hard-coded the transport mode in the client config VADP_TRANSPORT_MODE=Hotadd.

oddly enough, the next backup still showed NBD.

i would expect that if i explicitly set Hotadd as the transport method, that it would simply error out if that was not capable.... but that isnt happening. infact it appears to be completely ignoring anything to do with the transport method configuration. i have tried setting to san (which i KNOW dosent work) and there isnt even an acknowledgement in the log that the method was tried. i also tried adding a space at either end of the transport mode... still nothing. does anyone have any ideas on WHY this may be happening?

below you will find the output from my test group that i have been working with on this. and below THAT is the results of the "nsr client info".

backup job output

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Initializing VADP Configuration.

Using VUSVADP01 as the VADP proxy server host for client VUSWEB20.

Application information attribute VADP_HOST for client VUSWEB20 is set to "susvc01.llsa.local".

Application information attribute VADP_DISABLE_FLR for client VUSWEB20 is set to 'No'.

81095:nsrvadp_save: Failed to obtain "VADP_DISABLE_CBT" application information attribute for client VUSWEB20. Default value of "No" is taken.

Application information attribute 'VADP_QUIESCE_SNAPSHOT' for client 'VUSWEB20' is set to default value of "Yes".

Application information attribute VADP_BACKUPROOT for client VUSWEB20 is set to "c:\mnt".

Application information attribute VADP_TRANSPORT_MODE for client VUSWEB20 is set to "hotadd|nbd".

Application information attribute VADP_MAX_RETRIES for client VUSWEB20 is set to "0".

Application information attribute VADP_MAX_BACKOFF_TIME for client VUSWEB20 is set to "10".

Trying to reset the client resource application information attributes.

Application information attribute VADP_HYPERVISOR for client VUSWEB20 is set to "susvc01.llsa.local".

Application information attribute VADP_DISABLE_CBT for client VUSWEB20 is set to 'No'.

90245:nsrvadp_save: An unknown application information attribute is found for client 'VUSWEB20'.

VC\ESX Host Name        : 10.0.20.44

VC\ESX Port             : 443

VC\ESX Username         : Administrator

Mount point             : c:\mnt\VUSWEB20

Flavor                  : fullvm

Transport Mode          : hotadd:nbd

Max retry               : 0

Max backoff time        : 10

VDDK Install Directory: C:\Program Files\Legato\nsr\plugins\VDDK.

VADP VM-TMP Directory Location: c:\mnt\VUSWEB20\VUSWEB20\VUSWEB20-tmp\letters\1.

Temporary Directory for VM created.

VADP TMP Directory Location: c:\mnt\VUSWEB20\VUSWEB20\tmp.

Temporary Directory for VADP created.

Temporary vmMntLoc Directory for VADP created.

Performing the backup via the Virtual Center.

Application information attribute VADP_VM_NAME for client VUSWEB20 is set to "VUSWEB20".

Trying to connect to the VM using VM name 'VUSWEB20'.

Changed Block Trackgin(CBT) is already enabled on the VM.Performing a File level incremental over image level backup.

Creating a quiesced snapshot for 'VUSWEB20'.

10/03/16 11:59:51.684912 Creating snapshot for 'vm-87' ...

10/03/16 11:59:51.731710  Task is 0% complete

10/03/16 11:59:51.747309  Task is 10% complete

10/03/16 11:59:52.745671  Task is 20% complete

10/03/16 11:59:53.744032  Task is 30% complete

10/03/16 11:59:54.742394  Task is 40% complete

10/03/16 11:59:55.756355  Task is 50% complete

10/03/16 11:59:56.754717  Task is 60% complete

10/03/16 11:59:57.753078  Task is 70% complete

10/03/16 11:59:58.751440  Task is 80% complete

10/03/16 11:59:59.749801  Task is 90% complete

10/03/16 12:00:00.748163  Task is 95% complete

10/03/16 12:01:53.001445  Task is 95% complete

10/03/16 12:01:53.017045  Task is 95% complete

10/03/16 12:01:53.235436  Task is -1% complete

The snapshot was created successfully for VM 'VUSWEB20'.

Downloading VM configuration for VUSWEB20.

Configuration downloaded successfully for VM 'VUSWEB20'.

Following are the details of the client VUSWEB20:

VirtualCenter : 10.0.20.44

ESX Server : susvm02.llsa.local

DataCenter Name : LLSA

ConfigDatastore Name : LUN60_VNX

Resourcepool Name : /

Following is the disk information of the client VUSWEB20:

Disk 1 Label : Hard disk 1

Disk 1 Size : 40.0 GB

Disk 1 Datastore name : LUN60_VNX

Disk 2 Label : Hard disk 2

Disk 2 Size : 13.0 GB

Disk 2 Datastore name : LUN60_VNX

Total Number of Disks : 2

Obtained the virtual disks for the VM.

Change block tracking is enabled for VM: VUSWEB20

MORef for the VM is obtained.

Obtained the virtual disks for the VM.

Total Changed Sectors: 278272 Total Changed Sectors: 285312 83034:nsrvadp_save: Using CBT for identifying modified files.

Started Backup Operation.

Completion time: 10/3/2016 12:21:14 PM

Total time taken: 0 hr(s) 00 min(s) 43 sec(s)

Cleaningup VADP mount operation.

10/03/16 12:21:23.799887 Deleting snapshot for 'snapshot-54952' ...

10/03/16 12:21:23.877885  Task is 0% complete

10/03/16 12:21:28.542166  Task is 33% complete

10/03/16 12:21:29.571739  Task is 66% complete

10/03/16 12:21:29.852532  Task is 75% complete

10/03/16 12:21:30.336120  Task is 100% complete

10/03/16 12:21:30.694910  Task is 100% complete

10/03/16 12:21:30.710510  Task is -1% complete

Snapshot for VM VUSWEB20 is deleted.

Temporary directory 'c:\mnt\VUSWEB20' for VADP deleted.

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Client info

nsradmin> print type:"NSR client"; name:VUSWEB20

                        type: NSR client;

                        name: VUSWEB20;

                      server: susbck05.llsa.local;

                   client id: \

16ca44f7-00000004-4fe8ab89-4fea567f-001c1800-b73b72fe;

            scheduled backup: Enabled;

                     comment: ;

             Save operations: ;

            archive services: Disabled;

                    schedule: Default;

               browse policy: Policy 1;

            retention policy: Policy 1;

                  statistics: elapsed = 848565, index size (KB) = 103456,

                              amount used (KB) = 103456, entries = 566194;

                   directive: VCB directive;

                       group: index, EVADP10;

                    save set: *FULL*;

  Backup renamed directories: Disabled;

          Checkpoint enabled: Disabled;

      Checkpoint granularity: Directory;

Parallel save streams per save set: Disabled;

                    priority: 500;

   File inactivity threshold: 0;

File inactivity alert threshold: 0;

               remote access: *@*;

                 remote user: ;

                    password: ;

         NAS management user: ;

     NAS management password: ;

        NAS file access user: ;

    NAS file access password: ;

              backup command: nsrvadp_save;

                 Pre command: ;

                Post command: ;

     application information: VADP_HYPERVISOR=susvc01.llsa.local,

                              VADP_VM_NAME=VUSWEB20,

                              VADP_TRANSPORT_MODE=san | Hotadd,

                              VADP_DISABLE_CBT = NO;

     ndmp vendor information: ;

                        ndmp: No;

                  NAS device: No;

             NDMP array name: ;

  NAS device management name: ;

storage replication policy name: ;

       De-duplication backup: No;

         De-duplication node: ;

                        Pool: standard;

          Data Domain backup: Yes;

       Data Domain interface: IP;

               Client direct: Disabled;

         Probe resource name: ;

              virtual client: Yes;

          Block based backup: No;

               physical host: SUSVM02;

           Proxy backup type: VADP;

           Proxy backup host: VUSVADP01;

             executable path: ;

    server network interface: ;

                     aliases: VUSWEB20, VUSWEB20.llsa.local;

                  index path: ;

          owner notification: ;

                 parallelism: 4;

physical client parallelism: Disabled;

     autoselect storage node: Disabled;

               storage nodes: nsrserverhost, VUSVADP01;

       recover storage nodes: ;

   save session distribution: max sessions;

                  hard links: Disabled;

             short filenames: Disabled;

                 backup type: Filesystem;

              client OS type: Windows NT Server on Intel;

                        CPUs: 4;

           NetWorker version: 8.2.3.1.Build.1109;

              enabler in use: Yes;

       licensed applications: ;

               licensed PSPs: ;

nsradmin>

68 Posts

October 4th, 2016 01:00

Hi,

I've never tried to set this on a "normal" client ressource.

Please post the application information for your VADP proxy host VUSVADP01.

Marcel

68 Posts

October 4th, 2016 05:00

You want to use hotadd? Have you checked all prerequisites?

Which VMware version is running?

NW 8.2 SP1 and later VMware Integration Guide, Rev 15

Page 136

--

l You must perform the following prerequisites on the NetWorker server/proxy machine

in order to run vSphere version 5.5 and 6.0:

1. Since the registry key for SSL verification is not set by default, add the following

keypath in the registry:

'HKEY_LOCAL_MACHINE/SOFTWARE/Wow6432Node/VMware, Inc./VMware

Virtual Disk Development Kit'

Add a DWORD VerifySSLCertificates and set it to zero ('VerifySSLCertificates=0').

This will disable SSL verification for all VDDK Hotadd operations.

2. Install .NET framework 3.5.1 or later on the proxy. In Windows 2008 R2, even

though the .NET framework is bundled with the operating system, ensure that you

enable the framework under Server Manager- > features.

3. Install VC++ runtime 9.0 (VC++2008 SP1) on the proxy. The following link provides

more details:

http://www.microsoft.com/en-us/download/details.aspx?id=2092

The section Limitations to vSphere 5.5 and 6.0 support on page 138 provides

information on limitations when using vSphere 5.5 or 6.0 with the VADP solution.

--

October 4th, 2016 05:00

yes sir,

i have configured those pre-reqs. the thing that got me started in the first place was seeing that SOME vm's were taking 6+ hours to back up under 100gb. when i checked the log,those VM's were using NBD instead of Hotadd.

i cannot seem to find any rhyme or reason to why, which is why i started playing with the individual configs. some of the hotadd systems are on different hosts and luns as the proxy, so that doesnt apear to be the issue.

none of these vms' are over 250gb (many under 100gb). so i dont think its a block size issue.

any other suggestions?

October 4th, 2016 05:00

Thanks for the pointer.

so, now i am getting the expected result. it simply sits there, not doing anything. in vcenter you can see that a snapshot is taken, but its not mounting it. the systems are on the SAME LUN and VM host...... nothing in the log at all at this time.

the backups run in the context of the vcenter administrator account... so permissions shouldent be a problem.

thoughts or suggestions?

October 4th, 2016 05:00

ah, are you thinking it gets its config from the VADP proxy? i will try that out immediatly.

in the meantime here are teh settings from VUSVADP01

nsradmin> print type:"NSR client"; name:VUSVADP01

                        type: NSR client;

                        name: VUSVADP01;

                      server: susbck05.llsa.local;

                   client id: \

a59ad8ad-00000004-4fe8ab82-4fe8ab81-00011800-b73b72fe;

            scheduled backup: Enabled;

                     comment: ;

             Save operations: ;

            archive services: Enabled;

                    schedule: Default;

               browse policy: Month;

            retention policy: Month;

                  statistics: elapsed = 690, index size (KB) = 0,

                              amount used (KB) = 0, entries = 0;

                   directive: VCB directive;

                       group: Default, index;

                    save set: All;

  Backup renamed directories: Disabled;

          Checkpoint enabled: Disabled;

      Checkpoint granularity: Directory;

Parallel save streams per save set: Disabled;

                    priority: 500;

   File inactivity threshold: 0;

File inactivity alert threshold: 0;

               remote access: ;

                 remote user: ;

                    password: ;

         NAS management user: ;

     NAS management password: ;

        NAS file access user: ;

    NAS file access password: ;

              backup command: ;

                 Pre command: ;

                Post command: ;

     application information: VADP_MAX_BACKOFF_TIME=10,

                              VADP_TRANSPORT_MODE=hotadd|nbd,

                              VADP_HOST=susvc01.llsa.local,

                              "VADP_BACKUPROOT=c:\\mnt", VADP_MAX_RETRIES=0,

                              VADP_DISABLE_FLR=No;

     ndmp vendor information: ;

                        ndmp: No;

                  NAS device: No;

             NDMP array name: ;

  NAS device management name: ;

storage replication policy name: ;

       De-duplication backup: No;

         De-duplication node: ;

                        Pool: ;

          Data Domain backup: Yes;

       Data Domain interface: IP;

               Client direct: Enabled;

         Probe resource name: ;

              virtual client: No;

          Block based backup: No;

               physical host: ;

           Proxy backup type: ;

           Proxy backup host: ;

             executable path: ;

    server network interface: ;

                     aliases: VUSVADP01, vusvadp01.llsa.local,

                              VUSVADP01.llsa.local, vusvadp01;

                  index path: ;

          owner notification: ;

                 parallelism: 12;

physical client parallelism: Disabled;

     autoselect storage node: Disabled;

               storage nodes: VUSVADP01, nsrserverhost;

       recover storage nodes: VUSVADP01, SUSBCK05;

   save session distribution: max sessions;

                  hard links: Disabled;

             short filenames: Disabled;

                 backup type: ;

              client OS type: ;

                        CPUs: ;

           NetWorker version: ;

              enabler in use: No;

       licensed applications: ;

               licensed PSPs: ;

nsradmin>

68 Posts

October 4th, 2016 05:00

Yes, please give it a try

You can't mix the transport modes, so it makes sense to define it only on the proxy.

October 4th, 2016 05:00

disregard. as of now it finally kicked off (45 min after teh jobs started).

thank you for your assistance today!

68 Posts

October 4th, 2016 06:00

You're welcome.

It may take some time to start the backup, but 45 min seems to be too long.

Maybe you want to check the logs again or start the group in debug/verbose mode to see what happens.

If you don't need FLR VADP_DISABLE_FLR could help.

Check anti virus configuration on the VADP proxy (if installed).

October 4th, 2016 06:00

well this is maddening. after reviewing the backup logs, i get this..... at no point is NBD even entered as an OPTION.

Initializing VADP Configuration.

Using VUSVADP01 as the VADP proxy server host for client VUSWEB20.

Application information attribute VADP_HOST for client VUSWEB20 is set to "susvc01.llsa.local".

Application information attribute VADP_DISABLE_FLR for client VUSWEB20 is set to 'No'.

Application information attribute VADP_DISABLE_CBT for client VUSWEB20 is set to 'Yes'.

Application information attribute 'VADP_QUIESCE_SNAPSHOT' for client 'VUSWEB20' is set to default value of "Yes".

Application information attribute VADP_BACKUPROOT for client VUSWEB20 is set to "c:\mnt".

Application information attribute VADP_TRANSPORT_MODE for client VUSWEB20 is set to "hotadd".

Application information attribute VADP_MAX_RETRIES for client VUSWEB20 is set to "0".

Application information attribute VADP_MAX_BACKOFF_TIME for client VUSWEB20 is set to "10".

Trying to reset the client resource application information attributes.

Application information attribute VADP_HYPERVISOR for client VUSWEB20 is set to "susvc01.llsa.local".

Application information attribute VADP_DISABLE_CBT for client VUSWEB20 is set to 'Yes'.

90245:nsrvadp_save: An unknown application information attribute is found for client 'VUSWEB20'.

VC\ESX Host Name        : 10.0.20.44

VC\ESX Port             : 443

VC\ESX Username         : Administrator

Mount point             : c:\mnt\VUSWEB20

Flavor                  : fullvm

Transport Mode          : hotadd

Max retry               : 0

Max backoff time        : 10

VDDK Install Directory: C:\Program Files\Legato\nsr\plugins\VDDK.

VADP VM-TMP Directory Location: c:\mnt\VUSWEB20\VUSWEB20\VUSWEB20-tmp\letters\1.

Temporary Directory for VM created.

VADP TMP Directory Location: c:\mnt\VUSWEB20\VUSWEB20\tmp.

Temporary Directory for VADP created.

Temporary vmMntLoc Directory for VADP created.

Performing the backup via the Virtual Center.

Application information attribute VADP_VM_NAME for client VUSWEB20 is set to "VUSWEB20".

Trying to connect to the VM using VM name 'VUSWEB20'.

Performing a File level incremental over image level backup.

Creating a quiesced snapshot for 'VUSWEB20'.

10/04/16 09:33:54.970131 Creating snapshot for 'vm-87' ...

10/04/16 09:33:55.016930  Task is 0% complete

10/04/16 09:33:55.032529  Task is 0% complete

10/04/16 09:33:55.048129  Task is 10% complete

10/04/16 09:33:56.046503  Task is 20% complete

10/04/16 09:33:57.044878  Task is 30% complete

10/04/16 09:33:58.043252  Task is 40% complete

10/04/16 09:33:59.057226  Task is 50% complete

10/04/16 09:34:00.055601  Task is 60% complete

10/04/16 09:34:01.053975  Task is 70% complete

10/04/16 09:34:02.052349  Task is 80% complete

10/04/16 09:34:03.066323  Task is 90% complete

10/04/16 09:34:04.064698  Task is 95% complete

10/04/16 09:36:04.072421  Task is 95% complete

10/04/16 09:36:08.611904  Task is 95% complete

10/04/16 09:36:08.845898  Task is -1% complete

The snapshot was created successfully for VM 'VUSWEB20'.

Downloading VM configuration for VUSWEB20.

Configuration downloaded successfully for VM 'VUSWEB20'.

Following are the details of the client VUSWEB20:

VirtualCenter : 10.0.20.44

ESX Server : susvm02.llsa.local

DataCenter Name : LLSA

ConfigDatastore Name : LUN60_VNX

Resourcepool Name : /

Following is the disk information of the client VUSWEB20:

Disk 1 Label : Hard disk 1

Disk 1 Size : 40.0 GB

Disk 1 Datastore name : LUN60_VNX

Disk 2 Label : Hard disk 2

Disk 2 Size : 13.0 GB

Disk 2 Datastore name : LUN60_VNX

Total Number of Disks : 2

Obtained the virtual disks for the VM.

Change block tracking is enabled for VM: VUSWEB20

Successfully mounted the VM as a file system.

Using 'nbd' as the transport mode for Hard disk 1.

Using 'nbd' as the transport mode for Hard disk 2.

Starting NetWorker save operation.

C:\Program Files\Legato\nsr\\bin\save.exe -s susbck05.llsa.local -c VUSWEB20 -w 1478266428 -y 1478266428 -g EVADP10 -LL -a "device interface=data domain" -a "Data Domain interface=IP" -a "DIRECT_ACCESS=No" -b "standard" -m VUSWEB20 -t 1475587371 -o vss:*=off;REQUESTED_LEVEL:level=incr; -l incr -q -W 78 -N VM:VUSWEB20.llsa.local:*IMAGE* -x c:\mnt\VUSWEB20\VUSWEB20\VUSWEB20\__VADP_MOUNT__\500d2f59-b447-5b9d-1522-319ca87346f5\VUSWEB20\letters\

90034:save: The VCB directive is enabled for client 'VUSWEB20' via the VCB proxy server 'VUSVADP01'.

107204:save: 'Backup renamed directories' disabled, renamed directories will not be backed up

VUSWEB20: FULLVM                  level=incr,    159 MB 00:09:45    263 files

completed savetime=1475588287

94694:save: The backup of save set 'FULLVM' succeeded.

Cleaningup VADP mount operation.

10/04/16 09:47:55.398581 Deleting snapshot for 'snapshot-54997' ...

10/04/16 09:47:55.460979  Task is 0% complete

10/04/16 09:48:09.563018  Task is 20% complete

10/04/16 09:48:11.512968  Task is 33% complete

10/04/16 09:48:12.776535  Task is 50% complete

10/04/16 09:48:14.196099  Task is 75% complete

10/04/16 09:48:14.632888  Task is 80% complete

10/04/16 09:48:15.116475  Task is 100% complete

10/04/16 09:48:15.568864  Task is 100% complete

10/04/16 09:48:15.584463  Task is -1% complete

Snapshot for VM VUSWEB20 is deleted.

Temporary directory 'c:\mnt\VUSWEB20' for VADP deleted.

October 4th, 2016 06:00

i appreciate the additional pointers. now that i know the actual transport mode IS functioning, i can troubleshoot for throughput problems (items you suggested).

Thank you again!

October 4th, 2016 07:00

bingo.

id give another "correct" answer, but its already done .

68 Posts

October 4th, 2016 07:00

Test the following on the VADP proxy:

  • stop nsrexecd
  • delete any folder in ...\nsr\plugins\VDDK\tmp\
  • rename ...\nsr\tmp
  • start nsrexecd
  • run the job again
No Events found!

Top