Data Domain: DDPCONNCHK How to troubleshoot DD Boost connectivity and performance

• A media server cannot connect to a specific DDR using DD Boost protocol. (This also applies to DDBoost/RMAN plug-in cannot connect to DDR)
• A media server failed to backup to DDR using DD Boost protocol
• DD Boost Managed File Replication (MFR) failed
• DD Boost backup or restore performance slow
• DD Boost MFR performance slow
• DDPCONNCHK utility can be used to simulate complete Backup/WRITE IO, Restore/READ IO and Expire/DELETE IO from the affected Backup Host by excluding the Backup Application itself but including all other components at all other locations, by using exact vales from the Backup App configuration.
• In addition, DDPCONNCHK utility can be configured with proper options to simulate Managed File Replication or MFR over DDBOOST from the affected Backup Host between relevant Source (SRC) DD and Destination (DST) DD.
• It is always recommended to perform complete independent Backup/WRITE IO, Restore/READ IO and Expire/DELETE IO simulation to both Data Domains, along with complete MFR simulation during Clone or MFR troubleshooting.

WHERE TO OBTAIN THE DDPCONNCHK TOOL:
Download the ddpconnchk.zip file from: https://central.dell.com/solutions/Networker-Tools
The zipped package contains several versions of the utility across multiple OS platforms.


Notes:
-ddpconnchk tool must be run on the media server, which is the host from where the data is written to the Data Domain using DD Boost. So first find out the OS/bits of the media server in question. Based on the media server's OS/bits, download the corresponding ddpconnchk file, and put it on the media server.
-ddpconnchk is an executable file, so it cannot be sent as email attachment.

HOW TO MAKE DDPCONNCHK READY TO RUN ON MEDIA SERVER: 

• Winscp or scp to copy the file from download location to the media server.
• No installation required ddpconnchk is a standalone executable.
• Gunzip or Winzip or similar products can be used to extract package.
• Executable permissions are required to run ddpconnchk, for example:

  chmod +x 

• Ddpconnchk may be copied and run from any directory the user is authorized. For example you can use paths /tmp, or C:\Dell, or C:\ddtools.

HOW TO RUN DDPCONNCHK:
Basic syntax:

#<ddpconnchkfilename> -s <DD Hostname Exactly in Backup App config> -l <Logical DDBOOST SU Exactly in Backup App config> -u <Owner of SU> -p <Password of Owner of SU> -p <ddboost_user_password>  -v

Basic examples:
For Windows: 

C:\ddtools>ddpconnchk -s datadomain1.abc.com -l LSU1 -u ostuser -p xxxxxx  -v

For Linux:

#./ddpconnchk_linux_64 -s datadomain2.com l LSU1 -u dduser -p xxxxxx -v

 
EXAMPLE 1: General check for DD Boost connectivity

#./ddpconnchk_linux_64 -s dd-dm2.abc.com -l LSU1 -u ostuser -p xxxxxx -v

Sample output:

Loading plugin shared library
Could not load ddpi_is_dfc_target_hostname: /lib64/libDDBoost.so: undefined symbol: ddpi_is_dfc_target_hostname
 
Plugin shared library load FAILED
USING BUILTIN DDP PLUGIN INSTEAD
 
DDP CLIENT LIBRARY VERSION 2:6:0:2-366182
 
SERVER: dd-dm2.abc.com
----------------------------------
*** BASIC CONNECTIVITY TEST, dd-dm2.abc.com
1) Translate server name dd-md2-01.abc.com to IP address
Server dd-dm2.abc.com -> IP address 10.xx.xx.xx

2) pmap_getport() for prog MOUNT (100005), vers 3, proto TCP
Portmap (MOUNT) reports TCP port 2052

3) Establish TCP connection to MOUNT service

4) pmap_getport() for prog NFS (100003), vers 3, proto TCP
Portmap(NFS) reports TCP port 2049

5) Establish TCP connection to NFS service

6) Establish DDP connection, for permission check
DDBOOST properties:
DDBOOST enabled = 1
opt_dup_enable = 1
 
Basic DDP Connectivity Test PASSED
 
*** CONNECT SERVER TEST, dd-dm2.abc.com
 
1) Establish DDP connection
DDP Connect Server Test PASSED
*** LIST SUs TEST, dd-md2-01.abc.com
connect_server_with_user_pwd()...
connect_server_with_user_pwd() DONE
 
open_su_list()...
open_su_list() DONE
 
Read SU list...
0> dd-dm2-lsu1
  Bytes available 99025806163968
  Bytes used 1085185994221356
1> dd-dm2-lsu2
  Bytes available 99025806163968
  Bytes used 1085185994221356
2> dd-dm2-lsu3
  Bytes available 99025806163968
  Bytes used 1085185994221356
3> ...
4 SUs total
Read SU list DONE
 
disconnect_server()...
disconnect_server() DONE
 
List SUs Test PASSED
 
No further tests possible.
Specify -l<su_name> to enable
more tests
 
SUCCESS: All tests completed

Notes:

• The shared library message at the beginning is not significant. Ddpconnchk tool has its own integrated library to use.
• The "-v" parameter is for verbose logging, which provides the result of a step-by-step ddpconnchk check. 
• If you prefer not to put the password in command line, run ddpconnchk without. A you are prompted to type in password.
• A successful test shows, hostname that is resolved to the correct IP, ports are open, DD Boost user and password are correct, and the media server can see storage-units on DD. This confirms that DD configuration, and the connectivity between media server and DD is good.
• If ddpconnchk is unable to connect, the error should give you and idea why it was not able to connect.

EXAMPLE 2: Use ddpconnchk to test DD Boost performance.
It tests DD Boost performance by writing a test image. The test image will be automatically removed after the test.
Below is the performance test using one stream.
 

C:\ddtools>ddpconnck -s <DD Hostname Exactly in Backup App config> -l <Logical DDBOOST SU Exactly in Backup App config> -u <Owner of SU> -p <Password of Owner of SU> -T writeimage -i 1g -n <Stream Count to Match with Backup App> -vvv

Notes:
Use 1, 4, 8 etc for Stream Count to see if there is improvement in speed. There may be a saturation point for number of streams. Higher the stream count dos not always provide better throughput.
Skip -p option if password contains conflicting special characters (such as # ? ; : , etc), and type the password when prompted.

Sample output:

.....
*** EXTENDED TEST: WRITE IMAGE, dd-dm2-lsu1/sfadvdfdas00163/1073741824
Cumulative Write Throughput: 85.33 MB/s
Cumulative Read Throughput: 25.60 MB/s

 
To do a ddpconnchk performance check with 10 streams (-n 10):

C:\ddtools>ddpconnchk -s <ddr_name> -u <ddboost_user> -p <ddb_user_password> -l <lsu_name> -T writeimage -i 1g -n 10

• -i 1g determines the size that is written to DDR 1g = 1 GB file which is the maximum size 
• -n 10 determines how many files get created. 
• These files are temporary and are deleted once the test operation has been completed.

EXAMPLE 3: Use ddpconnchk to check connectivity of DD Boost replication (MFR):

user@Affected_Backup_Host# ddpconnck -s <SRC DD Hostname Exactly in Backup App config> -l <SRC SU Exactly in Backup App config> -u <Owner of SRC SU> -p <Password of Owner of SRC SU> -T writeimage -i 1g -n <Stream Count to Match with Backup App> -vvv    (Use 1, 4, 8 etc for Stream Count to see if there is improvement in speed)
user@Affected_Backup_Host# ddpconnck -s <DST DD Hostname Exactly in Backup App> -l <DST SU Exactly in Backup App> -u <Owner of DST SU> -p <Password of Owner of DST SU> -T writeimage -i 1g -n <Stream Count to Match with Backup App>     (Use 1, 4, 8 etc for Stream Count)

user@Affected_Backup_Host# ddpconnck -s <SRC DD Hostname Exactly in AVA config> -l <SRC SU Exactly in AVA config> -u <Owner of SRC SU> -p <Password of Owner of SRC SU> -S <DST DD Hostname Exactly in AVA config> -L <DST SU Exactly in AVA config> -U <Owner of DST SU> -P <Password of Owner of DST SU> -T optdup -i 1g -n <Stream Count to Match with AVA>     (Use 1, 4, 8 etc for Stream Count to see if there is improvement in speed)

 
Notes:
"-S, -U, -P" (the uppercase) is for the second DDR.
This is similar to run ddpconnchk twice, one to DDR1, and one to DDR2, with all lowercase as below. But it is best to run the complete command above.
     

#./ddpconnchk -s dd-dm-src.abc.com -l lsu1 -u user1 -p xxxxxx -v
#./ddpconnchk -s dd-dm-dst.abc.com -l lsu2 -u user1 -p xxxxxx -v

EXAMPLE 4: Run ddpconnchk to check the MFR/optdup throughput.

#ddpconnchk -s dd-dm-src.abc.com -l src_lsu1 -u user1 -p xxxxxx -S dd-dm-dst.abc.com -L dst_lsu1 -U user1 -P xxxxxx -T optdup -i 1g -n 5 -v

*** EXTENDED TEST: OPT DUP
Src: dd-dm-src /src_lsu1
Dst: dd-dm-dst/dst_lsu1
ImgSize: 262144 bytes
NumThrd: 1
Cumulative Filecopy Throughput: 0.12 MB/s
Total read time: 0 seconds

 
EXAMPLE 5:  To list dynamic tracking of Fibre Channel (DFC) devices seen by a client.
 

# ddpconnchk -D scan_all

Sample output:

/dev/sg135:
 Server Name: xxx-xxxxx
 Server ID:   xxxxxxxx
 
 /dev/sg134:
 Server Name: xxxx-xxxxx
 Server ID:   xxxxxxxx
 
 2 Generic SCSI devices
 2 DFC LUN devices
 DFC evaluation completed

• pmap_getport() failed is related to Network environment, such as firewall issue.
• connect_server_user_pwd() FAILED is due to username or password incorrect, or DD Boost access list incorrect.
• You can confirm a DD Boost user by checking the Auto Support (ASUP) registry protocol.ost.user.
• You can confirm username and password by logging into DD using a putty session as DD Boost user, to test its password.
• Use #ddboost access show, and #ddboost ifgroup show config all, to check DD Boost access.