How to troubleshoot Data Domain DDBoost connectivity and performance

How to troubleshoot DataDomain DDBoost connectivity and performance

Summary: To provide the detailed usage for DDBoost connectivity checking tool (ddpconnchk).

The ddpconnchk tool can be used for troubleshooting the following issues:

• A media server cannot connect to a specific DDR via DDBoost protocol. (This also applies to DDBoost/RMAN plug-in can not connect to DDR)
• A media server failed to backup to DDR via DDBoost protocol
• DDBoost MFR (Managed File Replication) failed
• DDBoost backup/restore performance slow
• DDBoost MFR performance slow

Contact support to get the ddpconnchk tool

Notes:

The ddpconnchk tool needs to be ran on the media server or client directly connecting to the DD via DDBoost protocol.


First find out the OS/bits of the media server in question. Based on media server's OS/bits, download the corresponding ddpconnchk file, 
and put it on the media server.

The ddpconnchk tool is a executable file, so it cannot be sent as email attachment.

HOW TO MAKE DDPCONNCHK READY TO RUN ON MEDIA SERVER:

• Winscp or scp to copy the file from his desktop to the media server.
• No installation required ddpconnchk is a standalone executable
• gunzip or Winzip etc. may be used to extract package.
• Executable permissions will be required to run ddpconnchk, ie chmod +x 
• ddpconnchk may be copied and executed in any directory the user is authorized. ie   /tmp, or C:\EMC, or C:\ddtools etc......

How to put ddpconnchk on your client:

Linux/UNIX Operating Systems:

1. Log in to the system where ddpconnchk was copied to. Go to the directory that the package was copied to using the cd command.
2. If the downloaded package is .tar file then you would need to untar it via tar -xf <ddpconnchk_YOUR_OS_YOUR_BIT.tar>
3. Set the permissions of ddpconnchk: chmod 755 ddpconnchk*

Example: 
[root@hostname ~]# tar -xf <ddpconnchk_YOUR_OS_YOUR_BIT.tar>
[root@hostname ~]# chmod 755 ddpconnchk*
[root@hostname ~]# ls -l | grep ddpconnchk
-rwxr-xr-x. 1 59899 59899 3679696 Feb  2 22:54 ddpconnchk
-rw-r--r--. 1 root  root  3696640 May  5  2022 ddpconnchk_linux_x86_64.tar
-rw-r--r--. 1 59899 59899   12086 Feb  2 22:44 ddpconnchk_README.txt

Windows Operating Systems:

1. Log in to the Windows system the ddpconnchk_win_64.zip was copied to. Using Windows File Explorer go to the directory that the package was copied to.
2. Extract the ddpconnchk_win_64.zip into a new directory. 
3. Depending on your environment you may need to open an administrative command prompt to the ddpconchk directory. This can be done from File->Open Command Prompt->Open Command Prompt As Administrator.

Note: The package includes a README.txt with additional information and examples for diagnosing issues.

EXAMPLE 1: General check for DDBoost connectivity:

If you prefer not to put the password in command line, run ddpconnchk without -p <ddboost_user_password>. You will be prompted to type in your DDBoost user password.

For Windows:
C:\ddtools>ddpconnchk -s <ddr_name> -u <ddboost_user> -p <ddboost_user_password> -v 

For Linux/Unix:
#./ddpconnchk_linux_64 -s <ddr_name> -u <ddboost_user> -p <ddboost_user_password> -v
 

Common Errors:  Please check all of the errors listed in the summary following the dppconnchk test.

********************************************************
** DDP connect FAILED
**   Servername: <DD SERVER name or IP address>
**   Username: <DDBoost User ID>
**   Password: *********
**   Error: the user has insufficient access rights
**
** - Verify OST is licensed and enabled on the server
** - Verify username/passwd matches values
**   configured on server
** - Verify that access by this host is allowed
********************************************************
DDP Connect Server Test FAILED

a)  DDBoost should be enabled, if it is not enabled, then please ask the customer to enable.

Good State:

# ddboost status
DD Boost status: enabled

Bad state:

# ddboost status
DD Boost status: disabled

# ddboost enable
DD Boost enabled.

b) Ensure that the password attempted with ddpconnchk is correct and has not expired.  (sometimes you may need to put either "" around it or '')

# user password aging show
User                Password       Minimum Days     Maximum Days     Warn Days       Disable Days   Status
                    Last Changed   Between Change   Between Change   Before Expire   After Expire
-----------------   ------------   --------------   --------------   -------------   ------------   -------
EMCTEST1            Feb 27, 2023   0                99999            7               never          enabled

## user show list
User list from node "localhost".
Name                Uid   Role       Last Login From   Last Login Time            Status    Disable Date
-----------------   ---   --------   ---------------   ------------------------   -------   ------------
EMCTEST1            507   admin      <unknown>         never                      enabled   never

c) Ensure that the client is allowed to connect to the DD.  In some cases, a DDBoost plugin upgrade or a change in DNS setting on the client or the environment can cause the detected hostname to not match previous settings.  Below example shows both wild card ("*") and specific clients.   If the wildcard match is not present, please check in ddfs.info for "did not match access list entries" around the time of the failed tests.

# ddboost clients show config
Client               Encryption Strength   Authentication Mode
------------------- -------------------   -------------------
*                    none                  none
MyClient.mycomp.com  none                  none
-------------------  -------------------   -------------------
(**) The global security settings take precedence over these client(s) specific settings.

# Example from ddfs.info:

07/03 14:59:25.659490 [7ff122a44170] WARNING: Client MyOtherClient.mycomp.com did not match access list entries
07/03 14:59:25.659521 [7ff122a44170] nfsproc3_ost_mnt_3_svc: client access denied for 10.10.10.10 version 7.0

EXAMPLE 2: Use ddpconnchk to test DDBoost performance for backups & restores:

It will test DDBoost performance by writing test image. The test image will be automatically removed after the test. Below is the performance test using 1 stream.

For Windows:
C:\ddtools>ddpconnchk -s <ddr_name> -u <ddboost_user> -p <ddboost_user_password> -l <ddboost_storage-unit_name> -T writeimage -i 1g

For Linux/Unix:
#./ddpconnchk_linux_64 -s <ddr_name> -u <ddboost_user> -p <ddboost_user_password> -l <ddboost_storage-unit_name> -T writeimage -i 1g

To do a ddpconnchk performance check with additional streams you can do (-n #):

For Windows:
C:\ddtools>ddpconnchk -s <ddr_name> -u <ddboost_user> -p <ddb_user_password> -l <ddboost_storage-unit_name> -T writeimage -i 1g -n 5

For Linux/Unix:
#./ddpconnchk_linux_64 -s <ddr_name> -u <ddboost_user> -p <ddboost_user_password> -l <ddboost_storage-unit_name> -T writeimage -i 1g -n 5

• -i 1g determines the size written to DDR 1g = 1GB file which is the maximum size 

• -n 5 determines how many files will be created. 

• These files are temporary and will be deleted at the end of test operation

• A successful test will show, hostname resolved to the correct IP, ports are open, DDBoost user/password are correct, and the media server can see the storage-units on DD. This confirms DD configuration is good, and the connectivity between media server and DD is good.

• If ddpconnchk is unable to connect the error should give you and idea why it was not able to connect.

EXAMPLE 3: Use ddpconnchk to check connectivity of DDBoost replication (MFR):

Linux/Unix:
#./ddpconnchk -s <ddr_name> -u <ddboost_user> -p <ddboost_user_password> -l <ddboost_storage-unit_name> -S <ddr_name> -U <ddboost_user> -P <ddboost_user_password> -L <ddboost_storage-unit_name> -T optdup -v

Windows:
C:\ddtools>ddpconnchk -s <ddr_name> -u <ddboost_user> -p <ddboost_user_password> -l <ddboost_storage-unit_name> -S <ddr_name> -U <ddboost_user> -P <ddboost_user_password> -L <ddboost_storage-unit_name> -T optdup -v

Notes:
"-S, -U, -P" (the uppercase) is for the 2nd DDR.

This is similar to run ddpconnchk twice, one  to DDR1, and one to DDR2, with all lowercase as below. But it is best to run the complete
command above.
      
EXAMPLE 3: Run ddpconnchk to check the MFR/optdup throughput:

C:\ddtools>ddpconnchk -s <ddr_name> -u <ddboost_user> -p <ddboost_user_password> -l <lsu_name> -S <ddr_name> -U <ddboost_user> -P <ddboost_user_password> -L <ddboost_storage-unit_name> -T optdup -i 1g -n 5 -v

#./ddpconnchk -s <ddr_name> -u <ddboost_user> -p <ddboost_user_password> -l <lsu_name> -S <ddr_name> -U <ddboost_user> -P <ddboost_user_password> -L <ddboost_storage-unit_name> -T optdup -i 1g -n 5 -v


EXAMPLE 4: For DDVTL to list DFC devices seen by a client:

# ddpconnchk -D scan_all  /dev/sg135:  Server Name: xxx-xxxxx  Server ID:   xxxxxxxx  /dev/sg134:  Server Name: xxxx-xxxxx  Server ID:   xxxxxxxx  

2 Generic SCSI devices

 2 DFC LUN devices  
DFC evaluation completed

Additional Information:

If ddpconnchk failed, check the specific error message for further troubleshooting:

• pmap_getport() failed is mainly related to Network eviroment, such as firewall issue.

• connect_server_user_pwd() FAILED is mainly due to user/password incorrect, or DDBoost access list incorrect.

• You can confirm DDBoost user by checking ASUP registry protocol.ost.user

• You can confirm user/password by logging into DD via a putty session as DDBoost user, to test its password.

• Use #ddboost access show, and #ddboost ifgroup show config all, to check DDBoost access

Usage for ddpconnchk syntax:

-s <server_name/ip> # For primary server (local, opt-dup source DD system)-u <username>
-p <passwd>
-l <lsu_name>
-S <server_addr/ip> # For secondary server (remote, opt-dup target DD system)
-U <username>
-P <passwd>
-L <lsu_name>
-T optdup | writeimage # Select extended test
-v # Verbose output
-B # Use builtin OST API (not libstspiDataDomain)