IDPA 2.1: NDMP Restore Fails with Error "Backup is invalid or not an NDMP backup"

The following errors can be seen in the NDMP restore job log: 

2018-12-06 15:03:04 avndmp Info <6444>: [avndmp_assist] Plugin exited with 'code 157: miscellaneous error'
2018-12-06 15:03:04 avndmp Error <11799>: [avndmp_assist] Backup is invalid or not a NDMP backup.
2018-12-06 15:03:04 avndmp Info <11780>: [avndmp_ctl_sup] Global avndmp_assist exit code is: 161

To verify the DD Boost plugin version on Avamar, run the following command:

strings /usr/local/avamar/lib/libDDBoost.so | grep "[0-9]\.[0-9]\.[0-9]\.[0-9]"

Or  

grep -i engine ddrmaint.log | tail -1

In DDFS log (/ddr/var/log/debug/ddfs.info) on Data Domain, the following errors are seen:

12/06 15:02:45.099 (tid 0x90a9b30): nfs_rpc_svc_idx0 accepted cfa 255 from 10.12.255.85:39910
12/06 15:02:45.103 (tid 0x7ff35b151220): nfsproc3_ostmntsec_3_svc: connection failed permission (corrupted credentials) from host 10.12.255.85 - ost_decrypt_mnt_sec_request(): failed to finalize plain text (101077092, error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt)

   

Avamar server 7.5.0 uses DD Boost library version 3.4.0.2. This issue exists on DD Boost library 3.4.0 and is fixed on 3.4.1.

Avamar server version 7.5.1 has DD Boost library version 3.4.1.1 embedded in it. 

Starting from version 3.4 plugin, Data Domain does authentication using preshared-keys (PSKs) generated from the password. The client generates a PSK from the application-supplied password, encrypts some authentication information, and sends it to DDR. The DDR tries to generate the same PSK (since it already knows the password for a user) and decrypt the contents the client sent and validate.

With DD Boost ifgroup enabled, the data domain falls into the reconnected path, which calls the PSK connection path and relies on the password crypt_hash stored in the nfs_conn structure. On DD Boost plugin 3.4.0.2 though, Data Domain does not generate the password crypt_hash again during reconnection. The PSK key is not the same on the client and DDR, hence the decryption failure.

Workaround:
Use one of the following methods to get around this issue.    

Method 1:
Use username-password authentication. (Disable session security on the Avamar Server)
 
Follow the information in article 000067229: Avamar 7.3 and later: Backups or replications fails with certificate error to download the SessionSecurityConfiguration avp (Log in as a registered Dell Support user may be required to view this article.):

• Prechecks before installing avp:   
  • Verify maintenance task, backup, restore, and replication is not running while installing SessionSecurityConfiguration avp
  • Verify that the Avamar server has a current checkpoint

1. Go to the Avamar Installation Manager (AVI) (http://<IP/fqdn of the Avamar server/avi) page in a Web Browser and log in as root user:    
   1. Under the Maintenance tab, select SessionSecurityConfiguration and click Run:    

2. Select Disabled mode from the drop-down menu under the Security settings tab:    

2. Click Continue.

Follow the information in article 000222279: Avamar: Manage Session Security Settings with Avinstaller Installation Package (AVP)

Method 2: 
Upgrade the Avamar Client Catalog to version 7.5.1-101 and then upgrade the NDMP Accelerator node to the same version (7.5.1).

Method 3:
Disable the DD Boost ifgroups on the Data Domain.
   

1. Log in to the Data Domain:   
   1. Run the below command to view the existing ifgroups on the Data Domain: 

ddboost ifgroup show config all

Group-name    Status        Interface      Clients     Replication
-----------   --------      ---------      -------     -----------
default       disabled           0         1             0
backupgroup   enabled            2         0             0
-----------   --------      ---------      -------     -----------

Group-name    Status     Interfaces
-----------   -------   -----------
backupgroup   enabled   10.x.x.x
backupgroup   enabled   192.x.x.x
-----------   -------   -----------

 

2. Run the following command to disable the ifgroup in question: 

ddboost ifgroup disable <group-name>