Avamar : Intégration de Data Domain : Échec de la réplication avec : 5009-filecopy operation failed (nfs : I/O error)

La réplication échoue avec les erreurs suivantes :
(exemple extrait des journaux Avamar) :

2020-02-26 11:27:39 avtar Info <42385>: id:1 Range-Replication selected because it is supported since DDOS 6.0 (container.1.cdsf)
2020-02-26 11:27:39 avtar Info <40160>: - Establishing a connection to the Data Domain system with basic authentication (Connection mode: A:0 E:0).
2020-02-26 11:27:39 avtar Info <40160>: - Establishing a connection to the Data Domain system with basic authentication (Connection mode: A:0 E:0).
2020-02-26 11:27:39 avtar Info <40160>: - Establishing a connection to the Data Domain system with basic authentication (Connection mode: A:0 E:0).
2020-02-26 11:27:40 avtar Info <40160>: - Establishing a connection to the Data Domain system with basic authentication (Connection mode: A:0 E:0).
2020-02-26 11:27:41 avtar Error <0000>: Replication failed - id: 4, unexpected exception caught: ddr_replthread:4:replicateSlice: ddp_filecopy_status failed
[24648] [140103118698240] Wed Feb 26 11:27:41 2020
    ddp_filecopy_status() failed, start_offset[0], length[20971520], Err: 5009-filecopy operation failed (nfs: I/O error)
 [24648] [140103120803584] Wed Feb 26 11:27:40 2020
    ddp_filecopy_stop() failed, Err: 5004-nfs filecopy stop failed (nfs: No such file or directory)
...
 [24648] [140103116592896] Wed Feb 26 11:27:40 2020
    ddp_filecopy_stop() failed, Err: 5004-nfs filecopy stop failed (nfs: No such file or directory)
 [24648] [140103180687104] Wed Feb 26 11:27:39 2020
    ddp_access() failed, Path avamar-123456789/STAGING/5be457387ae27688587b250879ad303532b2fd33/BACKUP-C97A790FDAC757D5B883F6E3A5BF0BE9344B0355-1D5EC41EE3438C2, mode 0 Err: 5004-nfs lookup failed (nfs: No such file or directory)
 [24648] [140103180687104] Wed Feb 26 11:27:39 2020
    ddp_access() failed, Path avamar-123456789/STAGING/5be457387ae27688587b250879ad303532b2fd33/BACKUP-C97A790FDAC757D5B883F6E3A5BF0BE9344B0355-1D5EC41EE3438C2, mode 0 Err: 5004-nfs lookup failed (nfs: No such file or directory)
2020-02-26 11:27:41 avtar Error <0000>: Replication failed - id: 5, unexpected exception caught: ddr_replthread:5:replicateSlice: ddp_filecopy_status failed
...
[24648] [140103116592896] Wed Feb 26 11:27:41 2020
    ddp_filecopy_status() failed, start_offset[0], length[20971520], Err: 5009-filecopy operation failed (nfs: I/O error)
 [24648] [140103120803584] Wed Feb 26 11:27:41 2020
    ddp_filecopy_status() failed, start_offset[0], length[20971520], Err: 5009-filecopy operation failed (nfs: I/O error)
...
 [24648] [140103121856256] Wed Feb 26 11:27:40 2020
    ddp_filecopy_stop() failed, Err: 5004-nfs filecopy stop failed (nfs: No such file or directory)
 [24648] [140103119750912] Wed Feb 26 11:27:40 2020
    ddp_filecopy_stop() failed, Err: 5004-nfs filecopy stop failed (nfs: No such file or directory)
 [24648] [140103117645568] Wed Feb 26 11:27:40 2020
    ddp_filecopy_stop() failed, Err: 5004-nfs filecopy stop failed (nfs: No such file or directory)
2020-02-26 11:27:42 avtar Error <18788>: Replication failed - a replication thread encountered error 4910, desc: Unknown exception caught.
2020-02-26 11:27:42 avtar Info <18789>: Backup Replication Stats: original backup size: 0 (0 bytes), precompression size: 0 (0 bytes), network bytes sent: 0 (0 bytes), sent: 0.00%
2020-02-26 11:27:42 avtar Error <18797>: Replication failed - The replicator returned an error.
2020-02-26 11:27:42 avtar Error <18773>: MReplication failed -- backup: 1D5EC41EE3438C2, from DD-6800-1.company.com(1):avamar-134567890 to DD-6800-2.company.com(1):avamar-123456789, DDR result code: 4912, desc: Replication failed
2020-02-26 11:27:43 avtar Warning <18125>: Calling DDR_REPLICATE returned result code:5009 message:I/O error
2020-02-26 11:27:43 avtar Error <10612>: Replication failed -- could not replicate file /cur/5be457387ae27688587b250879ad303532b2fd33/1D5EC41EE3438C2/ddr_files.xml to /STAGING/5be457387ae27688587b250879ad303532b2fd33/BACKUP-C97A790FDAC757D5B883F6E3A5BF0BE9344B0355-1D5EC41EE3438C2/ddr_files.xml, LSU: avamar-123456789, DDR result code: 5009, desc: I/O error
2020-02-26 11:27:43 avtar Info <40176>: - Multi-stream restore summary (current 0, max active 0, max cloned 0, total cloned 0, ddr handle 2): 
2020-02-26 11:27:43 avtar FATAL <0000>: Replication failed
2020-02-26 11:27:43 avtar FATAL <40009>: DDR encountered errors.
2020-02-26 11:27:43 avtar Info <9772>: Starting graceful (staged) termination, DDR_ERROR event received (fatal severity) (wrap-up stage)
2020-02-26 11:27:43 avtar Info <0000>: Entering the 'final' phase of termination, DDR_ERROR need to exit)
2020-02-26 11:27:43 avtar Info <40176>: - Multi-stream restore summary (current 0, max active 0, max cloned 0, total cloned 0, ddr handle 2): 
2020/02/26-18:27:43.18948 [avtar]  INTERNAL ERROR: <0001> assert error (result == MSG_ERR_NONE), /local/jenkins/workspace/server_rollup_hotfix_SLES11-64/src/avtar/ade/adelib/interfaceade.cpp line 396
2020-02-26 11:27:43 avtar Info <5675>: Replicated 0 of 2 backups (0 errors) for client "test-files.company.com"
2020-02-26 11:27:43 avtar Error <6655>: Replicate interrupted.
2020-02-26 11:27:43 avtar Info <7925>: Restored 0 bytes from selection(s) with 0 bytes in 1 files
2020-02-26 11:27:43 avtar Info <7883>: Finished at 2020-02-26 11:27:43 MST, Elapsed time: 0000h:00m:04s
2020-02-26 11:27:43 avtar Info <40176>: - Multi-stream restore summary (current 0, max active 0, max cloned 0, total cloned 0, ddr handle 1): 
2020-02-26 11:27:43 avtar Info <6149>: Error summary: 14 errors: 6655, 18773, 10612, 18797, 40009, 18788, 0(7)
2020-02-26 11:27:43 avtar Info <8468>: Sending wrapup message to parent
2020-02-26 11:27:43 avtar Info <5314>: Command failed (14 errors, 2 warnings, exit code 10007: miscellaneous error)

Bien que l’erreur « Err : Échec de l’opération 5009-filecopy (nfs: I/O error) » est une erreur générique pour les problèmes Data Domain. Une enquête plus approfondie est nécessaire sur Data Domain pour trouver la cause première.

Dans ce cas, le problème était dû au fait que le chiffrement n’était pas configuré de la même manière sur les Data Domains source et cible.

Les événements suivants ont été consignés dans le journal :

02/26 22:01:51.161 (tid 0x7fa4fa49ffe0): filecopy ctx 541 dst DD-6800-2.company.com: ASSOC failed: Encrypted replication incompatible between source and destination (5342) (5342)
02/26 22:01:51.161 (tid 0x7fa4fa49ffe0): NOTICE: MSG-REPL-00508: filecopy ctx 541: Replication: originator disconnected from dst DD-6800-2.company.com
02/26 22:01:51.161 (tid 0x7fa4fa49ffe0): filecopy failed for , errno=5342, errstr=ASSOC failed: Encrypted replication incompatible between source and destination (5342)
02/26 22:01:51.161 (tid 0x7fa4fa49ffe0): nfc_worker - Unrecoverable filecopy for src_path: /data/col1/avamar-123456789/cur/5be457387ae27688587b250879ad303532b2fd33/1D5ECC7D985F54C/container.1.cdsf error = 5342. (Not attempting retry)
02/26 22:01:51.161 (tid 0x7fa4fa49ffe0): ERROR: MSG-NFS-00005: Failed to do a filecopy, err = [5342: ASSOC failed: Encrypted replication incompatible between source and destination (5342), ddr/repl/lrepl_originator.c, filecopy_do_associate, 1585]
02/26 22:01:51.162 (tid 0x7fa4fa49ffe0): nfc_worker - filecopy_send_file completed for task=64 filecopy idx=64 error=5

Cela a été vérifié en vérifiant le paramètre de chiffrement sur les Data Domains source et cible (ou cible) :

Source :

ddboost file-replication option show encryption

Encryption for file-replication is: disabled 

Cible :

ddboost file-replication option show encryption

Encryption for file-replication is: enabled 

1. Vérifiez si le chiffrement doit être activé ou désactivé sur les deux systèmes Data Domain.

2. Connectez-vous à la destination où la configuration du chiffrement est incorrecte.

3. Vérifiez, définissez et vérifiez le chiffrement :

Pour activer le chiffrement :

ddboost file-replication option show encryption

Encryption for file-replication is: disabled
Authentication-mode is: anonymous

ddboost file-replication option set encryption enabled

Encryption for file-replication set to "enabled".

ddboost file-replication option show encryption

Encryption for file-replication is: enabled
Authentication-mode is: anonymous

Pour désactiver le chiffrement :

ddboost file-replication option show encryption

Encryption for file-replication is: enabled
Authentication-mode is: anonymous

ddboost file-replication option set encryption disabled

Encryption for file-replication set to "disabled".

ddboost file-replication option show encryption

Encryption for file-replication is: disabled
Authentication-mode is: anonymous

4. Redémarrez ddboost sur le Data Domain sur lequel le chiffrement a été activé ou désactivé.

ddboost status

DD Boost status: enabled

ddboost disable

The 'ddboost disable' command aborts running DD Boost operations.
        Are you sure? (yes|no) [no]: y
ok, proceeding.

DD Boost disabled.

ddboost enable 

DD Boost enabled.

ddboost status 

DD Boost status: enabled

5. Exécutez à nouveau la réplication.