DLm VTE with Guaranteed Replication enabled causes hung tape mounts

The DLm virtuent application reads the Data Domain report file to determine if a directory is marked eligible for Guaranteed Replication and permits GR activities on those directories.  

01/04/2022 17:45:09 vte2 VT: DLm0409I: Command received: 'SET GR=YES'
01/04/2022 17:45:09 vte2 VT: DLm0160I: All devices Guaranteed Replication set to YES
01/04/2022 17:47:17 vte2 VT: DLm0405I: Device 0295 Display: 'KA14425 ' 'MPRIVATS' (blink) (alternating)
01/04/2022 17:47:18 vte2 VT: DLm0527W: Device 0295 unable to allocate an SL PRIVAT volume: none available(CLASS=CLASS0) (GR)
01/04/2022 17:47:18 vte2 VT: DLm0489E: Device 0295 Error loading volume 'PRIVAT' - no available (CLASS=CLASS0) on replicating FS
01/04/2022 17:47:18 vte2 VT: DLm0487I: Device 0295 volume PRIVAT mount will be retried

...
This pattern continues for all tape mounts to all drives enabled for GR for the affected tapelib directories.  If GR is configured for all VTE devices, this creates a data unavailable condition.

Virtuent / vtd running on the affected VTE considers all GR directories to be ineligible for guaranteed replication.

Virtuent / vtd "caches" or maintains a local copy of the DD report file from /opt/DLm/DD-1/dlmconfig/ddReport.txt 
The file is read periodically and the virtuent copy updated.  In this case the virtuent application is reading the file once and not updating it's copy, and vtd considers nothing "eligible" for guaranteed replication due to the stale copy of ddReport.txt

At a previous time, a (Guaranteed) Replication error occurred which caused Virtuent to create a 'replication sync error file accompanying error DLm0548E.  With that error file, virtuent, subsequent GR requests are failed for the affected tapelib directory.  At some point, virtuent erases the error file in /opt/DLM/DD-1|2/dlmconfig/.GR but the failed replication status remains in virtuent until the cached copy of ddreport is updated by config install or vtd restart


2021-10-07T16:13:30.341623-05:00 vte1 dlm_gr[61516]: DLmTools:I: Started processing request for source {data/col1/D0010}: /opt/DLm/DD-1/dlmconfig/.GR/dd1s:#@,data#@,col1#@,D0010#@,tapelibD0010#@,DD1_P1_FS7 vte2.0204.A07961.1633641209
2021-10-07T16:13:30.341715-05:00 vte1 dlm_gr[61516]: DLmTools:I: Synchronizing data/col1/D0010...
2021-10-07T16:13:30.341795-05:00 vte1 dlm_gr[61516]: DLmTools:I: DD : started replication sync mtree://CDC-8500.<redacted>.com/data/col1/D0010
2021-10-07T16:15:30.385659-05:00 vte1 dlm_gr[61516]: DLmTools:E: Error: Command replication sync mtree://CDC-8500.<redacted>.com/data/col1/D0010 failed
2021-10-07T16:15:30.386074-05:00 vte1 dlm_gr[61516]: DLmTools:E: Error: Error: DD command returned rcode 255
2021-10-07T16:15:30.386158-05:00 vte1 dlm_gr[61516]: DLmTools:E: -------------------
2021-10-07T16:15:30.386235-05:00 vte1 dlm_gr[61516]: DLmTools:E: Connection to dd1 closed by remote host.
2021-10-07T16:15:30.386311-05:00 vte1 dlm_gr[61516]: DLmTools:E: -------------------
2021-10-07T16:15:30.386390-05:00 vte1 dlm_gr[61516]: DLmTools:E: Error: failed to sync replication session mtree://CDC-8500.<redacted>.com/data/col1/D0010. RC: 255. Output:#012Connection to dd1 closed by remote host.
2021-10-07T16:15:30.386464-05:00 vte1 dlm_gr[61516]: DLmTools:E: Error: failed to sync data/col1/D0010 to mtree://CDC-8500.<redacted>.com/data/col1/D0010: 2
2021-10-07T16:15:30.445644-05:00 vte1 dlm_gr[61516]: DLmTools:I: Info: Creating error file '/opt/DLm/DD-1/dlmconfig/.GR/error dd1s:#@,data#@,col1#@,D0010#@,tapelibD0010#@,DD1_P1_FS1 vte1.0131.B95896.1633641209'
2021-10-07T16:15:30.450184-05:00 vte1 dlm_gr[61516]: DLmTools:I: Info: Deleting original replication request file '/opt/DLm/DD-1/dlmconfig/.GR/dd1s:#@,data#@,col1#@,D0010#@,tapelibD0010#@,DD1_P1_FS1 vte1.0131.B95896.1633641209'
2021-10-07T16:15:30.756688-05:00 vte1 dlm_gr[61516]: DLmTools:I: Info: Creating error file '/opt/DLm/DD-1/dlmconfig/.GR/error dd1s:#@,data#@,col1#@,D0010#@,tapelibD0010#@,DD1_P1_FS7 vte2.0204.A07961.1633641209'
2021-10-07T16:15:30.760731-05:00 vte1 dlm_gr[61516]: DLmTools:I: Info: Deleting original replication request file '/opt/DLm/DD-1/dlmconfig/.GR/dd1s:#@,data#@,col1#@,D0010#@,tapelibD0010#@,DD1_P1_FS7 vte2.0204.A07961.1633641209'
2021-10-07T16:15:30.767417-05:00 vte1 dlm_gr[61516]: DLmTools:E: Error: failed to sync data/col1/D0010
2021-10-07T16:15:30.822095-05:00 vte1 dlm_gr[73147]: DLmTools:E: master: child 61516 failed to sync repl data/col1/D0010 with code 2, signal 0
2021-10-07T16:15:31.032082-05:00 vte1 dlm_gr[67633]: DLmTools:I: Started processing request for source {data/col1/D0010}: /opt/DLm/DD-1/dlmconfig/.GR/dd1s:#@,data#@,col1#@,D0010#@,tapelibD0010#@,DD1_P1_FS5 vte1.0127.C08798.1633641321

As of the authoring of this article, January of 2022, this issue will be fixed in a later DLm code release.  

As a workaround, a vtd stop / vtd start will permit GR activation
vtd stop / vtd start commands may be issued via

DLm System Manager ; Command interface
Putty / command line