Sybase server backup through NetWorker/NMDA fails with I/O errors ending with the crash of the Sybase sybmultbuf process

• NetWorker Module for Database Applications (NMDA) Sybase backup fails
• nsrdasv logged these errors:

24146:nsrdasv: Error from server SL4_BS: Msg 419402, Level 2, State 1
Backup Server: 4.194.2.1: Internal error. The srv_select() function was called to complete an I/O for 2240 bytes. After retrying 2241 times only 0 bytes have been read.24146:nsrdasv: Error from server SL4_BS: Msg 414102, Level 2, State 75
Backup Server: 4.141.2.75: [0] The 'read' call failed for database/archive device while working on stripe device 'nsrsyb::model..3.1./nsr/apps/tmp/sybtmp_1671421.txt::000' with error number 0 (Success). Refer to your operating system documentation for further details.24146:nsrdasv: Error from server SL4_BS: Msg 406203, Level 3, State 1
Backup Server Internal Error: 4.62.3.1: Multibuffering subprocesses died, archive nsrsyb::model..3.1./nsr/apps/tmp/sybtmp_1671421.txt::000.24146:nsrdasv: Error from server SL4_BS: Msg 414102, Level 2, State 66
Backup Server: 4.141.2.66: [0] The 'write' call failed for database/archive device while working on stripe device 'nsrsyb::model..3.1./nsr/apps/tmp/sybtmp_1671421.txt::000' with error number 32 (Broken pipe). Refer to your operating system documentation for further details.24146:nsrdasv: Error from server SL4_BS: Msg 406203, Level 3, State 8
Backup Server Internal Error: 4.62.3.8: Multibuffering subprocesses died, archive nsrsyb::model..3.1./nsr/apps/tmp/sybtmp_1671421.txt::000.24146:nsrdasv: Error from server SL4_BS: Msg 603102, Level 2, State 2
Backup Server: 6.31.2.2: Volume rejected.24146:nsrdasv: Error from server SL4_BS: Msg 101402, Level 2, State 2
Backup Server: 1.14.2.2: Unrecoverable I/O or volume error. This DUMP or LOAD session must exit.24146:nsrdasv: Error from server SL4_BS: Msg 414102, Level 2, State 66

• sybmultbuf cores when the NMDA API is doing some encryption using Openssl:

 A00: SYBMULTBUF: bs_trap; process received signal 11
 A00: 0x7fc08ac64b20 : (killpg+0x40) [0x7fc08ac64b20]
 A00: 0x7fc0898fba79 : (EVP_MD_CTX_clear_flags+0x9) [0x7fc0898fba79]
 A00: 0x7fc0898e9461 : (EVP_DigestInit_ex+0x21) [0x7fc0898e9461]
 A00: 0x7fc08990af31 : (HMAC_Init_ex+0x211) [0x7fc08990af31]
 A00: 0x7fc088a1a219 : (comssl_OpenSSL_hmac_init+0x69) [0x7fc088a1a219]
 A00: 0x7fc0889ab8f9 : (cryptoiface_get_hmac+0x89) [0x7fc0889ab8f9]
 A00: 0x7fc0889ac1c5 : (gsslgtov1_export_sec_context+0x285) [0x7fc0889ac1c5]
 A00: 0x7fc0889af5b0 : (gsslgtov1_init_sec_context+0xe20) [0x7fc0889af5b0]
 A00: 0x7fc0889a1dbc : (authgss_create+0x5dc) [0x7fc0889a1dbc]
 A00: 0x7fc0889af6e0 : (gsslgtov1_init_sec_context+0xf50) [0x7fc0889af6e0]
 A00: 0x7fc0889b0553 : (gsslgtov1_acquire_cred+0xad3) [0x7fc0889b0553]
 A00: 0x7fc08898673c : (lgto_parms+0x127c) [0x7fc08898673c]
 A00: 0x7fc088987909 : (acquire_lg_authgen_cred_lsp_host_override+0x299) [0x7fc088987909]
 A00: 0x7fc088987b87 : (acquire_lg_authgen_cred+0x37) [0x7fc088987b87]
 A00: 0x7fc0889822ba : (rap_bind+0x41a) [0x7fc0889822ba]
 A00: 0x7fc088992f71 : (resdb_net_encrypt+0xd1) [0x7fc088992f71]
 A00: 0x7fc0888789d4 : (nsr_ladb_net+0xc4) [0x7fc0888789d4]
 A00: 0x7fc088878379 : (nsr_findserver+0x109) [0x7fc088878379]
 A00: 0x7fc0887ad5b3 : (_nwbsa_open_options_session+0x243) [0x7fc0887ad5b3]
 A00: 0x7fc08879d145 : (BSAInit+0x275) [0x7fc08879d145]
 A00: 0x7fc08879b273 : (init_bsa+0x33) [0x7fc08879b273]
 A00: 0x7fc088797e1e : (syb_open+0xa8e) [0x7fc088797e1e]
 A00: 0x44d691 : (api_open+0x501) [0x44d691]
 A00: 0x42924f : (ctl_job+0x249) [0x42924f]
 A00: 0x425fe8 : (archive_proc+0x593) [0x425fe8]
 A00: 0x42526a : (sybmbuf_main+0x66e) [0x42526a]
 A00: 0x424bfa : (main+0x1b) [0x424bfa]
 A00: 0x7fc08ac50cf3 : (__libc_start_main+0xf3) [0x7fc08ac50cf3]
 A00: 0x424b09 : (_start+0x29) [0x424b09]
 D00: SYBMULTBUF ERROR: Emulator interprocess communication failed with error state = 9, error code=0, system message=Pipe I/O returned 0 bytes.
 Open Server Error: 16248.10.0: Error in srv_select() - file descriptor 12 is no longer active
 Backup Server: 4.194.2.1: Internal error. The srv_select() function was called to complete an I/O for 2240 bytes. After retrying 2241 times only 0 bytes have been read.
 Backup Server: 4.141.2.75: [0] The 'read' call failed for database/archive device while working on stripe device 'nsrsyb::...1::000' with error number 0 (Success). Refer to your operating system documentation for further details.
 Backup Server Internal Error: 4.62.3.1: Multibuffering subprocesses died, archive nsrsyb::...1::000.
 Backup Server: 4.141.2.66: [0] The 'write' call failed for database/archive device while working on stripe device 'nsrsyb::...1::000' with error number 32 (Broken pipe). Refer to your operating system documentation for further details.
 Backup Server Internal Error: 4.62.3.8: Multibuffering subprocesses died, archive nsrsyb::...1::000.
.....
Jul 6 20:23:36 2022: D02: SYBMULTBUF ERROR: Emulator interprocess communication failed with error state = 9, error code=0, system message=Pipe I/O returned 0 bytes.
Jul 6 20:23:36 2022: Open Server Error: 16248.10.0: Error in srv_select() - file descriptor 17 is no longer active
Jul 6 20:23:36 2022: Open Server Error: 16248.10.0: Error in srv_select() - file descriptor 17 is no longer active

$ ldd ./backupserver
        linux-vdso.so.1 (0x00007ffe6d6db000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007faea79ab000)
        librt.so.1 => /lib64/librt.so.1 (0x00007faea77a3000)
        libm.so.6 => /lib64/libm.so.6 (0x00007faea7421000)
        libnsl.so.1 => /lib64/libnsl.so.1 (0x00007faea7207000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007faea6fe7000)
        libc.so.6 => /lib64/libc.so.6 (0x00007faea6c22000)
        /lib64/ld-linux-x86-64.so.2 (0x00007faea7baf000)
        libtirpc.so.3 => /lib64/libtirpc.so.3 (0x00007faea69ef000)
        libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007faea679a000)
        libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007faea64b0000)
        libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007faea6299000)
        libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007faea6095000)
        libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007faea5e84000)
        libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007faea5c80000)
        libcrypto.so.1.1 => /lib64/libcrypto.so.1.1 (0x00007faea5797000)
        libresolv.so.2 => /lib64/libresolv.so.2 (0x00007faea5580000)
        libselinux.so.1 => /lib64/libselinux.so.1 (0x00007faea5356000)
        libz.so.1 => /lib64/libz.so.1 (0x00007faea513e000)
        libpcre2-8.so.0 => /lib64/libpcre2-8.so.0 (0x00007faea4eba000)

Workaround: 

Preload the NMDA libcrypto.so.1.0.0  library in the Sybase backupserver RUN script. Restart the Sybase backupserver,
 

1. On the Sybase client, log in as a Sybase user.
2. Go to the Sybase installation home directory
3. Source the Sybase environment script.

$ . ./SYBASE.sh

4. Export the LD_PRELOAD variable in the RUN script. Save the changes.

$ cd $SYBASE/$SYBASE_ASE/install
$ vi RUN_<SERVER>_BS
# Multibuf executable path:     /home/sybase16sp4/ASE-16_0/bin/sybmultbuf
# Backup Server name:   DELLSP4_BS
#
LD_PRELOAD=/usr/lib/nsr/apps/lib64/libcrypto.so.1.0.0
export LD_PRELOAD
/home/sybase16sp4/ASE-16_0/bin/backupserver \
-e/home/sybase16sp4/ASE-16_0/install/DELLSP4_BS.log \
-N25 \

5.  Restart the Sybase backupserver.

$ cd $SYBASE/$SYBASE_OCS/bin
$ Isql -U<user> -P<Password> -S<Sybase Server>
Isql> shutdown SYB_BACKUP
Isql> quit

$ cd $SYBASE/$SYBASE_ASE/install
$ startserver -f RUN_<Sybase server>_BS

$ ./showserver