Data Domain : How to troubleshoot Boostfs Installation, configuration and mount issues

BOOSTFS is a software component that may be installed in some operating systems, which provides the ability to run BOOST backups to a DataDomain without using any particular backup application, while at the same time, leveraging some of the features that come with using a BOOST-enabled backup application such as Netbackup, Networker and Avamar.

For more "to the point" installation and utilization details, check the Data Domain: Expedited configuration steps and details for DataDomain BOOST FS KB article.

This KB however contains some of the typical issues that may be found when configuring and using BOOSTFS, for customers to have a quick reference for those typical errors, error messages, and the way to solve them. Typical such errors would be "Unable to install Boostfs onto Linux server", "Boostfs mount fails with error" or "Backup or restore fails with error", to name a few.

 

1. Required dependency missing when installing the BOOST FS software

If there is a failure to install the provided BOOSTFS software RPM in the client Linux operating system, first determine which are the required package dependencies for the BOOSTFS software:

# rpm -qpR DDBoostFS-1.0.0.1-539441.rhel.x86_64.rpm fuse >= 2.8 fuse-libs >= 2.8 /bin/sh /bin/sh /bin/sh /bin/sh rpmlib(PayloadFilesHavePrefix) <= 4.0-1 rpmlib(CompressedFileNames) <= 3.0.4-1

In the example above the package is asking for the FUSE (Filesystem in USEr space) binaries and libraries to be installed (BOOSTFS is based on Linux's FUSE), as well as any package providing a shell (sh) and the default RPM libraries. Hence, this BOOSTFS package should install normally in the Linux system the "rpm" has been run on.


2. Cannot mount /mnt/mountpoint/: unexpected error 

When trying to mount the remote DD storage unit locally in the Linux client through BOOSTFS, you may get the error above. Review the BOOSTFS log file in the Linux client. For example, in the first example below, the lockbox file doesn't contain an entry for the host dd2500-abc.datadomain.com:

# less /opt/emc/boostfs/log/ddboostfs_0_0.log Jul 29 03:45:25 22795 3267069888 [E] bfs_lockbox_get_user_info: Failed to find key entry dd2500-abc.datadomain.com:LSU2 in config file /opt/emc/boostfs/lockbox/boostfs.lockbox. The requested Lockbox entry could not be found. Jul 29 03:45:25 22795 3267069888 [E] bfs_conn_lookup: connection lookup failed for node 0 file /.boostfs_sysinfo. (0 connections) Jul 29 03:45:25 22795 3267069888 [I] bfs_node_lookup: failed to obtain a connection for file /.boostfs_sysinfo Jul 29 03:45:25 22795 3267069888 [E] bfs_initialize_mntopts: initialization failed

The solution in this case would be to add a lockbox entry for the destination DD host and storage unit to be mounted locally. Syntax would be similar to the example below :

# /opt/emc/boostfs/bin/boostfs lockbox set -u sysadmin -d dd2500-abc.datadomain.com -s LSU2 Enter storage unit user password: Enter storage unit user password again to confirm: Lockbox entry set

Another possible error that can be seen in the local BOOSTFS client logs is the DD hostname not resolving from the client, for example :

# less /opt/emc/boostfs/log/ddboostfs_0_0.log Jul 29 04:05:50 22882 3322156992 [E] bfs_conn_open: connect failed (0 connections): 5037 Jul 29 04:05:50 22882 3322156992 [E] bfs_conn_lookup: connection lookup failed for node 0 file /.boostfs_sysinfo. (0 connections) Jul 29 04:05:50 22882 3322156992 [I] bfs_node_lookup: failed to obtain a connection for file /.boostfs_sysinfo Jul 29 04:05:50 22882 3322156992 [E] bfs_initialize_mntopts: initialization failed

 
Solution in this case would be to either add a static hosntame to IP mapping to the Linux client's /etc/host file, or configure the mapping in the DNS server being used.


3. The mount point /mnt/mountpoint is nonempty.
BoostFS cannot be mounted on a nonempty mount point. Please try mounting on an empty mount point.

The above message means the mount point specified (/mnt/mountpoint) can not be used to show the remote DD storage unit, as the mount point can't have another filesystem mounted on it previously, or have any contents. /mnt/mountpoint/ must be an empty and unused directory in the Linux client. If the path already has a mount you must specify another mount point. For example:

# mount /dev/mapper/vg00-lv_root on / type ext4 (rw) /dev/sda1 on /boot type ext4 (rw) boostfs on /mnt/mountpoint type fuse.boostfs (rw,nosuid,nodev)

In this case the remote DD storage unit is already mounted under /mnt/mountpoint/ and hence can't be mounted again at the same directory.


4. DDBoostfs mount hungs (may take even more than 10 minutes to return)

This could occur when a firewall in between the Linux client and the target DD device is dropping traffic to TCP ports 111 and 2049. Make sure the Linux client can reach the DD to TCP ports 111 and 2049 and then try again, for example:

# /opt/emc/boostfs//bin/boostfs mount -d dd2500.domain.com -s LSU2 /mnt/mountpoint/

This is an example of what you'd see in the Linux client BOOSTFS logs in such a case :

# less /opt/emc/boostfs/log/ddboostfs_0_0.log Jul 27 06:34:53 32762 3724339136 [E] bfs_conn_open: connect failed (0 connections): 5037 Jul 27 06:34:53 32762 3724339136 [E] bfs_conn_lookup: connection lookup failed for node 0 file /.boostfs_sysinfo. (0 connections) Jul 27 06:34:53 32762 3724339136 [I] bfs_node_lookup: failed to obtain a connection for file /.boostfs_sysinfo Jul 27 06:34:53 32762 3724339136 [E] bfs_initialize_mntopts: initialization failed

5. Operation not permitted when trying to access or list contents under the mount point

# ls -l /mnt/mountpoint/ ls: reading directory .: Operation not permitted total 0

Problem could be there is no TCP connectivity to the remote DD ports 111 and 2049, or that the lockbox authentication has somehow expired. Make sure the Linux client can still reach to the DD on the specified ports and try setting up the lockbox for the remote DD, storage unit and BOOST user again.


6. Insufficient access to or storage-unit does not exist, using kerberos authentication.

# boostfs mount -s LSU3 -d dd2500.domain.com /mnt/mountpoint/ Insufficient access to or storage-unit LSU3 does not exist

When using Kerberos authentication for the lockbox, the most likely reason for the error would be the kerberos authentication ticket to have expired. Check the mentioned ticket from Linux such as below :

# /opt/emc/boostfs/bin/boostfs kerberos query -s LSU3 -u username Client Principal: username@DOMAIN.COM Valid Starting: Tue Aug 23 11:02:49 2016 Expires: Tue Aug 23 21:01:07 2016 Renew Until: Tue Aug 30 11:02:49 2016 Service Principal: krbtgt/DOMAIN.COM@DOMAIN.COM

On the DDR verify the assigned BOOST user to the storage-unit matches the user on Linux client when setting up the lockbox and mounting BOOSTFS (the AD username and storage-unit user name must be exactly the same):

# ddboost storage-unit show Name Pre-Comp (GiB) Status User Report Physical Tenant-Unit Size (MiB) --------------- -------------- ------ -------- --------------- ----------- LSU3 0.0 RW username - - --------------- -------------- ------ -------- --------------- -----------

Also take care to confirm the date / time in the DD, the Linux client and the Kerberos server (AD) is not more than 5 minutes apart, or issues with tickets expired due to inconsistent time may occur.

Below is an example of what we would see in the DD logs if time was more than 5 minutes difference :

# log watch debug/ddfs.info 08/23 18:52:09.654 (tid 0x7f738141fb00): nfs3 accepted 3000004b6 552 from 10.64.229.125:55042 08/23 18:52:09.660 (tid 0x7f738140c890): [dd_rpc2_nfs x3000004b6] dd_rpc_gss_print_error:117 - event gss_error, gss_accept_sec_context failed: maj=0xd0000, min=0x96c73a25 08/23 18:52:09.660 (tid 0x7f738140c890): [dd_rpc2_nfs x3000004b6] dd_rpc_gss_print_error:126 - event gss_error, Unspecified GSS failure. Minor code may provide more information 08/23 18:52:09.660 (tid 0x7f738140c890): [dd_rpc2_nfs x3000004b6] dd_rpc_gss_print_error:141 - event gss_error, Clock skew too great 08/23 18:52:09.660 (tid 0x7f738140c890): [dd_rpc2_nfs x3000004b6] dd_rpc_gss_accept:217 - event gss_accept_failed, maj=851968, min2529638949 08/23 18:52:09.660 (tid 0x7f73814233f0): nfs3 destroyed tcp 3000004b6

And these are for the BOOSTFS Linux host logs:

# less /opt/emc/boostfs/log/ddboostfs_0_0.log Aug 23 18:04:39 1044 2019465280 [E] bfs_conn_open: connect failed (0 connections): 5075 Aug 23 18:04:39 1044 2019465280 [E] bfs_conn_lookup: connection lookup failed for node 0 file /.boostfs_sysinfo. (0 connections) Aug 23 18:04:39 1044 2019465280 [I] bfs_node_lookup: failed to obtain a connection for file /.boostfs_sysinfo Aug 23 18:04:39 1044 2019465280 [E] bfs_initialize_mntopts: initialization failed

7.  Boostfs mount with Kerberos authentication fails with the following error "Not able to access lockbox or lockbox entry cannot be found" 

# /opt/emc/boostfs/bin/boostfs mount -s LSU3 -d dd2500.domain.com /mnt/mountpoint/ Not able to access lockbox or lockbox entry cannot be found

 
If you intend to use BOOSTFS configuration options in "/opt/emc/boostfs/etc/boostfs.conf", verify the "[global]" setting in the file, and make sure it is not commented out (in order for any of the global options to work the key word "[global]" must be uncommented). This keyword is commented by default.

Example contents of the BOOSTFS "/opt/emc/boostfs/etc/boostfs.conf" configuration file:

# Comments are not allowed after the option value pair. # ############################################################################# [global] # Data Domain Hostname or IP address # data-domain-system=dd2500.domain.com

Example output that you would see in the Linux client BOOSTFS logs:

# less /opt/emc/boostfs/log/ddboostfs_0_0.log Aug 24 08:23:35 27227 3565582272 [E] bfs_lockbox_get_user_info: Failed to find key entry dd2500-rtp2.techsupp.local:LSU3 in config file /opt/emc/boostfs/lockbox/boostfs.lockbox. The requested Lockbox entry could not be found. Aug 24 08:23:35 27227 3565582272 [E] bfs_conn_lookup: connection lookup failed for node 0 file /.boostfs_sysinfo. (0 connections) Aug 24 08:23:35 27227 3565582272 [I] bfs_node_lookup: failed to obtain a connection for file /.boostfs_sysinfo Aug 24 08:23:35 27227 3565582272 [E] bfs_initialize_mntopts: initialization failed

Also the error may be the result of using a different hostname, storage unit name or BOOST user name in the Linux client "mount" command, compared to used used for setting up the lockbox, or the ones which correspond to the target DD. Make sure all names match across the DD, the BOOSTFS lockbox and the "mount" command.


8.  Attempt set "boostfs" user credentials fails with the following error :

bfs_krb5_err_handler: Kerberos error: -1765328360 [ERROR_MESSAGE=Failed getting initial credentials.]

Cannot set Kerberos credentials

1. Verify Active Directory user credentials are correct by logging into the DC with Active Directory user credentials 
2. Verify time on Linux client and Kerberos server are no more than 5 minutes apart. Kerberos authentication requires the clock times to be no more than 5 minutes apart.



9. When mounting the BOOSTFS in the Linux client, it ends with error "fusermount: mount failed: Operation not permitted."

This can be due to a number of reasons. "fusermount" is the underlying Linux command which ultimately uses FUSE to mount the DD BOOST storage unit locally in the Linux client namespace. Doing so requires special privileges. That's why "fusermount" is a SUID binary in the Linux client:

# ls -l /usr/bin/fusermount -rwsr-xr-x. 1 root root 38680 May 11 2019 /usr/bin/fusermount

Being a SUID binary means whatever the user is running the attempt to mount the storage unit, the "fusermount" will run with "root" privileges. If "fusermount" is not SUID to root, only the Linux client root user may be able to mount the remote BOOST storage unit. This may not be a problem if setting up the mount under /etc/fstab, but would be when the mount is to be done by a non-root user.


10. BOOSTFS mounts OK but attempts to access the storage unit contents at the mount point fail with permission errors

When mounting the storage unit manually using either DataDomain "boostfs" command or /etc/fstab (or even "mount.fuse" from the CLI), it is the default's underlying FUSE implementation to only allow access to files to the Linux client userid which mounted the BOOSTFS. So if the mount was made as root, only root may access the files. If the mount was made as user "postgres" (for backing up a PostgreSQL DB), only this user would have the permission to access the files.

If for different reasons one needs other users to be able to access the files in the mounted storage unit from the Linux cloud, the /etc/fuse.conf may need be tuned. See more details here:
http://manpages.ubuntu.com/manpages/bionic/en/man8/mount.fuse.8.html

You would need to set the following option in /etc/fuse.conf:

       user_allow_other
              Allow  non-root  users  to specify the allow_other or allow_root mount options (see
              below).
And then use the following one when mounting the BOOST storage unit, from the command line:
 

       allow_other
              This option overrides the security measure restricting  file  access  to  the  user
              mounting the filesystem.  So all users (including root) can access the files.  This
              option is by default only allowed to root, but this restriction can be removed with
              the configuration option described above (user_allow_other).

11. BOOSTFS does not support one-way or two-way authentication using certificates.   Use two-way-password instead

 

Aug  1 15:46:39.436 4632 1188 [I] [ddp log] [1218:4A4] number of sslquery 1
Aug  1 15:46:39.436 4632 1188 [E] [ddp log] [1218:4A4] number of ssl_query_failed2 = 1

a) When mounting the storage unit, if you encounter errors with SSL check to ensure that MTU between the client and the DD are not being reduced by the network.

b) Ensure that the DD is not configured to require one-way or two-way authentication using certificates.  

# ddboost option show
Option                           Value
------------------------------   -------
distributed-segment-processing   enabled
virtual-synthetics               enabled
global-authentication-mode       two-way
global-encryption-strength       medium
------------------------------   -------

# ddboost clients show config
Client   Encryption Strength   Authentication Mode
------   -------------------   -------------------
*        medium**                two-way
------   -------------------   -------------------
(**) The global security settings take precedence over these client(s) specific settings.

c) Ask the customer if they can change the configuration to two-way-password or none
 

ddboost option set global-authentication-mode none global-encryption-strength {none | medium | high}

or

  ddboost option set global-authentication-mode two-way-password global-encryption-strength {medium | high}

d) If the clients are also required to be one-way or two-way, then this will also have to be changed to allow for BoostFS to mount to the DD.  Request permission on which of the following changes would be their choice.
 

If the current settings on the DD are none, none, then no changes needs to be made.  Proceed to test a BoostFS mount.

# ddboost client show config
Client   Encryption Strength   Authentication Mode
------   -------------------   -------------------
*        none**                none
------   -------------------   -------------------


A) Modify their existing to use two-way-password and existing encryptions strength.
ddboost clients modify *  authentication-mode two-way-password encryption-strength { medium | high}

B) Add a new client for this current BoostFS mounting client. 
ddboost clients modify <client hostname>  authentication-mode two-way-password encryption-strength medium

BOOSTFS Configuration Related Questions:

1. What is the maximum number of mount point sections one can define in the boostfs configuration file?
There is no limit to the number of mounts in the mount point section of the boostfs.conf file. 

# Mount point sections are separated by [mountpoint] tags # # [/path/to/mount] # [/mnt/bofs] # Data Domain Hostname or IP address # data-domain-system=dd2500-1.yourdomain.com # Storage Unit # storage-unit=su-name # Storage Unit Username # storage-unit-username=sysadmin # Subdirectory within the storage-unit to mount to # directory-name=path/to/subdir

2. Networker supports BOOSTFS by default, and can handle on-the-fly mounts for performing certain tasks and backups

Even if Networker is a BOOST-enabled backup application, it also supports BOOSTFS on the Linux client systems where it is installed. One example NW backup taken from the Linux client command line, once the BOOSTFS lockbox has been set up, would be the following (PostgreSQL backup):

# nsroappbackup -z /nsr/apps/config/backup_postgresql.cfg 174908:(pid 16487):Saving the backup data in the pool 'DB'. 175019:(pid 16487):Received the media management binding information on the host 'dd.example.com'. 174910:(pid 16487):Connected to the nsrmmd process on the host 'dd.example.com'. + /usr/pgsql-11/bin/pg_dump --file=/nsr/apps/tmp/e3106c82_294324_16487/dump.sql --format=plain Continued processing with the returned value 0. + /bin/cp /data/postgresql.conf /nsr/apps/tmp/e3106c82_294324_16487/ Continued processing with the returned value 0. The files in the save set 'PostgreSQL_postgres_backuppostgre_full' at time '01/04/20 15:40:36' are: Size: Name: 2645 dump.sql 24000 postgresql.conf 2 File(s) 26645 bytes The backup command '/nsr/apps/config/scripts/backup-postgre-dbon1-full.sh' completed successfully. The backup completed successfully.

Once the backup job is configured, it can also be kicked off from the NW GUI and progress monitored there.messages.engineering would show log entries like the one below when the mount issued by NW is done:

Apr 1 15:20:49 dd.example.com ddfs[17040]: NOTICE: ddboost-<client.example.com-49808>: Boostfs: Apr 1 15:22:52.395 16275 704915520 [I] DDBoost Plugin Version is: [7.0.0.0.633508] Apr 1 15:20:49 dd.example.com ddfs[17040]: NOTICE: ddboost-<client.example.com-49808>: Boostfs: Apr 1 15:22:52.395 16275 704915520 [I] BoostFS Version info: [BOOSTFS:7.0.0.0-633922 FUSE:2.9.7] Apr 1 15:20:49 dd.example.com ddfs[17040]: NOTICE: ddboost-<client.example.com-49808>: Boostfs: Apr 1 15:22:52.395 16275 704915520 [I] bfs_lib_init: Mounting dd.example.com:LSU_NAME on /mnt/mountpoint