Data Domain: BoostFS Issues with Security Scanner Software or Apps
Summary: The boost file system on the Linux and Windows client behaves unexpectedly when Security Scanner or Security Applications scans or accesses files inside a Boost file system mount point. ...
Symptoms
Boost file system-mounted drive, or mount-point becomes unresponsive.
The backup operations from the application may fail with unsuccessful reads and writes on the files on the mount point. Example:
Apr 19 10:15:09.853 29214 2269886208 [E] [ddp log] [721E:7FC860125190] ddcl_ddcp_pwrite: Error in ddcl_ddcp_pwrite. [ERR=5002] Apr 19 10:15:13.990 29214 2269886208 [E] [ddp log] [721E:7FC860125190] ddcl_ddcp_pwrite: Error in ddcl_ddcp_pwrite. [ERR=5002]
Feb 24 19:52:46.803 423080 2833229568 [I] bfs_node_lookup: failed to obtain a connection; file=/image/azprd/20210224_test/data_D-AZPRD_I-1811368087_TS-CITBSP_FNO-1061_auvo0tkc err=2 Feb 24 19:52:46.803 423080 2833229568 [E] bfs_node_retry_conn: lookup failed path=/image/azprd/20210224_test/data_D-AZPRD_I-1811368087_TS-CITBSP_FNO-1061_auvo0tkc err=2 Feb 24 19:52:46.803 423080 2833229568 [E] bfs_open: lookup failed file=/image/azprd/20210224_test/data_D-AZPRD_I-1811368087_TS-CITBSP_FNO-1061_auvo0tkc err=2 Feb 24 19:52:46.803 423080 2833229568 [P] BFSPROC_EXIT_bfs_open: rc=-11
Performance Degradation, the read or write (backup or restore) operations become slow.
Cause
The Security Scanner Software or Applications installed on the machine scans the Boostfs mount point as well along with other drives and folders
BoostFS does not recommend multiple applications including the Security Scanners to access BoostFS mount point simultaneously.
This may result in failure depending on the nature of the access since we have a single file context maintained in the BoostFS.
Also, anti-virus/security scanner software may prevent some file-system specific operations.
Triaging:
Check for the "app-info" in boost file system log messages. This would be populated with the antivirus or Security Scanner software names. Here "s1-agent" and "s1-fanotify" are the security scanner application names.
ddboostfs_0_0.log:265760:Apr 21 16:51:09.048 26822 1603430144 [D] bfs_get_app_info: app_info=s1-agent ddboostfs_0_0.log:266094:Apr 21 16:51:10.102 26822 1603430144 [D] set_conn_app_info: app_info=s1-agent Feb 24 19:52:46.803 423080 2394937088 [D] bfs_get_app_info: app_info=s1-fanotify
This means that the scanner software is accessing the boost file system mount point.
Check the Application in DDBOOST Connections Detailed in data domain autosupports.
DDBOOST Connections Detailed with-fips-mode ----------------------------Active Clients: 5Clients: Client Idle CPUs Memory(MiB) Plugin Version OS Version Application Version Encrypted DSP Transport FIPS mode -------------------------------- ---- ---- ----------- --------------- -------------------------------------------------------- ------------------------------------------------ --------- --- --------- ----------- c3po01 YES 14 64,152 7.7.4.0-1017373 Linux 4.1.12-124.67.3.el7uek.x86_64 x86_64 BOOSTFS:7.7.4.0-1017610 FUSE:2.9.4 s1-agent NO YES IPv4 NO
Finally, check the settings of Scanner applications if it includes the Boost file system mount points.
Resolution
It is recommended to perform antivirus scans outside the Boost file system operations window.
Check if Security Scanner Application settings can be modified to exclude the Boost file system mount points from Scanning and any other operations performed by it.
Additional Information
This article applies to all Boost file system versions irrespective of any Security Scanner Software installed on it.
The reported issues pertain to multiple security scanners, including Carbon-Black, McAfee, Fanotify, s1-agent, CrowdStrike, and more.