Data Domain: Client security scan degrades system performance

Excessive readdir Activity

Security tools that repeatedly list or search directories trigger large numbers of readdir operations. This causes contention in the file manager and directory manager layers.

Indicators include:

• A sharp increase in daily readdir counts (for example, from ~1 million per day to tens or hundreds of millions).
• Elevated readdir thread activity (as in, nfsproc3_readdir_3_svc).
• Increased latency in file system operations.
  • Example: A sharp increase in readdir operations. A readdir operation is the result of a client request to read a directory. Two examples of this are a client running the 'ls' (list directory contents) command or a 'find' (search for files in a directory hierarchy). Many readdir threads cause contention in the file manager and directory manager layers.
    • The active readdir count can be found in a file system thread list (search for "nfsproc3_readdir_3_svc") or by reviewing the daily readdir count found in an autosupport. In the below readdir stats, total operations increased from roughly from 1 million readdir operations per day to 70 million per day.

      FM API histograms:
      op type                        mean      std-dev       <1ms      <10ms     <100ms        <1s       <10s      <100s      >100s      total        max        min
      
      autosupport.6:
      readdir                     3.415ms     18.469ms   12580518   31919253    1638591     135685       3749          1          0   46277797  10306.395      0.014
      autosupport.5:
      readdir                     3.459ms     18.829ms   12763092   32553822    1696951     141236       4027          1          0   47159129  10306.395      0.014
      autosupport.4:
      readdir                     3.434ms     18.627ms   13082191   33663810    1723265     143462       4051          1          0   48616780  10306.395      0.014
      autosupport.3:
      readdir                    15.599ms     33.340ms   13312985   36148759   14982367    1910057       4179          1          0   66358348  10306.395      0.014
      autosupport.2:
      readdir                    24.991ms     35.316ms   13609833   46780191   57380779    4995642       4588          1          0  122771034  10306.395      0.014
      autosupport.1:
      readdir                    27.739ms     37.025ms   13876217   69875224  107391735    9361188       5661          1          0  200510026  10306.395      0.014
      autosupport:
      readdir                    30.935ms     43.488ms   14379002   92945802  158095969   16689520      16549          1          0  282126843  10306.395      0.014