PowerScale: Sysctl Change Can Affect TCP Performance On Degraded Networks OneFS v8.1.0 and newer

A security vulnerability fix in the networking stack causes Transmission Control Protocol (TCP) performance to unexpectedly degrade.

A security fix for FreeBSD was introduced in CVE-2018-6922.
This CVE addresses mitigating a potential DDOS attack in OneFS versions:
8.1.0.4+
8.1.2.0+

Taken from https://www.cvedetails.com/cve/CVE-2018-6922/

One of the data structures that holds TCP segments in all versions of FreeBSD prior to 11.2-RELEASE-p1, 11.1-RELEASE-p12, and 10.4-RELEASE-p10 uses an inefficient algorithm to reassemble the data. 
This causes the CPU time spent on segment processing to grow linearly with the number of segments in the reassembly queue. 
An attacker who has the ability to send TCP traffic to a victim system can degrade the victim system's network performance and/or consume excessive CPU by exploiting the inefficiency of TCP reassembly handling, with relatively small bandwidth cost.
Publish Date : 2018-08-09 Last Update Date : 2019-01-16

If Isilon clients are experiencing performance issues, the recommended solution is to work with the network team to identify anything that could be degrading the network.
Packet captures a { text-decoration: none; color: #464feb; } tr th, tr td { border: 1px solid #e6e6e6; } tr th { background-color: #f5f5f5; } Usually show many retransmits, dropped packets, and out-of-order packets.

a { text-decoration: none; color: #464feb; } tr th, tr td { border: 1px solid #e6e6e6; } tr th { background-color: #f5f5f5; } If the network team cannot resolve the issue and the network is secure, adjust the sysctl setting to increase the TCP reassembly queue length.
(The security fix was put in place to prevent a malicious client from DDOS against the cluster. Changing the sysclt could allow an attack to occur.)

The default value following the March 2019 roll-up patch for 8.1.2.0 is 100.
Confirm the current value by running the following:

isilon1# sysctl net.inet.tcp.reass.maxqueuelen
net.inet.tcp.reass.maxqueuelen: 100

Temporarily change the value to a setting lower than this value:

sysctl net.inet.tcp.reass.maxqueuelen=2048

Have the clients check the network to confirm if the performance issue has been resolved.
If the issue is resolved, change the value for the whole cluster on reboot by editing the following file:

vi /etc/mcp/defaults/sysctl.conf
If the sysctl exists, edit that line. If it is not in the file, add these lines to the bottom of the file:

# Increase the maxqueuelen to increase performance in degraded networks. KB 537464
net.inet.tcp.reass.maxqueuelen=2048

Push the change to all nodes in the cluster:

 isi_sysctl_cluster net.inet.tcp.reass.maxqueuelen=2048

Calculation based on net.inet.tcp.recvbuf_max and mss of network interface