Dell Unity: Extreme decrease in SMB/NFS performance after upgrading to UnityOS 5.1.x (User Correctable)
Summary: After upgrading to UnityOS 5.1.x, there is a substantial decline in SMB/NFS performance and overall TCP throughput.
Symptoms
After an upgrade to UnityOS 5.1.x code, the Transmission Control Protocol (TCP) throughput for all file-based protocols can suddenly decline for both read and write operations. These file-based protocols include Network File System (NFS) and Server Message Block (SMB)
Unity performance metrics for file operations do not indicate any latency at the protocol level. However, the total number of incoming TCP packets may show an extreme decline.
In some instances, the effect of this can translate to a decline in throughput of more than half that attained previously.
The performance impact would be separated to a particular storage processor, although both storage processors can be separately affected.
Rebooting an effected storage processor often returns throughput to normal levels; however, the poor performance most likely returns moving forward.
Ktrace logging may indicate when throttling has turned on:KERNEL: 6:[core] ucb congestion level inc to 1, uncachedBuffInUse is x
KERNEL: 6:[core] ucb congestion level inc to 2, uncachedBuffInUse is x
If the memory usage continues to grow the level increases which will further impact performance:KERNEL: 6:[core] ucb congestion level dec to 2, uncachedBuffInUse is x
Once the memory usage reduces enough, the throttling turns off:KERNEL: 6:[core] ucb congestion level dec to 0, uncachedBuffInUse is x
This does not affect block-related performance.
Cause
Resolution
Upgrade to UnityOS 5.2.0.0.5.173 or later. Starting in this release, the thresholds were adjusted to greatly reduce the impact of this change. In UnityOS 5.2.x, the threshold for engaging the throttle was lowered. It requires the memory usage to be in a more critical state before the throttling starts.