PowerFlex: Rebuild progress on several SDSs is extremely slow

In an environment where the SDS hosts are connected to a network switch, a backward rebuild may be extremely slow for no obvious reason.

An affected SDS host has a lot of RCV_KA_DISCONNECT and SOCKET_DOWN messages in the logs, although the ports/interfaces will be UP and the link will be active/alive.

Symptoms

• ScaleIO UI indicates a constant backward rebuild ongoing at a slow rate
• Host's SDS was showing 0.0 KB/s transfer rate for rebuilds
• ifconfig on the affected SDS host shows a high number of dropped packets
• Checking the switch ports for the host the SDS is running on indicates CRC errors on one of the ports; port flapping occurs

Impact

Rebuild is extremely slow and will probably never finish as virtually no data is passing through the affected port/link.

PowerFlex does not poll ports for liveliness. It keeps trying the port over and over so some data will go through. Attempts are done at different rates as the ports on the switches are set to auto and the link speed will constantly get renegotiated.

If there is a secondary link, administratively shut down the affected port causing issues on the host. This will force data to travel through the secondary link and speed up the rebuild process.

Fix

1. Place each SDS into maintenance mode one at a time and then change the ports on the switch being used by this host to the appropriate speed link (1 Gbps / 10 Gbps).
2. Bring each SDS out of maintenance mode before moving to the next SDS.
   Hard setting each port to a fixed rate on the switches prevents auto-negotiation from running, which is thought to be part of the cause for the CRC errors. Also, this causes the port to hard fail in the event of an issue, instead of flapping.