PowerFlex: SDC Returns IO Errors with No Rebuild Triggered

 In certain network topologies, one or more, but not all, network interfaces on an SDC become unavailable, causing the SDC to fail communicating with one or more SDSs, resulting in IO errors.

• Errors returned for some or all executions of ping(1), from the SDC to each SDS
• IO errors are returned by the SDC to the client application.
• The SDC loses connection with some or all SDSs.
• No SDS/SDS or MDM/SDS disconnections are reported.
• No rebuild is triggered.

Impact

Applications running on the SDC cannot access data stored in the PowerFlex system.

The MDM does not switch data copy owner roles (Primary/Secondary) based on SDC access, causing the repetition of the following cycle:

- Requests sent by the SDC to the Primary data copy owner fail, as it is inaccessible through the interfaces available on the SDC.

- The SDC contacts the MDM for updates on the Primary data copy and its owner.

- The MDM has no reason to update the data roles, as the MDM itself and all other SDSs can communicate normally with the current Primary data copy owner.

- The MDM replies to the SDC with the Primary data copy owner, which is the same SDS as the one the SDC sent the request to previously.

This behavior follows the intended system design.

In case the cause is missing OS-level network routing configuration, change the routing configuration or network topology to have a path available from each SDC interface to at least one network interface on each SDS.

The OS-provided solution for adding persistent routes for various platforms follows. Refer to the official documentation for the exact usage and syntax.

A system may have a non-standard solution for managing the hosts’ routing table. In which case, the available solution should be used over the method described below.

Network addresses and gateways should be provided by the system’s network administrator.

For ESXi 5.5/6:

Add the appropriate esxcli network ip route ipv4 add statements to /etc/rc.local.

For RHEL6/7:

Make one of the following changes, as appropriate:

- Add the appropriate ADDRESS, NETMASK and GATEWAY configuration parameters and their values to the appropriate interface’s /etc/sysconfig/network-scripts/route-interface.

Or

- Add the appropriate any net statements to /etc/sysconfig/static-routes.

Requires that the net-tools RPM be installed

For Windows:      

Perform the appropriate route -p add command from Command Prompt.