Isilon: Gen6 DE Peer Node Pair Generates Events and Console Messages Indicating Problems with Their NTB Link

A Gen6 DE (H400, A200, A2000) peer node pair can sometimes start generating error messages indicating problems with their NTB link. Errors can include repeated 'link up/down' messages and link speed negotiation errors, for example:

mnv0: HW link down event
mnv0: HW link up event
mnv0: Failed to negotiate PCIe lane speed; expected 3 lane speed, got 2.
mnv0: transport link up
mnv0: peer up

Alternatively, the logs and/or console may show the NTB link as down and not coming up. There appear to be multiple possible causes for this issue, some of which are still under investigation. This issue may apply if one or both affected nodes were recently moved to a different chassis slot. It may also apply if their compute module was recently replaced. This issue does not affect EP nodes (F800, H600, H500).

Peer nodes communicate with each other through a special communication channel called the NonTransparent Bridge (NTB) embedded in the chassis backplane. In normal operation, both nodes in a peer pair must have a different PPD value to be able to communicate with each other through the NTB. PPD values are assigned based on the node's slot ID in the chassis. This issue occurs when the BIOS retains the old slot ID instead of detecting the new one after a node or compute module is moved. This may cause the PPD value to be set incorrectly, which causes a conflict that prevents the nodes from establishing an NTB link.

Run the following command on both nodes in the affected peer pair to verify whether the issue documented in this KB is applicable:

# sysctl dev.ntb_hw.0.debug_info.ppd

Each node responds with either:

dev.ntb_hw.0.debug_info.ppd: 73

Or:

dev.ntb_hw.0.debug_info.ppd: 93

• If one node in a pair responds with 73 and the other responds with 93, these nodes are not affected by the issue documented in this KB.
  Contact Isilon Technical Support for further assistance.

• If both nodes respond with the same number, whether 73 or 93, they are affected by the issue documented in this KB. This issue is resolved in updated node firmware in Node Firmware Package 10.1.6 and later, as well as in a code fix in OneFS 8.1.0.4 and later. While each update individually remediates the issue, both releases contain other important fixes, so it is recommended to install both.