Disable reboot after PCIe fatal error

Hello Robert,

server, card, and communication with the card work without problems if the card is installed in the R730 and its firmware not altered. In that case, the FPGA fetches its configuration from the flash at power-up.

The problem occurs as soon as we program the FPGA directly (even with a firmware that is identical to the one stored in the flash, which in turn is known to be working). As mentioned before, programming the FPGA directly and doing a rescan afterwards works on servers from several other vendors without problems.

lspci on the R730 shows the following for NUMA node 1:

[...]
80:01.0 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 1 (rev 01)
80:02.0 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 2 (rev 01)
80:03.0 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 3 (rev 01)
80:03.2 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 3 (rev 01)
80:05.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Map/VTd_Misc/System Management (rev 01)
80:05.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D IIO Hot Plug (rev 01)
80:05.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D IIO RAS/Control Status/Global Errors (rev 01)
80:05.4 PIC: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D I/O APIC (rev 01)
82:00.0 Signal processing controller [1108]: CERN/ECP/EDU Device 01a0
ff:08.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D QPI Link 0 (rev 01)
[...]

From what I can see, the numbers from the system event log (bus 128 / device 2 / function 0) correspond to the port of the PCIe root complex the card is attached to (Status for that port shows the expected speed and width, and is shown as active).
For this root complex port we see the following

        Capabilities: [90] Express (v2) Root Port (Slot+), MSI 00
               DevCap: MaxPayload 256 bytes, PhantFunc 0
                       ExtTag- RBE+
              DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported+
                       RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
                       MaxPayload 256 bytes, MaxReadReq 128 bytes

Could the error reporting in case of a fatal error be related?
Is there a possibility to turn this off?

Best regards

zorc,

I just stumbled onto your post while researching the identical problem. I still have no acceptable solution, though I seem to be narrowing it down.

Were you able to solve your issue?

If not, here's what I have so far:

Like you, our R730 reboots when we reconfigure the FPGA. While using the FPGA as a PCIe endpoint is not my first choice, it is fairly common, and, more importantly, it is how the board is designed. I am running CentOS 7 on our server. 

While this is my first PCIe card, I have designed many compact PCI hot-swappable boards. I haven't had this much trouble with any of them. The first part of the equation is getting support set up on the OS/driver side. First, the kernel must have hot-plug enabled, and the hot-plug pci drivers (such as pciehp) must be installed/enabled. This is usually selected during kernel compilation. Assuming all of that is in place, the board can be reprogrammed (with some caveats) as long as the PCIe config doesn't change, i.e. require new/different resources. The method of removing (or reprogramming) a board is typically:

   -Save the board PCIe config space to a file (many OEM board manufacturers provide this ability in the driver or example designs)

   -Stop the driver/Unbind the board from the driver (This prevents the driver from suddenly having its hardware yanked out from under it)

   -Reprogram FPGA

   -Bind board to driver

   -Reprogram the config space of the board with the config file that you saved earlier.

   -**Sometimes you may need to rescan the PCIe bus.

Ok. So here's the problem, right? None of this works. As soon as the FPGA is unconfigured, the server kindly reboots, and places two nice Fatal Error entries in the system log for us to scratch our heads over.

For my part, I believe that I have all of the OS/driver settings correct. I was even able to unbind the card from the driver, causing it to disappear from lspci. Then, BAM, the server reboots as soon as I hit "Reconfigure." That leaves only the server hardware and BIOS as suspects. An lspci of the slot that the board is connected to (for me, lspci -vvvks 80:02.0) shows what I currently believe to be the cause. Under the section that reads "Capabilities: [90] Express (v2) Root Port (Slot+), MSI 00", there is the entry "SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-"

The HotPlug- and Surprise- entries tell us that the slot is not configured (or configurable?) to A) perform its role in the hotplug dance (HotPlug-), and the one that should interest you, B) ignore the disappearance of a device in this slot, and don't throw an error (Surprise-).

I am currently running BIOS version 2.1.5 on my server, which is not the latest. It does not provide any method of setting these parameters. The latest version seems to be 2.4.3. I am not sure if this version adds the ability to modify these parameters, but I have an open ticket with Dell to find out.

I also found an Intel datasheet for the Xeon E5 that talks about the Slot Capabilities (SltCap) register, which is the register that controls the capabilities of the slot. It mentions that these bits are programmed through the BIOS, which places it firmly back in Dell's lap. It's on page 126. Here's a link to the document:

www.serversdirect.com/.../xeon-e5-v3-datasheet-vol-2.pdf

I know this isn't a solution...yet...but I hope it helps. I'll let you know what comes of the Dell interaction.

Even if you disable the AER bits (mandatory) the system can still reboot since the iDrac is always snooping the bus. On the R740, this is MUCH worse. iDrac thinks you always want to reboot, a debug iDrac has a setting to block this reboot but that isn't visible to mortals. Even after updating the driver (Intel is no help here) to make the reconfig work, the device can still fail later in production use. iDrac thinks any bus error is fatal, and won't let the kernel dump or even type out message about the type of error (root credits lost/DMA to an invalid address/bus timeouts/etc).

 

Using a $170K gen-3 psi bus snooper won't help for the post-reconfig problem at there is no trigger when iDrac reboot for you.