Dell R740XD NVME PCIE Error

Hi @bryans1991,

Can you let me know what is the backplane DPN# that you're using? Also, based on the backplane table ref: https://dell.to/3Zpw2rm which one are you trying to achieve?

I've seen this error before and since you mentioned of cables that were involved, you can check if the cables are properly seated on both sides of the connection, card and backplane. Here are the couple of steps: 

• Shut down the server(s)
• Disconnect NVMe cabling from backplane and motherboard / controller cards
• Inspect cabling for any visual damage
  • Replace any cables with visual damage
• Inspect connectors on backplanes, motherboard, controller cards for any visual damage
  • Pay special attention for debris, black plastic shavings, bent pins
    • If debris is found, please try to clean off / remove if able.
  • Replace any parts with visual damage
• If no damage is found on any component, please reseat the NVMe. 
  • Please ensure the cabling is fully seated into the connectors

Thanks for your response Joey. I have verified all cables are not damaged, everything has been reseated as well. Hardware repurpose has also been done. Within the idrac gui I can see both NVME cards as well. Part number for the backplane is FJH5T

Trying to achieve 24x NVME 

Also I verified the SATA settings that AHCI is not enabled and set to SATA with NVME tab marked as well. 

Hello @bryans1991 

740xd is a complex system as I can understand from the field experience due to firmware issues and dependency, I will suggest only the below steps.

1. Power off system.

2. Check each card individually if individually both the cards are fine then check the firmware versions of both the cards.

3. Power drain the system check again.

4. Upgrade the System firmware from bundle.

Hi @bryans1991,

I don't think it's possible. FJH5T might not be able to support U.2 SSD. According to the table, the only backplane is 3.5" x12. 

Hello, yeah they can be finicky at times but overall love using them. I verified all updates on the units and still no luck. I can't find individual updates for the NVME Expander controller though. 

Really? That is the part is what is on the 24-bay backplane we have and all the ports have NVME pins. 

Also, the midplane on the rear flex bay is 001YX3

From the original message you sent me, the backplane looks identical to figure 2 - 24x 2.4" NVME 

Hello bryans1991,

I see you have completed the step Joey mentioned: reseat cables with special attention for debris, black plastic shavings, bent pins.

Could you give the system a power drain and check if you still get the error:

drain flea power (shut down, disconnect power cables and Network cables, hold in power button 20 seconds with cords removed).  After flea power drain, system has to set for 3 minutes for DRAC to reset without any power plugged in

During this time please confirm cables are seated correctly and not like the attached image.

 Then plug in NIC and power but wait 2 minutes before power on to give DRAC time to initialize.

Do you have the ability to contact Support directly in case deeper troubleshooting and posibility parts may need to be replaced?

Thanks for your info ! I ran all of these steps and still the same thing. 

It is out of warranty so no support but I have bought new risers and cables and still the same issue on multiple units. Still, it doesn't even flash an orange light ( hardware error ) nothing in event logs either and all 24x NVME U.2 drives can be read and raided with no issues. 

Wouldn't PCIE Downtrain on bus :95 D:4 F:0 would be pcie slot 4 which is the NVME Card ? Or is that wrong. 

Hello bryans1991,

Have you tried swapping slots 3 with 4? See if it follows or stays in the slot?

The bus :95 D:4 F:0 does not necessarily mean slot 4.

How to identify faulty part when facing BUS DEVICE FUNCTION error

https://dell.to/3SLMgJq

If you do not have an active warranty then you can ask about a Post Standard Support contract to work the issue with Support.

Had this same issue on the same model server. The problem happened after a BIOS and iDRAC update, then could not see but one of the 24 NVMe drives. After pulling power and restarting, then got a similar PCIe error during POST. Replaced the two NVMe extender cards (DPN W0DNK)  slot 3 & 4 from another spare server and then everything booted fine. 

@Meistermind​   Also... booting into the Life Cycle Controller, both the extender cards were missing when viewing the firmware versions.

@Meistermind​ I have a pretty similar experience. I have two R440s that are in the process of being updated about reconfigured. I'll verify BIOS / Firmware versions shortly, but on system A I had two slots (PCIe 1 & 2) that allowed for bifurcation in the BIOS, and after updating to the latest BIOS about 3 days ago, I now only have PCIe slot 2 that is showing me a bifurcation option. This seems to be a problem with the update, in my opinion. Will reply back with screenshots / specific version numbers.