Hi,

Possible sources could be:

1. HBA firmware and mode  

   - Confirm the SAS9207‑8i is in IT mode (not IR) and on a current, stable P20 firmware (ideally 20.00.07.00). Older or OEM‑tweaked firmware revisions are known to cause weird ZFS errors under load.

   - If it’s an eBay card, strongly suspect a bad or counterfeit flash; reflash with official Broadcom/LSI firmware and BIOS (or no BIOS) from a trusted bundle.

2. SAS cabling and backplane path  

   - Check exactly how the R720xd front 12‑bay backplane is wired: which SFF‑8087 ports on the backplane go to the 9207‑8i vs any PERC/H710 mini still installed.

   - Reseat both ends of the SAS cables, then, if possible, swap the 8087 cables or backplane ports (e.g. move from backplane port A to B) and see if the errors follow the cable/port or stay on the same bays.

   - If errors move when you swap ports/cables, you’ve found a bad cable or backplane port.

3. RAM issues  

   - ZFS checksum errors across multiple drives, especially when reading rather than writing, can be caused by flaky RAM corrupting data in memory even when disks are fine.

   - Run a long overnight MemTest86+ from USB (multiple passes). Any error means replace that DIMM set before trusting the pool again.

4. Mixed controllers / leftover PERC  

   - If the server still has a PERC H710 mini connected to the same backplane as the SAS9207‑8i, make sure there is no dual‑ownership or mis‑split of the backplane; each bay group must belong to exactly one controller.

   - On R720xd, verify that the backplane is either fully on the 9207‑8i, or properly split with one side on HBA for ZFS and the other on PERC for something else; misconfig can yield phantom errors and weird LED states.

5. Rear OS/app SSDs and cabling  

   - The two 2.5" rear bays on R720xd use a separate backplane and cable set; bad cables or odd interactions have been seen when adding SATA SSDs there.

   - Try temporarily disconnecting the rear flex‑bay or moving the OS/app SSDs to different ports (or even a simple internal SATA connector) and see if the ZFS pool stops throwing erro

rs during Plex reads.

OK, starting at the top. I checked the firmware and it was 20.00.00 and the BIOS was 7.29.00.00.  So I went to Broadcom website and downloaded the latest and flashed it, now it reads in sas2flsh.exe as firmware 20.00.07.00 and BIOS 7.29.02.00.  I used the IT mode FW file.  Also sas3flsh.exe lists the card as an SAS2308_2(D1) if that matters.  THe problem is that I can't test it out because now I can't boot from the rear drives.  The TrueNAS installer sees the rear drives, and I was able to install to it, but for the life of me I can't figure out what to set to get it to boot.  And when I hit CTRL-C to enter the LSI configuration it says it will open after initialization but it never opens, it just nays no boot devices.  Also, I don't know if it matters, but in the boot sequence when the HBA card comes up it shows as AVAGO Technologies SAS.....   And there is no PERC mini card installed.  As far as I can tell the only HBA controller is the 9207-8i. 

Also, during the boot sequence when it gets to the HBA card the Samsung SSD that is in the back bay is shown, and then under it it says "1 supportable devices are presented for system boot selection!"  But again, when I press CTRL-C it never goes into the card config.

Please help me figure out how to get it to boot again from the back drives.   

Hello,

Thanks for the detailed information and for the work you’ve already done. Based on your testing and the behavior you’re seeing, here are a couple of steps that may help narrow the issue down while keeping in mind some important Dell support boundaries.

Important Notes

• TrueNAS is not a Dell‑supported operating system.
  Dell does not validate ZFS, TrueNAS SCALE, or any BSD/Linux‑based storage distributions on PowerEdge hardware.
• IT‑mode HBAs (e.g., LSI/Avago/Broadcom SAS9207‑8i in IT mode) are not supported by Dell on the R720xd platform.
  These cards can work, but firmware/compatibility issues must be handled by the community or Broadcom, not Dell.

Because of those limitations, you may need to wait for guidance from the TrueNAS/ZFS community for deeper analysis. That said, here are a few practical steps that may help stabilize or diagnose the system:

Recommended Steps

Press F11 during POST and see if your boot drive is listed.

Confirm the SAS9207‑8i Firmware Flash Is Clean

Even after flashing P20.00.07.00 IT mode, make sure:

• The card shows as LSI/Avago SAS2308 IT Mode (no vendor‑modified OEM ID)
• The OptionROM/BIOS loads properly (or disable OptionROM if not booting from HBA)
• Only the front backplane is connected to the HBA—not the rear bays

Improperly‑flashed SAS2308 cards are common and can cause disappearing drives or unreadable pools under load.

Memory Validation

ZFS checksum errors during reads often point to memory instability rather than disks.

• Run MemTest86+ for several passes (ideally overnight)
• Any RAM error means the pool cannot be trusted until DIMMs are replaced

Now because TrueNAS and IT‑mode HBAs are not supported by Dell, we cannot provide deep debugging on ZFS, custom HBAs, or non‑Dell firmware. However, the steps above should help you isolate the issue.

If the problem persists we may need to wait for any community members with TrueNAS experience that can share their recommendations.

@DELL-Charles R​ When I press F11 the boot drive isn't listed, I am UEFI boot mode if that makes a difference.  The card does show as LSI SAS2308-IT during the boot process.  I downloaded the firmware file from the Broadcom website.  My 9207-8I has 2 cables, both are connected to the front back plane.  Then there is another cable that runs from the front backplane to the rear drives.  I was able to boot from the rear drives before I flashed the firmware to 20.00.07.00.  And when I flashed it I used erase option 6 in the sas3flsh.exe before flashing the firmware and BIOS.  Now I am unable to test anything because I am unable to boot from the rear drives.  I would put in an internal drive but I don't currently have an adapter for the SATA power for an internal drive.   

Because IT‑mode HBAs such as the SAS9207‑8i are not supported by Dell on the R720xd detailed debugging of HBA firmware behavior or ZFS boot paths falls outside Dell’s validated configurations.

That said, based on what you’ve described, here are a couple recommendations that may help you move forward:

1. Reseat and simplify the backplane cabling path

The R720xd front 12‑bay backplane and the rear 2.5‑inch flex bay are two separate backplanes with different controller paths.

Since you flashed and erased the HBA (using erase 6), the PCIe device order and enumerations can change. If the rear bays are indirectly cabled through the front backplane, the boot controller assignment may no longer be valid.

Try this:

• Temporarily disconnect the front‑to‑rear interconnect cable so only the front backplane is attached to the 9207‑8i.
• Leave the rear backplane connected only to the onboard SATA/SAS controller.
• Reboot and check F11 → Boot Manager again.

This helps determine whether the interconnect cabling is causing the system to try routing the rear bays through the HBA (which does not support booting in your setup after the flash).

2. Re‑enable the rear‑bay controller in BIOS after the HBA flash

Using erase 6 resets a number of OptionROM/PCIe device parameters, and it can change how the R720xd enumerates storage controllers at POST.

Since you could boot from the rear bays before the flash but not after, it’s likely that the system’s boot path reverted to defaults.

Check the following in BIOS (F2):

• Integrated Devices → SATA/SAS configuration
  Make sure the rear drive controller is still enabled.
• Boot Settings → Boot Mode (UEFI/Legacy)
  Ensure this mode matches how TrueNAS was originally installed—
  a UEFI install will not appear in Legacy mode, and vice‑versa.
• Boot Sequence → Hard Drive Section
  Confirm the OS drive re‑appears once the rear controller is enabled.

This can restore the rear backplane as a valid boot source in UEFI boot manager.

@DELL-Charles R​ Thank you, I will try that this evening when I get home and will report back.