1 Rookie
•
2 Posts
0
51
October 3rd, 2025 23:25
PowerEdge R760xa - unable to get PXB communication between two GPUs (stuck at NODE)
I'm trying to understand why I can't get true PCIe P2P communication between the two NVIDIA L40S GPUs installed in PCIe Slots 36 and 38 that are connected to the same socket (CPU 1) in my R760xa:
$ sudo nvidia-smi topo -m
GPU0 GPU1 CPU Affinity NUMA Affinity GPU NUMA ID
GPU0 X NODE 0,2,4,6,8,10 0 N/A
GPU1 NODE X 0,2,4,6,8,10 0 N/A
NODE means both GPUs can communicate via the CPU they are both attached to (CPU 1). However, the goal is to get PXB (instead of NODE) which means the GPUs can communicate directly with each other over the PCIe bus, bypassing the CPU entirely. I need the lowest latency that direct P2P PXB provides, but I'm stuck at NODE.
From what I understand, maybe incorrectly, the following two settings could play a role (in addition to other things): Memory Mapped I/O above 4 GB (alt name Above 4G Decoding) and Resizable BAR Support.
For some reason I'm not able to locate either of these two in the BIOS.. are they actually supported in 16th Gen? Other thoughts what might be blocking the PXB comms?
DELL-Joey C
Moderator
•
4K Posts
1
October 6th, 2025 03:48
Hi,
After doing some research, here's what I gathered. For PXB, the path goes through multiple PCIe switches. On the R760xa, slots 36 and 38 are on the right GPU riser (RF1A) and are controlled by Processor 1. They are x16 + x16, double‑width slots intended for big GPUs like L40S/H100, and (crucially) they’re fed as independent x16 links rather than being fanned out from a shared downstream PCIe switch. In other words, you’re ending up with two separate CPU1 root ports, which explains the
NODEpath between the two L40S cards. So there isn’t a supported R760xa riser that places two double‑width GPUs behind the same PCIe switch, which would be required to get PIX/PXB between them.terrum
1 Rookie
•
2 Posts
0
October 6th, 2025 14:21
Thank you, Joey. Your input is very insightful and confirms my own suspicion that I'm not able to get PIX/PXB between my GPUs because of this server model (current hardware) limitations.
Couple of follow up questions: any chance an improved version of R760xa riser being offered in the future that would place two double‑width GPUs behind the same PCIe switch?
If that's not an option even in theory for R760xa (e.g. due to this model core design specifics), what would be the next closest (costs wise) server model choice that comes with a riser that offers PIX/PXB support at the hardware level?
DELL-Charles R
Moderator
•
4.6K Posts
1
October 6th, 2025 15:04
Hello,
There is no public roadmap or documentation indicating that Dell plans to release a new riser for the R760xa with a shared PCIe switch to enable PIX/PXB.
As far a cost wise questions I could recommend contact a Technical Sales Representative for options.