Switch Configurations – RoCE and iWarp Reference Guide
About copying commands from this guide
About copying commands from this guide
CAUTION: Many PDF editors and viewers add a line break to the end of each line of text in a PDF. As a result, when you copy commands that wrap across multiple lines in a PDF, the copied command is in the wrong format. It contains erroneous line breaks that might cause the script to fail.
To address this known limitation, do one of the following:
- Paste the copied commands into a text editor and remove the line breaks.
- Use the HTML version of this document when you are copying commands.
Revision history
Revision history
The revision history of the document is listed in the following table.
| Date | Document revision | Description of changes |
| April 2026 | 6 | Added the chapter "Rack aware cluster" |
| April 2026 | 5 |
|
| May 2025 | 4 | Updates due to product rebranding |
| March 2025 | 3 | Updates due to feedback |
| January 2025 | 2 | Updates related to release |
| March 2023 | 1 | Updates not documented |
| March 2022 | 0 | Initial publication |
Reference Guide
Reference Guide
Switch configuration recommendations for Mellanox, Intel E810, and Broadcom
This guide consists of switch configuration recommendations for all Mellanox, Intel E810, and Broadcom based deployments.
Mellanox, Intel E810, and Broadcom based deployments
Dell Technologies recommends that you use DCB (PFC/ETS) for all Mellanox and Intel E810 based deployments on switch ports that are used for RDMA traffic.
The following table shows topology recommendations for both converged and non-converged deployment configurations.
| Topology | Converged | Non-converged |
| Flow Control | DCB (PFC/ETS) | DCB (PFC/ETS) 1 2 |
| UDP RDMA (RoCE) |
|
|
| iWARP | Intel E810 (iWARP) | Intel E810 (iWARP) |
Note:
- Dell Technologies has updated its guidance for all Mellanox and Intel E810-based deployments. If you have existing deployments that followed our earlier guidance, when using Network ATC with Azure Stack HCI OS 23H2 and later, we recommend that you configure PFC/ETS for all RDMA ports, including the ones using Qlogic 41262 cards with iWarp traffic.
- For brownfield deployments where DCB is already configured on the switch, changing the cluster-traffic priority to 7 is optional.
- Historically, Dell Technologies used Priority 5 for cluster traffic. Microsoft supports both Priority 5 and Priority 7. This guide references Priority 5, but Priority 7 is also supported. You can implement either priority.
ToR switches
Top-of-rack (ToR) switches that are used in Azure Local instance deployments and that forward RDMA storage-network traffic must be listed in the Azure Local ToR switch list on the Microsoft page: Physical network requirements for Azure Local > Network switches for Azure Local.
Note: For deployments with ToR switches that are not listed, you must use the switchless storage-network deployment option.
1 This recommendation is based on Microsoft and networking guidelines to include DCB (PFC) for all Mellanox and Intel E810 based deployments.
2 Intel E810 adapters support both RoCE and iWarp protocols. There is no change in the switchport configurations for either card or protocols.
Dell Networking S4112F-ON switch
Sample configurations for Dell Networking S4112F-ON switch in Mellanox and Intel E810-based deployments
| S4112F-ON (TOR1) | S4112F-ON (TOR2) |
|
|
|
|
|
|
| S4112F-ON (TOR1) | S4112F-ON (TOR2) |
|
|
| S4112F-ON (TOR1) | S4112F-ON (TOR2) |
|
|
Dell Networking S4148F-ON switch
Sample configurations for Dell Networking S4148F-ON switch in Mellanox and Intel E810-based deployments
| S4148F-ON (TOR1) | S4148F-ON (TOR2) |
|
|
|
|
|
|
| S4148F-ON (TOR1) | S4148F-ON (TOR2) |
|
|
| S4148F-ON (TOR1) | S4148F-ON (TOR2) |
|
|
Dell Networking S5212F-ON switch
Sample configurations for Dell Networking S5212F-ON switch in Mellanox and Intel E810-based deployments
| S5212F-ON (TOR1) | S5212F-ON (TOR2) |
|
|
|
|
|
|
|
|
| S5212F-ON (TOR1) | S5212F-ON (TOR2) |
|
|
| S5212F-ON (TOR1) | S5212F-ON (TOR2) |
|
|
Dell Networking S5148F-ON switch
Sample configurations for Dell Networking S5148F-ON switch in Mellanox and Intel E810-based deployments
| S5148F-ON (TOR1) | S5148F-ON (TOR2) |
|
|
|
|
|
|
|
|
| S5148F-ON (TOR1) | S5148F-ON (TOR2) |
|
|
| S5148F-ON (TOR1) | S5148F-ON (TOR2) |
|
|
Dell Networking S5232F-ON switch
Sample configurations for Dell Networking S5232F-ON switch in Mellanox and Intel E810-based deployments
| S5232F-ON (TOR1) | S5232F-ON (TOR2) |
|
|
|
|
|
|
|
|
| S5232F-ON (TOR1) | S5232F-ON (TOR2) |
|
|
| S5232F-ON (TOR1) | S5232F-ON (TOR2) |
|
|
Dell Networking S5248F-ON switch
Sample configurations for Dell Networking S5248F-ON switch in Mellanox and Intel E810-based deployments
| S5248F-ON (TOR1) | S5248F-ON (TOR2) |
|
|
|
|
|
|
|
|
| S5248F-ON (TOR1) | S5248F-ON (TOR2) |
|
|
| S5248F-ON (TOR1) | S5248F-ON (TOR2) |
|
|
Switch security and other settings
The following switch settings are generic settings that a customer or Dell Services personnel may require during a switch configuration or deployment. Many of the commands are related to switch environment security. Use these commands and settings only if required for a given environment.
Backup switch license
!!!
!!! REQUIRED - Backup license
!!!
! license is located in base OS /mnt/license
! copy license to management VM and provide to customer
! usb key mounts to base OS /mnt/usb
! alternatively one can use SCP or other protocols if the network is available
! below command assumes USB key is inserted
system "sudo -i"
cp /mnt/license/<SVCTAG>.lic /mnt/usb
exit
Disable support assist
!!!
!!! REQUIRED - EULA reject
!!!
eula-consent support-assist reject
! confirm EULA reject
FIPS compliance
crypto fips enable
! confirm fips enablement
Set Daylight Saving time
! by default this has been set to Pacific Time Zone US Daylight Saving time.
clock summer-time PDT 2 Sun Mar 02:00 1 Sun Nov 02:00 60
Customer banner MTOD
banner motd #
Insert your own customer banner here.
#
Password update
! enable password <NEW.PASSWORD>
! enable password <NEW.PASSWORD>
! username azsadmin-lmuvl password <NEW.PASSWORD> role network-admin
Password attributes and max-auth-tries
!
password-attributes character-restriction upper 1 lower 1 numeric 1 special-char 1 min-length 15 lockout-period 15 max-retry 3
!
ip ssh server max-auth-tries 3
Login statistics, session timeout, and concurrent session limits
!
login-statistics enable
!
exec-timeout 600
!
login concurrent-session limit 3
SSH server settings
ip ssh server cipher aes256-ctr aes192-ctr aes128-ctr
ip ssh server mac hmac-sha1 hmac-sha2-256
ip ssh server enable
Configure RADIUS servers
! radius-server host <WDS Server IP> key 0 <secret> authentication accounting
! role name Prefix-BMCAdmin
! description Radius authenticated accounts
! rule 1 permit read-write
! aaa group server radius Prefix-BMCAdmin
! server <IP>
! source-interface mgmt0
! aaa authentication login default group <Group>
! aaa accounting default group <Group>
! aaa authentication login mschapv2 enable
TACACS
! feature tacacs+
! tacacs-server key <secret>
! ip tacacs source-interface mgmt0
! tacacs-server host <IP>
! aaa group server tacacs+ tacacs
! server <IP>
Configure syslog
! logging server 10.128.0.116 7 facility syslog use-vrf management
! no logging console
Configure syslog source interface
logging source-interface mgmt 1/1/1Configure logging audit enable
logging audit enableConfigure logging to not display to non-authorized users
logging console disableConfigure unused ports to non-default VLAN
!
interface vlan 2
description "Unused port vlan"
shutdown
!
interface range ethernet 1/1/4-1/1/28,1/1/33-1/1/34
shutdown
switchport access vlan 2
!
Remove access to system command
system-cli disableRemove default user admin
no username adminReset password of Linux user linuxadmin
system-user linuxadmin password <password>Rack Aware Cluster
Rack Aware Cluster
Rack aware cluster overview
An Azure Local rack aware cluster places cluster nodes across physically separated rooms to meet availability, safety, and regulatory requirements while maintaining cluster functionality.
An Azure Local rack aware cluster places the nodes of a single hyperconverged infrastructure (HCI) cluster across two physically separated rooms (availability zones) within the same building. This architecture supports environments where hardware must be isolated in to meet regulatory requirements, safety protocols, or operational constraints. The isolation creates separate fault domains while maintaining cluster functionality for critical industrial workloads.
Reference architecture Option A: Dedicated storage links
Option A is the recommended rack aware cluster networking design that uses dedicated storage links to isolate RDMA traffic between rooms.
Option A is the Dell reference design for rack aware clusters. In this topology, each storage VLAN has its own direct path between a specific pair of top-of-rack (ToR) switches. This design minimizes latency, isolates RDMA traffic, and reduces hop count. Each storage VLAN is carried over a separate room-to-room link:
- VLAN 711 (SMB1) connects ToR1 and ToR3
- VLAN 712 (SMB2) connects ToR2 and ToR4
Dell Engineering validates Option A as the recommended networking topology for Dell-based Azure Local deployments. Dell also supports Option B, Option C, and Option D topologies. However, Option A provides the clearest RDMA traffic isolation and the simplest operational model for two-room deployments.
For all Microsoft recommendations on the Option A network topology and other reference architectures, see:
- Azure Local rack aware cluster reference architecture
- Azure Local rack aware cluster room-to-room connectivity
Key characteristics of rack aware cluster
- Supports up to 4+4 nodes (four nodes per room or zone).
- Requires a round-trip latency of 1 millisecond (ms) or less between rooms.
- Uses two independent sets of bundled links for room-to-room storage traffic.
- Supports environments with or without Software Defined Networking (SDN) or Layer 2 virtual networks.
Key design principles of rack aware cluster
- Confine RDMA traffic: Storage VLANs 711 and 712 are Layer 2 broadcast domains that are restricted to the ToR layer and the room-to-room links. RDMA traffic never traverses the spine layer.
- Limit MLAG usage: Use Multi-Chassis Link Aggregation Group (MLAG) only for management and compute traffic. Keep storage interfaces point-to-point to minimize RDMA session hops, leveraging the low-latency, lossless nature of SMB Direct.
- Disaggregate storage VLANs: Assign VLAN 711 (SMB1) to ToR1 and ToR3. Assign VLAN 712 (SMB2) to ToR2 and ToR4. Each node storage network interface controller (NIC) port supports only one storage VLAN.
- Enable Quality of Service (QoS):Configure QoS on all interfaces that carry RDMA traffic, including room-to-room links. Missing QoS configurations can result in packet loss, performance degradation, and cluster instability.
Per-machine NIC layout
NIC port layout for each Azure Local node, including management, compute, and storage connections to top-of-rack switches.
| Network interface controller | Port | Intent | Switch connection |
| NIC1 | Port A | Management and Compute | ToR 1 or ToR 3 |
| NIC1 | Port B | Management and Compute | ToR 2 or ToR 4 |
| NIC2 | Port A | Storage (SMB1) | ToR 1 or ToR 3 |
| NIC2 | Port B | Storage (SMB2) | ToR 2 or ToR 4 |
Dell S5248F-ON port assignment recommendations for all TOR switches
Dell S5248F‑ON port layout color coding:
- Yellow indicates Azure Local node connectivity for management, compute, and storage traffic.
- Blue indicates room-to-room SMB connectivity between top-of-rack switches (ToR1 to ToR3 and ToR2 to ToR4).
- Red indicates uplinks to spine or aggregate switches.
- Green indicates top-of-rack VLT interconnects used for management and compute traffic only.
| Port range | Port type | Individual port speed | Description |
| Ethernet 1/1/1-1/1/48 | SFP28 | 25 Gbps | Machine access ports |
| Ethernet 1/1/49-1/1/52 | QSFP28-DD | 200 Gbps (100 Gbps per logical port) | Room-to-room SMB connections |
| Ethernet 1/1/53-1/1/54 | QSFP28 | 100 Gbps | Spine or Aggregate switch uplinks |
| Ethernet 1/1/55-1/1/56 | QSFP28 | 100 Gbps | ToR VLTI |
Network reference architecture Option A: Dedicated storage links examples
Switch configuration examples for room-to-room connectivity using Option A. These examples illustrate the recommended port channel and interface configuration on Dell S5248F‑ON switches.
For more information, see Azure Local rack aware cluster reference architecture - Option A.
Port channel configuration example
| S5248F-ON (TOR1) (Zone 1) | S5248F-ON (TOR2) (Zone 1) |
|
|
| S5248F-ON (TOR3) (Zone 2) | S5248F-ON (TOR4) (Zone 2) |
|
|
Interface configuration example
| S5248F-ON (TOR1) (Zone 1) | S5248F-ON (TOR2) (Zone 1) |
|
|
| S5248F-ON (TOR3) (Zone 2) | S5248F-ON (TOR4) (Zone 2) |
|
|
References
References
Dell Technologies documentation
These links provide more information from Dell Technologies:
- Dell AX Solutions for Microsoft documentation
- Dell AX System for Azure Local specification sheet
- iDRAC documentation
- Azure Local Support Matrix
- For more information about manually configuring iDRAC, see PowerEdge: Support Articles for the iDRAC and the CMC.