Dell EMC SmartFabric OS10 User Guide Release 10.5.0

To configure RoCE, classify the ingress and egress traffic as lossy and lossless traffic. Based on the classification, assign the ECN threshold and bandwidth for the respective queues. For RoCEv1, ECN threshold configuration is not required.

Configuration notes

• Use the trust-map or policy-map CLI commands to configure dot1p and DSCP traffic-class markings. For RoCEv2, classification is based only on DSCP.
• Use the qos-map CLI command to apply the traffic class to queues.
• Use the network-type policy-map to classify any of the priority values as lossless and fine-tune the respective buffer value depending on traffic congestion.
• Adjust the ECN threshold based on the traffic pattern.
• Use the queuing-type policy-map to modify the bandwidth allocation for lossy and lossless queues.
• If you are using RoCEv1, only bandwidth allocation is required. ECN and ECN queue association are not required.
• To ensure lossless traffic flow, configure PFC on all lossless interfaces.

The following example describes the steps to configure RoCE on the switch. This configuration example uses priority 3 for RoCE.

1. Enter CONFIGURATION mode.

   OS10# configure terminal
   OS10 (config)#

2. Enable the Data Center Bridging Exchange protocol (DCBX). See Data center bridging exchange (DCBX) for more information.

   OS10 (config)# dcbx enable
   

3. Create a VLAN. In this example, VLAN 55 switches the RoCE traffic. You can configure any value from 1 to 4093.

   OS10 (config)# interface vlan 55
   OS10 (conf-if-vl-55)# no shutdown

4. Apply the dot1p trust globally or at the interface level. In this example, the dot1p trust is applied globally.
   NOTE

   • If PFC configuration is not enabled on all the ports in the switch, do not apply dot1p trust globally. Apply the dot1p trust on the specific interfaces.
   • For RoCEv1, use the trust-map dot1p default command or the user-defined trust-map dot1p configuration.
   • For RoCEv2, use the trust-map dscp default command or the user-defined trust-map dscp configuration.

   OS10 (config)# system qos
   OS10 (config-sys-qos)# trust-map dot1p default

5. Create a network-qos type class-map and policy-map for priority flow control (PFC). This configuration fine tunes the buffer settings for the particular priority.

   OS10 (config)# class-map type network-qos pfcdot1p3
   OS10 (config-cmap-nqos)# match qos-group 3
   
   OS10 (config)# policy-map type network-qos policy_pfcdot1p3
   OS10(config-pmap-network-qos)# class pfcdot1p3
   OS10(config-pmap-c-nqos)# pause
   OS10(config-pmap-c-nqos)# pfc-cos 3

   NOTE When you use the pause command without any parameters, the system uses the default buffer settings. To modify the buffer settings, use the pause command and specify the buffer size, pause threshold, and resume threshold. See Priority flow control and the pause command for more information.
6. Create queuing-type class-maps and policy-map for enhanced transmission selection (ETS), bandwidth, and ECN configurations. See Enhanced transmission selection and Bandwidth allocation for more information.

   Bandwidth configuration for RoCEv1:

   OS10 (config)# class-map type queuing Q0
   OS10 (config-cmap-queuing)# match queue 0
   OS10 (config)# class-map type queuing Q3
   OS10 (config-cmap-queuing)# match queue 3

   OS10 (config)# policy-map type queuing policy_2Q
   OS10 (config-pmap-queuing)# class Q0
   OS10 (config-pmap-c-que)# bandwidth percent 30
   OS10 (config-pmap-c-que)# exit
   OS10 (config-pmap-queuing)# class Q3
   OS10 (config-pmap-c-que)# bandwidth percent 70

   Bandwidth and ECN configuration for RoCEv2 with ECN queue association:

   OS10 (config)# class-map type queuing Q0
   OS10 (config-cmap-queuing)# match queue 0
   OS10 (config)# class-map type queuing Q3
   OS10 (config-cmap-queuing)# match queue 3

   OS10(config)# wred wred_ecn
   OS10(config-wred)# random-detect ecn
   OS10(config-wred)# random-detect color green minimum-threshold 1000 maximum-threshold 2000 drop-probability 100
   OS10(config-wred)# random-detect color yellow minimum-threshold 500 maximum-threshold 1000 drop-probability 100
   OS10(config-wred)# random-detect color red minimum-threshold 100 maximum-threshold 500 drop-probability 100
   OS10(config-wred)# exit

   OS10(config)# policy-map type queuing policy_2Q
   OS10(config-pmap-queuing)# class Q0
   OS10(config-pmap-c-que)# bandwidth percent 30
   OS10(config-pmap-c-que)# exit
   OS10(config-pmap-queuing)# class Q3
   OS10(config-pmap-c-que)# bandwidth percent 70
   OS10(config-pmap-c-que)# random-detect wred_ecn
   OS10(config-pmap-c-que)# end
   OS10#

7. Create a QoS map for ETS to map the lossy and lossless traffic to the respective queues.

   OS10 (config)# qos-map traffic-class 2Q
   OS10(config-qos-map)# queue 0 qos-group 0-2, 4-7
   OS10(config-qos-map)# queue 3 qos-group 3

8. Perform the following configurations on all switch interfaces where you want to support RoCE:
   • For RoCEv1:
     1. Enter INTERFACE mode and enter the no shutdown command.

        OS10# configure terminal
        OS10 (config)# interface ethernet 1/1/1
        OS10 (conf-if-eth1/1/1)# no shutdown

     2. Change the switch port mode to Trunk mode.

        OS10 (conf-if-eth1/1/1)# switchport mode trunk
        

     3. Specify the allowed VLANs on the trunk port.

        OS10 (conf-if-eth1/1/1)# switchport trunk allowed vlan 55

     4. Apply the network-qos type policy-map to the interface.

        OS10 (conf-if-eth1/1/1)# service-policy input type network-qos policy_pfcdot1p3

     5. Apply the queuing policy to egress traffic on the interface.

        OS10 (conf-if-eth1/1/1)# service-policy output type queuing policy_2Q

     6. Enable ETS on the interface.

        OS10 (conf-if-eth1/1/1)# ets mode on

     7. Apply the qos-map for ETS configurations on the interface.

        OS10 (conf-if-eth1/1/1)# qos-map traffic-class 2Q

     8. Enable PFC on the interface.

        OS10 (conf-if-eth1/1/1)# priority-flow-control mode on

   • For RoCEv2:
     1. Enter INTERFACE mode and enter the no shutdown command.

        OS10# configure terminal
        OS10 (config)# interface ethernet 1/1/1
        OS10 (conf-if-eth1/1/1)# no shutdown

     2. Apply the network-qos type policy-map to the interface.

        OS10 (conf-if-eth1/1/1)# service-policy input type network-qos policy_pfcdot1p3

     3. Apply the queuing policy to egress traffic on the interface.

        OS10 (conf-if-eth1/1/1)# service-policy output type queuing policy_2Q

     4. Enable ETS on the interface.

        OS10 (conf-if-eth1/1/1)# ets mode on

     5. Apply the qos-map for ETS configurations on the interface.

        OS10 (conf-if-eth1/1/1)# qos-map traffic-class 2Q

     6. Enable PFC on the interface.

        OS10 (conf-if-eth1/1/1)# priority-flow-control mode on

View configuration and statistics

Use the following show commands to view the configuration and statistics:

• To view the PFC and ETS configuration details at the interface level, use the show qos interface command:

  OS10# show qos interface ethernet 1/1/4

• To view the buffer allocation for the ingress interface, use the show qos ingress buffers command:

  OS10# show qos ingress buffers interface ethernet 1/1/4

• To view the buffer utilization at the ingress interface, use the show qos ingress buffer-stats command:

  OS10# show qos ingress buffer-stats interface ethernet 1/1/4

• To view the buffer allocation for the egress interface, use the show qos egress buffers command:

  OS10# show qos egress buffers interface ethernet 1/1/4

• To view the buffer utilization at the egress interface, use the show qos egress buffer-stats command:

  OS10# show qos egress buffer-stats interface ethernet 1/1/4

• To view the PFC configuration, operational status, and statistics on the interface, use the show interface interface-name priority-flow-control details command:

  OS10(config)# show interface ethernet 1/1/15 priority-flow-control details

• To view the ECN markings on an interface, use the show queuing statistics interface interface-name wred command:

  OS10# show queuing statistics interface ethernet 1/1/1 wred

• To view any egress packet loss, use the show queuing statistics command:
  NOTE There should not be any packet drops in lossless queues.

  OS10# show queuing statistics interface ethernet 1/1/1

• To view qos map details such as dot1p or DSCP to traffic class mapping and traffic class to queue mapping, use the show qos maps command:

  OS10# show qos maps