Typical Configuration Example of Lossless Network

Owned by Tracy Yang
Nov 18, 2024
8 min read

Networking Requirements

In Figure 1, in the following data center network PoD, two Server Clusters are connected to the network through devices Leaf 1, Leaf 2, Spine 1 and Spine 2. A lossless network is required to apply in the network devices to prevent packet loss during data transmission, ensuring high reliability and performance, especially for applications that are sensitive to latency and data integrity.

Setting up a lossless network involves careful planning, consistent configuration across all devices, and continuous monitoring and management. Follow the configuration roadmap below to ensure all components are correctly configured:

  • Configure PFC on all relevant ports to prevent packet loss by pausing traffic during congestion.

  • Employ a PFC watchdog mechanism to detect and mitigate PFC deadlocks.

  • Enable Explicit Congestion Notification (ECN) on all switches and endpoints to signal congestion without dropping packets. Configure ECN thresholds to detect and mark packets early before severe congestion occurs.

  • Enable dynamic load balancing, the traffic of Equal-Cost Multi-Path (ECMP) routing can be distributed across different member links through dynamic load balancing.

Figure 1. Typical Configuration Example of Lossless Network

RoCE example.jpg

Procedure

The following configuration steps are for Leaf 1. The configurations for Leaf 2, Spine 1 and Spine 2 are similar to Leaf 1 and will not be repeated.

Step 1         On Leaf 1, configure PFC on all relevant ports to prevent packet loss by pausing traffic during congestion.

The following commands complete the configurations:

  • Configure PFC profile pfc1.

  • Apply PFC profile to the port te-1/1/1.

admin@PICOS# set class-of-service pfc-profile pfc1 admin@PICOS#set class-of-service interface te-1/1/1 pfc-profile pfc1 admin@PICOS# commit

  • Show the class of service statistics information on specified interface.

The class 0~7 in PFC frame corresponds to the following "802.1P" item. The value of ”RxPFC“ item will be incremented by 1 if te-1/1/1 receives a PFC frame. The value of ”TxPFC“ item will be incremented by 1 if te-1/1/1 sends out a PFC frame.

admin@PICOS# run show class-of-service interface te-1/1/1 Interface : te-1/1/1 802.1P Priority Flow Control RxPFC TxPFC ----------- --------------------- --------------- --------------- 0 true 0 500 1 true 0 0 2 true 0 71 3 true 0 0 4 true 0 0 5 true 0 0 6 true 0 102 7 true 0 0 trust mode : no-trust Default ieee-802.1 : 0 Default dscp : 0 Default inet-precedence : 0 Local-priority Queue-Schedule Code-points -------------- --------------------------- ------------------------- 0 SP,0kbps 1 SP,0kbps 2 SP,0kbps 3 SP,0kbps 4 SP,0kbps 5 SP,0kbps 6 SP,0kbps 7 SP,0kbps

Step 2         (Optional) Configure the PFC buffer.    

After the PFC function is enabled, the default storage space is available for priority queues. You can flexibly make good use of the storage space through configuring buffer thresholds based on certain queues as needed.

The following commands complete the configurations:

  • Set the upper threshold of guaranteed buffer for PFC queue 3 on the ingress interface te-1/1/1 as 24000 cells.

  • Set the static threshold of shared buffer for PFC queue 1 on the ingress interface te-1/1/1 as 10000 cells.

  • Set the offset value of shared buffer for PFC queue 1 on the ingress interface te-1/1/1 as 3000 cells.

admin@PICOS# set interface gigabit-ethernet te-1/1/1 ethernet-switching-options buffer ingress-queue 3 guaranteed 24000 admin@PICOS# set interface gigabit-ethernet te-1/1/1 ethernet-switching-options buffer ingress-queue 1 threshold 10000 admin@PICOS# set interface gigabit-ethernet te-1/1/1 ethernet-switching-options buffer ingress-queue 1 reset-offset 3000 admin@PICOS# commit      

Step 3         Configure PFC watchdog.

The following commands complete the configurations:

  • Enable PFC watchdog on queue5 of interface te-1/1/1.

  • Configure the time interval of PFC deadlock detection to 10 x 100ms, where 100ms is the default value of PFC deadlock detection timer granularity.

  • Configure the restore time to 10 x 100ms when PFC deadlock occurs, where 100ms is the default value of PFC deadlock restore timer granularity.

admin@PICOS# set class-of-service interface te-1/1/1 pfc-watchdog code-point 5 enable true  admin@PICOS# set class-of-service pfc-watchdog code-point 5 detect-interval 10   admin@PICOS# set class-of-service pfc-watchdog code-point 5 restore-interval 10   admin@PICOS# commit        

After the configuration, use command run show pfc-watchdog config to view the configuration information about PFC watchdog.

admin@PICOS# run show pfc-watchdog config       PORT       ACTION          QUEUE    DETECTION TIME    RESTORATION TIME ----------  -----------   ------------  ----------------  ------------------ te-1/1/1          forward            5              1000                 1000

Use command run show pfc-watchdog stats to view the statistics information about PFC watchdog, including the number of PFC pause storms that have been detected and restored, as well as the number of packets that have been dropped, on the PFC queues on an interface.

admin@PICOS# run show pfc-watchdog stats QUEUE STATUS STORM DETECTED/RESTORED TX OK/DROP TX LAST OK/DROP ------------ ----------- ------------------------- ---------------- ----------------- te-1/1/1:5 stormed 9/8 82072626556/0 32053822365/0

Step 4         Configure ECN.

Users have to continuously monitor network performance and ECN marking rates. Make dynamic adjustments to the ECN thresholds as needed to respond to changing network conditions.

The following commands complete the configurations:

  • Enable WRED on queue 0 of interface te-1/1/1;

  • Set the maximum threshold to 400 and the minimum threshold to 200 on queue 0 of interface te-1/1/1;

  • Set the drop probability to 50% on queue 0 of interface te-1/1/1;

  • Enable ECN (Explicit Congestion Notification) on queue 0 of interface te-1/1/1.

admin@PICOS# set interface gigabit-ethernet te-1/1/3 wred queue 0 enable true admin@PICOS# set interface gigabit-ethernet te-1/1/3 wred queue 0 max_thresh 400 admin@PICOS# set interface gigabit-ethernet te-1/1/3 wred queue 0 min_thresh 200 admin@PICOS# set interface gigabit-ethernet te-1/1/3 wred queue 0 drop_probability 50 admin@PICOS# set interface gigabit-ethernet te-1/1/3 wred queue 0 ecn_thresh 1 admin@PICOS# commit

Show the WRED information of the specified interface.

admin@PICOS# run show interface gigabit-ethernet te-1/1/1 wred Queue Num Min Thresh Max Thresh Drop Probability ECN Thresh Status ---------- ---------- ---------- ---------------- ----------- ---------- 0 200 400 50% Enabled Enabled 1 0 0 0% Disabled Disabled 2 0 0 0% Disabled Disabled 3 0 0 0% Disabled Disabled 4 0 0 0% Disabled Disabled 5 0 0 0% Disabled Disabled 6 0 0 0% Disabled Disabled 7 0 0 0% Disabled Disabled

  Step 5         Enable normal mode of Dynamic Load Balancing for ECMP.

By enabling dynamic load balancing, the traffic of Equal-Cost Multi-Path (ECMP) routing can be distributed across different member links through dynamic load balancing, maximizing load balancing among the member links.

admin@PICOS# set interface ecmp hash-mapping dlb-normal admin@PICOS# commit

 

Copyright © 2024 Pica8 Inc. All Rights Reserved.