Networking Requirements

In Figure 1, in the following data center network PoD, two Server Clusters are connected to the network through devices Leaf 1, Leaf 2, Spine 1 and Spine 2. A lossless network is required to apply in the network devices to prevent packet loss during data transmission, ensuring high reliability and performance, especially for applications that are sensitive to latency and data integrity.

Setting up a lossless network involves careful planning, consistent configuration across all devices, and continuous monitoring and management. Follow the configuration roadmap below to ensure all components are correctly configured:

Configure PFC on all relevant ports to prevent packet loss by pausing traffic during congestion.
Employ a PFC watchdog mechanism to detect and mitigate PFC deadlocks.
Enable Explicit Congestion Notification (ECN) on all switches and endpoints to signal congestion without dropping packets. Configure ECN thresholds to detect and mark packets early before severe congestion occurs.
Enable dynamic load balancing, the traffic of Equal-Cost Multi-Path (ECMP) routing can be distributed across different member links through dynamic load balancing.

Figure 1. Typical Configuration Example of Lossless Network

Procedure

The following configuration steps are for Leaf 1. The configurations for Leaf 2, Spine 1 and Spine 2 are similar to Leaf 1 and will not be repeated.

Step1 On Leaf 1, configure PFC on all relevant ports to prevent packet loss by pausing traffic during congestion.

The following commands complete the configurations:

Configure PFC profile pfc1.
Apply PFC profile to the port te-1/1/1.

admin@PICOS# set class-of-service pfc-profile pfc1
admin@PICOS#set class-of-service interface te-1/1/1 pfc-profile pfc1
admin@PICOS# commit

Show the class of service statistics information on specified interface.

The class 0~7 in PFC frame corresponds to the following "802.1P" item. The value of ”RxPFC“ item will be incremented by 1 if te-1/1/1 receives a PFC frame. The value of ”TxPFC“ item will be incremented by 1 if te-1/1/1 sends out a PFC frame.

admin@PICOS# run show class-of-service interface te-1/1/1
Interface : te-1/1/1
802.1P         Priority Flow Control    RxPFC              TxPFC
-----------    ---------------------    ---------------    ---------------
0              false                     0                  500
1              false                     0                  0
2              false                     0                  71
3              false                     0                  0
4              false                     0                  0
5              false                     0                  0
6              false                     0                  102
7              false                     0                  0
trust mode : no-trust
Default ieee-802.1 : 0
Default dscp : 0
Default inet-precedence : 0
Local-priority   Queue-Schedule                Code-points
--------------   ---------------------------   -------------------------
0                SP,0kbps
1                SP,0kbps
2                SP,0kbps
3                SP,0kbps
4                SP,0kbps
5                SP,0kbps
6                SP,0kbps
7                SP,0kbps

Step2 Configure PFC watchdog.

The following commands complete the configurations:

Enable PFC watchdog on queue5 of interface te-1/1/1.
Configure the time interval of PFC deadlock detection to 10 x 100ms, where 100ms is the default value of PFC deadlock detection timer granularity.
Configure the restore time to 10 x 100ms when PFC deadlock occurs, where 100ms is the default value of PFC deadlock restore timer granularity.

admin@PICOS# set class-of-service interface te-1/1/1 pfc-watchdog code-point 5 enable true 
admin@PICOS# set class-of-service pfc-watchdog code-point 5 detect-interval 10  
admin@PICOS# set class-of-service pfc-watchdog code-point 5 restore-interval 10  
admin@PICOS# commit

After the configuration, use command run show pfc-watchdog config to view the configuration information about PFC watchdog.

Use command run show pfc-watchdog stats to view the statistics information about PFC watchdog, including the number of PFC pause storms that have been detected and restored, as well as the number of packets that have been dropped, on the PFC queues on an interface.

Step3 Configure ECN.

Users have to continuously monitor network performance and ECN marking rates. Make dynamic adjustments to the ECN thresholds as needed to respond to changing network conditions.

The following commands complete the configurations:

Enable WRED on queue 0 of interface te-1/1/1;
Set the maximum threshold to 400 and the minimum threshold to 200 on queue 0 of interface te-1/1/1;
Set the drop probability to 50% on queue 0 of interface te-1/1/1;
Enable ECN (Explicit Congestion Notification) on queue 0 of interface te-1/1/1.

Show the WRED information of the specified interface.

Step4 Enable normal mode of Dynamic Load Balancing for ECMP.

By enabling dynamic load balancing, the traffic of Equal-Cost Multi-Path (ECMP) routing can be distributed across different member links through dynamic load balancing, maximizing load balancing among the member links.