Overview

Lossless networks are crucial for the implementation of RDMA over Converged Ethernet (RoCE), a network protocol that enables high-throughput and low-latency data communication between nodes in a network. RoCE leverages the advantages of Remote Direct Memory Access (RDMA) technology over standard Ethernet networks, making it highly suitable for applications requiring rapid data transfer and minimal latency, such as high-performance computing, distributed storage, AI deep learning model training and big data analytics.

Advantages

1. Low Latency: RoCE reduces the latency of data transfers by bypassing the CPU and the operating system in the data path, allowing direct memory-to-memory data transfers.

2. High Throughput: By enabling direct memory access, RoCE can achieve higher data transfer rates compared to traditional Ethernet communications.

3. Efficient CPU Usage: Because RoCE offloads data transfer tasks from the CPU, it frees up CPU cycles for other processing tasks, enhancing overall system performance.

4. Ethernet Compatibility: RoCE leverages the existing Ethernet infrastructure, making it cost-effective to deploy in data centers without requiring specialized networking equipment.

Versions of RoCE

There are two main versions of RoCE:

1. RoCEv1: This version operates at Layer 2 of the OSI model, meaning it is not routable beyond the local Ethernet network. It requires all devices to be in the same Ethernet broadcast domain.

2. RoCEv2: This version operates at Layer 3, making it routable over IP networks. RoCEv2 packets can traverse multiple subnets, allowing for greater scalability and flexibility in network design.

Supported Platforms

To achieve lossless network communication, high throughput, and low latency data exchange between nodes, features like PFC Watchdog, Explicit Congestion Notification (ECN), and Dynamic Load Balancing are only supported on Trident3 and Tomahawk3 platforms.

Switch Configuration

To effectively leverage RoCE in your network, configuring your switches properly is crucial. Here’s the guide to configure lossless network on network switches, focusing on key features: PFC, PFC Watchdog, ECN and Dynamic Load Balancing (DLB).

• Interface

  • Enable with PFC on the required priority.

• Global Configuration

  • Configure ECN via WRED

    • Enable WRED.

    • Set the maximum and minimum thresholds.

    • Set drop probability.

    • Enable ECN.

  • Configure PFC Watchdog

    • Enable PFC on the interface before enabling PFC watchdog.

    • Enable PFC watchdog.

    • Configure the time interval of PFC deadlock detection.

    • Configure the restore time and restore action when PFC deadlock occurs.

  • Configure Dynamic Load Balancing